Ordering of broadcast messages is not preserved

[colonelpanic8]

I believe that this is because there is a race condition at

bcast/bcast.go

Line 112 in 79e4f35

}

and

bcast/bcast.go

Line 150 in 79e4f35

}

I believe that what is happening is that sometimes go routines that were scheduled later than other goroutines by these lines for a given channel get called first.

I see three ways to approach fixing this:

Option 1 is to simply bite the bullet and call out to each member's channel synchronously.
Option 2 is to start using a logical clock for events and passing those values through to the consumer
Option 3 also involves a logical clock, but it hides it from consumers by having an intermediate goroutine/channel pair running for each member that handles ensuring that the events are properly serialized

[colonelpanic8]

Just noticed that #8 was filed recently. I am also using 1.5.1. Might this be related?

[colonelpanic8]

evidence that this is indeed what is happening

in bcast %d &{UpdateStarted <nil>}
in bcast %d &{HostsAdded [10.160.10.1:7187]}
in bcast %d &{UpdateFinished <nil>}
&{UpdateStarted <nil>}
&{UpdateFinished <nil>}
&{HostsAdded [10.160.10.1:7187]}
in bcast %d &{UpdateStarted <nil>}
&{UpdateStarted <nil>}
in bcast %d &{HostsAdded [10.160.10.2:7187 10.160.10.3:7187]}
in bcast %d &{UpdateFinished <nil>}
&{UpdateFinished <nil>}
&{HostsAdded [10.160.10.2:7187 10.160.10.3:7187]}
in bcast %d &{UpdateStarted <nil>}
&{UpdateStarted <nil>}
in bcast %d &{HostsAdded [10.160.10.5:7187 10.160.10.4:7187]}
&{HostsAdded [10.160.10.5:7187 10.160.10.4:7187]}
in bcast %d &{HostsRemoved [10.160.10.2:7187]}
&{HostsRemoved [10.160.10.2:7187]}
in bcast %d &{HostsRemoved [10.160.10.1:7187]}
in bcast %d &{UpdateFinished <nil>}
&{UpdateFinished <nil>}
[0xc8200ab1a0 0xc8201d2000 0xc8201d20e0 0xc8201d2120 0xc8201d21e0]
PASS
ok      code.uber.internal/go-common.git/client/riakbasho       0.011s

The things that aren't labeled in bcast are from something that is just reading the channel. Notice that the orderings are different