Stop sharing Gerrit event queues in tests

When connections are set up in tests, multiple Gerrit connections which
are configured to connect to the same fake Gerrit server share a change
database so that changes sent to Gerrit via one connection are reflected
back to Zuul on another.  They also share an event queue so that events
injected on one are seen by another.

Unfortunately, that part doesn't work, and in fact, events are only seen
by one of the gerrit connections.  This happens to work since it doesn't
matter which gerrit connection receives an event, which is why we haven't
noticed the problem in tests.

Where we do see the problem in Zuulv3 is in shutting down the connections.
When a GerritConnection is stopped, a sentinal object (None) is added to
the event queue.  When the GerritConnection gets an event from the queue,
it first checks whether it has stopped before processing that event.
Because in tests (but not in production) multiple GerritConnections share
an event queue, the connection that adds the None object to the queue
may not be the one that receives it, which causes the test to raise an
exception and not stop correctly.

We did not notice this in v2 because the order in which the Queue.Queue
class decides to awaken a thread is deterministic enough that the thread
which submitted the sentinel was always the one that received it.  In
v3, the thread order is sufficiently different that the thread for the
*other* connection is reliably the one which receives it.

To correct this, stop using a shared queue between the differing
GerritConnection objects, and instead add a helper method to the testcase
class which will add an event to every connection for a given server.

Change-Id: Idd3238f5ab8f5e09e295c0fa028e140c089a2a3f
3 files changed