Fix race in waitUntilSettled
The following sequence can happen:
- Lock run handler in test (test thread)
- Verify that queues are empty (test thread)
- Nodepool request completes (fake nodepool thread)
- Nodepool request complete event added to queue (zookeeper thread)
- Nodepool request record removed (zookeeper thread)
- waitUntilSettled declares settled (test thread)
This should prevent that by ensuring that we check for empty queues
*after* verifying that there are no outstanding nodepool requests.
Change-Id: I0d7fbffec8d9d9c8a53d571afab82dd59a2537e4
diff --git a/tests/base.py b/tests/base.py
index 4a3daeb..1fb391c 100755
--- a/tests/base.py
+++ b/tests/base.py
@@ -1606,10 +1606,15 @@
self.eventQueuesJoin()
self.sched.run_handler_lock.acquire()
if (not self.merge_client.jobs and
- all(self.eventQueuesEmpty()) and
self.haveAllBuildsReported() and
self.areAllBuildsWaiting() and
- self.areAllNodeRequestsComplete()):
+ self.areAllNodeRequestsComplete() and
+ all(self.eventQueuesEmpty())):
+ # The queue empty check is placed at the end to
+ # ensure that if a component adds an event between
+ # when locked the run handler and checked that the
+ # components were stable, we don't erroneously
+ # report that we are settled.
self.sched.run_handler_lock.release()
self.launch_server.lock.release()
self.log.debug("...settled.")