Fix stuck node requests across ZK reconnection
When a request is fulfilled by nodepool, we add it to the scheduler's
event queue, and later, the scheduler processes the event and accepts
the nodes. If there is a ZooKeeper disconnection in the interim, then
we will have noticed it and not locked the nodes, however, the scheduler
will still pass on the request to the pipeline manager and we will
attempt to run jobs on the unlocked nodes, which will continually
fail.
This change extends the handling of a lost request so that if it happens,
we retry the request (which is what would happen if the request is lucky
enough to have been lost before fulfillment).
This extends the fix in 94e95886e2179f4a6aeecad687509bc7b1ab7fd3.
Change-Id: If81a790ed8b16594f4f9186d9256200b8d5e707e
3 files changed