Handle nodepool allocation failure
When a request is either fulfilled or failed, pass it through to
the scheduler which will accept the request (which means deleting
it in the case of a failure) and pass it on to the pipeline manager
which will set the result of the requesting job to NODE_FAILURE
and cause any sub-jobs to be SKIPPED.
Adjust the request algorithm to only request nodes for jobs that
are ready to run. The current behavior requests all jobs for a
build set asap, but that has two downsides: it may request and
return nodes more aggressively than necessary (if you have chosen
to create a job tree, you *probably* don't want to tie up nodes
until they are actually needed). However, that's a grey area,
and we may want to adjust or make that behavior configurable later.
More pressing here is that it makes the logic of when to return
nodes *very* complicated (since SKIPPED jobs are represented by
fake builds, there is no good opportunity to return their nodes).
This seems like a good solution for now, and if we want to make
the node request behavior more aggressive in the future, we can
work out a better model for knowing when to return nodes.
Change-Id: Ideab6eb5794a01d5c2b70cb87d02d61bb3d41cce
diff --git a/zuul/scheduler.py b/zuul/scheduler.py
index 5f51cbf..5e49f20 100644
--- a/zuul/scheduler.py
+++ b/zuul/scheduler.py
@@ -811,22 +811,19 @@
request = event.request
build_set = request.build_set
- try:
- self.nodepool.acceptNodes(request)
- except Exception:
- self.log.exception("Unable to accept nodes from request %s:"
- % (request,))
- return
+ self.nodepool.acceptNodes(request)
if build_set is not build_set.item.current_build_set:
self.log.warning("Build set %s is not current" % (build_set,))
- self.nodepool.returnNodeset(request.nodeset)
+ if request.fulfilled:
+ self.nodepool.returnNodeset(request.nodeset)
return
pipeline = build_set.item.pipeline
if not pipeline:
self.log.warning("Build set %s is not associated with a pipeline" %
(build_set,))
- self.nodepool.returnNodeset(request.nodeset)
+ if request.fulfilled:
+ self.nodepool.returnNodeset(request.nodeset)
return
pipeline.manager.onNodesProvisioned(event)