Mutex repo updates and merge ops in executor
The executor maintains a set of all of the repos known to Zuul.
This is primarily for the purpose of having a local cache from which
to clone the repos which are used in jobs. To that end, before each
job starts, it submits a request to a queue to update all the repos
in the job. This makes sure they are up to date before being cloned
for the job.
Since we have that set of repos handy, we can also use them to perform
merge operations. In other words, the executor con also act as a
merger. This can be useful under heavy load, or in the case of a
very simple Zuul installation which has no other merger.
However, merge and update operations must not run at the same time, as
simultaneous access to the git repo may cause errors. To that end,
set up a mutex around merge jobs and update tasks.
Since the primary purpose of the repos is to perform update tasks,
create a special gearman worker for the merge tasks so that we
can de-prioritize them in the executor. In this, we delay response
to a NOOP packet until the update queue is empty. That means that
if the gearman server notifies us that a merge job is ready, we
won't grab it unless our merger is otherwise idle (we can still
race here and get an update task between NOOP and GRAB_JOB, but
there's little we can do about that, and the worst case is that
we briefly delay a merge job).
Since the executor jobs are now in a different worker, their NOOP/
GRAB_JOB cycle is unimpeded, even if the merge worker is waiting.
Change-Id: Icf3663b1a2ce5309e496b1106d5adee6579e37c7
2 files changed