Merge "Dependencies from undefined projects"
diff --git a/doc/source/gating.rst b/doc/source/gating.rst
index 43a5928..fff2924 100644
--- a/doc/source/gating.rst
+++ b/doc/source/gating.rst
@@ -212,19 +212,20 @@
}
-Cross projects dependencies
----------------------------
+Cross Project Testing
+---------------------
When your projects are closely coupled together, you want to make sure
changes entering the gate are going to be tested with the version of
other projects currently enqueued in the gate (since they will
eventually be merged and might introduce breaking features).
-Such dependencies can be defined in Zuul configuration by registering a job
-in a DependentPipeline of several projects. Whenever a change enters such a
-pipeline, it will create references for the other projects as well. As an
-example, given a main project ``acme`` and a plugin ``plugin`` you can
-define a job ``acme-tests`` which should be run for both projects:
+Such relationships can be defined in Zuul configuration by registering
+a job in a DependentPipeline of several projects. Whenever a change
+enters such a pipeline, it will create references for the other
+projects as well. As an example, given a main project ``acme`` and a
+plugin ``plugin`` you can define a job ``acme-tests`` which should be
+run for both projects:
.. code-block:: yaml
@@ -280,3 +281,82 @@
When your job fetches several repositories without changes ahead in the
queue, they may not have a Z reference in which case you can just check
out the branch.
+
+
+Cross Repository Dependencies
+-----------------------------
+
+Zuul permits users to specify dependencies across repositories. Using
+a special header in Git commit messages, Users may specify that a
+change depends on another change in any repository known to Zuul.
+
+Zuul's cross-repository dependencies (CRD) behave like a directed
+acyclic graph (DAG), like git itself, to indicate a one-way dependency
+relationship between changes in different git repositories. Change A
+may depend on B, but B may not depend on A.
+
+To use them, include "Depends-On: <gerrit-change-id>" in the footer of
+a commit message. Use the full Change-ID ('I' + 40 characters).
+
+
+Gate Pipeline
+~~~~~~~~~~~~~
+
+When Zuul sees CRD changes, it serializes them in the usual manner when
+enqueuing them into a pipeline. This means that if change A depends on
+B, then when they are added to the gate pipeline, B will appear first
+and A will follow. If tests for B fail, both B and A will be removed
+from the pipeline, and it will not be possible for A to merge until B
+does.
+
+Note that if changes with CRD do not share a change queue then Zuul
+is unable to enqueue them together, and the first will be required to
+merge before the second is enqueued.
+
+Check Pipeline
+~~~~~~~~~~~~~~
+
+When changes are enqueued into the check pipeline, all of the related
+dependencies (both normal git-dependencies that come from parent commits
+as well as CRD changes) appear in a dependency graph, as in gate. This
+means that even in the check pipeline, your change will be tested with
+its dependency. So changes that were previously unable to be fully
+tested until a related change landed in a different repo may now be
+tested together from the start.
+
+All of the changes are still independent (so you will note that the
+whole pipeline does not share a graph as in gate), but for each change
+tested, all of its dependencies are visually connected to it, and they
+are used to construct the git references that Zuul uses when testing.
+When looking at this graph on the status page, you will note that the
+dependencies show up as grey dots, while the actual change tested shows
+up as red or green. This is to indicate that the grey changes are only
+there to establish dependencies. Even if one of the dependencies is
+also being tested, it will show up as a grey dot when used as a
+dependency, but separately and additionally will appear as its own red
+or green dot for its test.
+
+Multiple Changes
+~~~~~~~~~~~~~~~~
+
+A Gerrit change ID may refer to multiple changes (on multiple branches
+of the same project, or even multiple projects). In these cases, Zuul
+will treat all of the changes with that change ID as dependencies. So
+if you say that change in project A Depends-On a change ID that has
+changes in two branches of project B, then when testing the change to
+project A, both project B changes will be applied, and when deciding
+whether the project A change can merge, both changes must merge ahead
+of it.
+
+A change may depend on more than one Gerrit change ID as well. So it
+is possible for a change in project A to depend on a change in project
+B and a change in project C. Simply add more "Depends-On:" lines to
+the footer.
+
+Cycles
+~~~~~~
+
+If a cycle is created by use of CRD, Zuul will abort its work very
+early. There will be no message in Gerrit and no changes that are part
+of the cycle will be enqueued into any pipeline. This is to protect
+Zuul from infinite loops.
diff --git a/doc/source/launchers.rst b/doc/source/launchers.rst
index b95354f..0a1e0e7 100644
--- a/doc/source/launchers.rst
+++ b/doc/source/launchers.rst
@@ -66,6 +66,11 @@
**LOG_PATH**
zuul also suggests a unique path for logs to the worker. This is
"BASE_LOG_PATH/pipeline-name/job-name/uuid"
+**ZUUL_VOTING**
+ Whether Zuul considers this job voting or not. Note that if Zuul is
+ reconfigured during the run, the voting status of a job may change
+ and this value will be out of date. Values are '1' if voting, '0'
+ otherwise.
Change related parameters
~~~~~~~~~~~~~~~~~~~~~~~~~
diff --git a/test-requirements.txt b/test-requirements.txt
index c68b2db..4ae3eb3 100644
--- a/test-requirements.txt
+++ b/test-requirements.txt
@@ -2,6 +2,11 @@
coverage>=3.6
sphinx>=1.1.2,!=1.2.0,!=1.3b1,<1.3
+# NOTE(tonyb) Pillow isn't directly needed but it's pulled in via
+# Collecting Pillow (from blockdiag>=1.5.0->sphinxcontrib-blockdiag>=0.5.5
+# So cap as per global-requirements until https://launchpad.net/bugs/1501995
+# is properly fixed
+Pillow>=2.4.0,<3.0.0 # MIT
sphinxcontrib-blockdiag>=0.5.5
discover
fixtures>=0.3.14
diff --git a/tests/base.py b/tests/base.py
index 5ddb160..abbdb0a 100755
--- a/tests/base.py
+++ b/tests/base.py
@@ -624,6 +624,8 @@
result = 'RUN_ERROR'
else:
data['result'] = result
+ data['node_labels'] = ['bare-necessities']
+ data['node_name'] = 'foo'
work_fail = False
changes = None
diff --git a/tests/test_scheduler.py b/tests/test_scheduler.py
index 7ed7efb..6276569 100755
--- a/tests/test_scheduler.py
+++ b/tests/test_scheduler.py
@@ -111,6 +111,9 @@
self.assertReportedStat(
'zuul.pipeline.gate.org.project.total_changes', value='1|c')
+ for build in self.builds:
+ self.assertEqual(build.parameters['ZUUL_VOTING'], '1')
+
def test_initial_pipeline_gauges(self):
"Test that each pipeline reported its length on start"
pipeline_names = self.sched.layout.pipelines.keys()
@@ -1256,6 +1259,9 @@
self.getJobFromHistory('nonvoting-project-test2').result,
'FAILURE')
+ for build in self.builds:
+ self.assertEqual(build.parameters['ZUUL_VOTING'], '0')
+
def test_check_queue_success(self):
"Test successful check queue jobs."
diff --git a/zuul/cmd/client.py b/zuul/cmd/client.py
index 6e14ff5..59ac419 100644
--- a/zuul/cmd/client.py
+++ b/zuul/cmd/client.py
@@ -263,6 +263,12 @@
'number': {
'title': 'Number'
},
+ 'node_labels': {
+ 'title': 'Node Labels'
+ },
+ 'node_name': {
+ 'title': 'Node Name'
+ },
'worker.name': {
'title': 'Worker'
},
@@ -276,7 +282,7 @@
'worker.fqdn': {
'title': 'Worker Domain'
},
- 'worker.progam': {
+ 'worker.program': {
'title': 'Worker Program'
},
'worker.version': {
diff --git a/zuul/launcher/gearman.py b/zuul/launcher/gearman.py
index 828e2a9..69fb71b 100644
--- a/zuul/launcher/gearman.py
+++ b/zuul/launcher/gearman.py
@@ -277,6 +277,7 @@
ZUUL_PROJECT=item.change.project.name)
params['ZUUL_PIPELINE'] = pipeline.name
params['ZUUL_URL'] = item.current_build_set.zuul_url
+ params['ZUUL_VOTING'] = job.voting and '1' or '0'
if hasattr(item.change, 'refspec'):
changes_str = '^'.join(
['%s:%s:%s' % (i.change.project.name, i.change.branch,
@@ -422,9 +423,11 @@
build = self.builds.get(job.unique)
if build:
+ data = getJobData(job)
+ build.node_labels = data.get('node_labels', [])
+ build.node_name = data.get('node_name')
if not build.canceled:
if result is None:
- data = getJobData(job)
result = data.get('result')
if result is None:
build.retry = True
diff --git a/zuul/model.py b/zuul/model.py
index 3a228f5..38628f9 100644
--- a/zuul/model.py
+++ b/zuul/model.py
@@ -585,6 +585,8 @@
self.retry = False
self.parameters = {}
self.worker = Worker()
+ self.node_labels = []
+ self.node_name = None
def __repr__(self):
return ('<Build %s of %s on %s>' %
@@ -806,7 +808,9 @@
'canceled': build.canceled if build else None,
'retry': build.retry if build else None,
'number': build.number if build else None,
- 'worker': worker
+ 'node_labels': build.node_labels if build else [],
+ 'node_name': build.node_name if build else None,
+ 'worker': worker,
})
if self.pipeline.haveAllJobsStarted(self):
diff --git a/zuul/scheduler.py b/zuul/scheduler.py
index 4375f98..2340800 100644
--- a/zuul/scheduler.py
+++ b/zuul/scheduler.py
@@ -534,6 +534,16 @@
def onBuildStarted(self, build):
self.log.debug("Adding start event for build: %s" % build)
build.start_time = time.time()
+ try:
+ if statsd and build.pipeline:
+ jobname = build.job.name.replace('.', '_')
+ key = 'zuul.pipeline.%s.job.%s.wait_time' % (
+ build.pipeline.name, jobname)
+ dt = int((build.start_time - build.launch_time) * 1000)
+ statsd.timing(key, dt)
+ statsd.incr(key)
+ except:
+ self.log.exception("Exception reporting runtime stats")
event = BuildStartedEvent(build)
self.result_event_queue.put(event)
self.wake_event.set()
@@ -551,14 +561,25 @@
try:
if statsd and build.pipeline:
jobname = build.job.name.replace('.', '_')
+ key = 'zuul.pipeline.%s.all_jobs' % build.pipeline.name
+ statsd.incr(key)
+ for label in build.node_labels:
+ # Jenkins includes the node name in its list of labels, so
+ # we filter it out here, since that is not statistically
+ # interesting.
+ if label == build.node_name:
+ continue
+ dt = int((build.start_time - build.launch_time) * 1000)
+ key = 'zuul.node_type.%s.job.%s.wait_time' % (
+ label, jobname)
+ statsd.timing(key, dt)
+ statsd.incr(key)
key = 'zuul.pipeline.%s.job.%s.%s' % (build.pipeline.name,
jobname, build.result)
if build.result in ['SUCCESS', 'FAILURE'] and build.start_time:
dt = int((build.end_time - build.start_time) * 1000)
statsd.timing(key, dt)
statsd.incr(key)
- key = 'zuul.pipeline.%s.all_jobs' % build.pipeline.name
- statsd.incr(key)
except:
self.log.exception("Exception reporting runtime stats")
event = BuildCompletedEvent(build)