blob: 34d23f930bb7e54b1090cd29b3cce8fd1e560bfd [file] [log] [blame]
James E. Blair1de8d402017-05-07 17:08:04 -07001:title: Project Configuration
2
3.. _project-config:
4
5Project Configuration
6=====================
7
8The following sections describe the main part of Zuul's configuration.
9All of what follows is found within files inside of the repositories
10that Zuul manages.
11
12Security Contexts
13-----------------
14
15When a system administrator configures Zuul to operate on a project,
16they specify one of two security contexts for that project. A
17*config-project* is one which is primarily tasked with holding
18configuration information and job content for Zuul. Jobs which are
James E. Blairac3c7ae2017-07-31 09:01:08 -070019defined in a config-project are run with elevated privileges, and all
20Zuul configuration items are available for use. It is expected that
21changes to config-projects will undergo careful scrutiny before being
22merged.
James E. Blair1de8d402017-05-07 17:08:04 -070023
24An *untrusted-project* is a project whose primary focus is not to
25operate Zuul, but rather it is one of the projects being tested or
26deployed. The Zuul configuration language available to these projects
James E. Blairac3c7ae2017-07-31 09:01:08 -070027is somewhat restricted (as detailed in individual sections below), and
James E. Blair1de8d402017-05-07 17:08:04 -070028jobs defined in these projects run in a restricted execution
29environment since they may be operating on changes which have not yet
30undergone review.
31
32Configuration Loading
33---------------------
34
35When Zuul starts, it examines all of the git repositories which are
James E. Blairac3c7ae2017-07-31 09:01:08 -070036specified by the system administrator in :ref:`tenant-config` and
37searches for files in the root of each repository. Zuul looks first
38for a file named ``zuul.yaml`` or a directory named ``zuul.d``, and if
39they are not found, ``.zuul.yaml`` or ``.zuul.d`` (with a leading
40dot). In the case of an :term:`untrusted-project`, the configuration
41from every branch is included, however, in the case of a
42:term:`config-project`, only the ``master`` branch is examined.
James E. Blair1de8d402017-05-07 17:08:04 -070043
44When a change is proposed to one of these files in an
James E. Blairac3c7ae2017-07-31 09:01:08 -070045untrusted-project, the configuration proposed in the change is merged
46into the running configuration so that any changes to Zuul's
James E. Blair1de8d402017-05-07 17:08:04 -070047configuration are self-testing as part of that change. If there is a
48configuration error, no jobs will be run and the error will be
49reported by any applicable pipelines. In the case of a change to a
James E. Blairac3c7ae2017-07-31 09:01:08 -070050config-project, the new configuration is parsed and examined for
James E. Blair1de8d402017-05-07 17:08:04 -070051errors, but the new configuration is not used in testing the change.
James E. Blairac3c7ae2017-07-31 09:01:08 -070052This is because configuration in config-projects is able to access
James E. Blair1de8d402017-05-07 17:08:04 -070053elevated privileges and should always be reviewed before being merged.
54
55As soon as a change containing a Zuul configuration change merges to
56any Zuul-managed repository, the new configuration takes effect
57immediately.
58
59Configuration Items
60-------------------
61
James E. Blairac3c7ae2017-07-31 09:01:08 -070062The ``zuul.yaml`` and ``.zuul.yaml`` configuration files are
James E. Blair1de8d402017-05-07 17:08:04 -070063YAML-formatted and are structured as a series of items, each of which
64is described below.
65
James E. Blairac3c7ae2017-07-31 09:01:08 -070066In the case of a ``zuul.d`` directory, Zuul recurses the directory and
67extends the configuration using all the .yaml files in the sorted path
68order. For example, to keep job's variants in a separate file, it
69needs to be loaded after the main entries, for example using number
70prefixes in file's names::
Tristan Cacqueray4a015832017-07-11 05:18:14 +000071
72* zuul.d/pipelines.yaml
73* zuul.d/projects.yaml
74* zuul.d/01_jobs.yaml
75* zuul.d/02_jobs-variants.yaml
76
James E. Blair1de8d402017-05-07 17:08:04 -070077.. _pipeline:
78
79Pipeline
80~~~~~~~~
81
82A pipeline describes a workflow operation in Zuul. It associates jobs
83for a given project with triggering and reporting events.
84
85Its flexible configuration allows for characterizing any number of
86workflows, and by specifying each as a named configuration, makes it
87easy to apply similar workflow operations to projects or groups of
88projects.
89
90By way of example, one of the primary uses of Zuul is to perform
James E. Blairac3c7ae2017-07-31 09:01:08 -070091project gating. To do so, one can create a :term:`gate` pipeline
92which tells Zuul that when a certain event (such as approval by a code
James E. Blair1de8d402017-05-07 17:08:04 -070093reviewer) occurs, the corresponding change or pull request should be
94enqueued into the pipeline. When that happens, the jobs which have
James E. Blairac3c7ae2017-07-31 09:01:08 -070095been configured to run for that project in the gate pipeline are run,
96and when they complete, the pipeline reports the results to the user.
James E. Blair1de8d402017-05-07 17:08:04 -070097
James E. Blairac3c7ae2017-07-31 09:01:08 -070098Pipeline configuration items may only appear in :term:`config-projects
99<config-project>`.
James E. Blair1de8d402017-05-07 17:08:04 -0700100
101Generally, a Zuul administrator would define a small number of
102pipelines which represent the workflow processes used in their
103environment. Each project can then be added to the available
104pipelines as appropriate.
105
James E. Blairac3c7ae2017-07-31 09:01:08 -0700106Here is an example :term:`check` pipeline, which runs whenever a new
James E. Blair1de8d402017-05-07 17:08:04 -0700107patchset is created in Gerrit. If the associated jobs all report
James E. Blairac3c7ae2017-07-31 09:01:08 -0700108success, the pipeline reports back to Gerrit with ``Verified`` vote of
James E. Blair32c52482017-07-29 07:49:03 -0700109+1, or if at least one of them fails, a -1:
James E. Blair1de8d402017-05-07 17:08:04 -0700110
James E. Blair32c52482017-07-29 07:49:03 -0700111.. code-block:: yaml
112
113 - pipeline:
114 name: check
115 manager: independent
116 trigger:
117 my_gerrit:
118 - event: patchset-created
119 success:
120 my_gerrit:
121 Verified: 1
122 failure:
123 my_gerrit
124 Verified: -1
James E. Blair1de8d402017-05-07 17:08:04 -0700125
James E. Blaireff5a9d2017-06-20 00:00:37 -0700126.. TODO: See TODO for more annotated examples of common pipeline configurations.
James E. Blair1de8d402017-05-07 17:08:04 -0700127
James E. Blair94375912017-07-28 17:20:27 -0700128.. attr:: pipeline
James E. Blair7145c582017-07-26 13:30:39 -0700129
James E. Blair9fd98ab2017-07-26 14:15:26 -0700130 The attributes available on a pipeline are as follows (all are
131 optional unless otherwise specified):
James E. Blair1de8d402017-05-07 17:08:04 -0700132
James E. Blair94375912017-07-28 17:20:27 -0700133 .. attr:: name
James E. Blair9fd98ab2017-07-26 14:15:26 -0700134 :required:
James E. Blair1de8d402017-05-07 17:08:04 -0700135
James E. Blair9fd98ab2017-07-26 14:15:26 -0700136 This is used later in the project definition to indicate what jobs
137 should be run for events in the pipeline.
James E. Blair1de8d402017-05-07 17:08:04 -0700138
James E. Blair94375912017-07-28 17:20:27 -0700139 .. attr:: manager
James E. Blair9fd98ab2017-07-26 14:15:26 -0700140 :required:
James E. Blaireff5a9d2017-06-20 00:00:37 -0700141
James E. Blair9fd98ab2017-07-26 14:15:26 -0700142 There are currently two schemes for managing pipelines:
James E. Blair1de8d402017-05-07 17:08:04 -0700143
James E. Blair32c52482017-07-29 07:49:03 -0700144 .. value:: independent
James E. Blaireff5a9d2017-06-20 00:00:37 -0700145
James E. Blair9fd98ab2017-07-26 14:15:26 -0700146 Every event in this pipeline should be treated as independent
147 of other events in the pipeline. This is appropriate when
148 the order of events in the pipeline doesn't matter because
149 the results of the actions this pipeline performs can not
150 affect other events in the pipeline. For example, when a
151 change is first uploaded for review, you may want to run
152 tests on that change to provide early feedback to reviewers.
153 At the end of the tests, the change is not going to be
154 merged, so it is safe to run these tests in parallel without
155 regard to any other changes in the pipeline. They are
156 independent.
James E. Blair1de8d402017-05-07 17:08:04 -0700157
James E. Blair9fd98ab2017-07-26 14:15:26 -0700158 Another type of pipeline that is independent is a post-merge
159 pipeline. In that case, the changes have already merged, so
160 the results can not affect any other events in the pipeline.
James E. Blair1761e862017-07-25 16:15:47 -0700161
James E. Blair32c52482017-07-29 07:49:03 -0700162 .. value:: dependent
James E. Blair1761e862017-07-25 16:15:47 -0700163
James E. Blair9fd98ab2017-07-26 14:15:26 -0700164 The dependent pipeline manager is designed for gating. It
165 ensures that every change is tested exactly as it is going to
166 be merged into the repository. An ideal gating system would
167 test one change at a time, applied to the tip of the
168 repository, and only if that change passed tests would it be
169 merged. Then the next change in line would be tested the
170 same way. In order to achieve parallel testing of changes,
171 the dependent pipeline manager performs speculative execution
172 on changes. It orders changes based on their entry into the
173 pipeline. It begins testing all changes in parallel,
174 assuming that each change ahead in the pipeline will pass its
175 tests. If they all succeed, all the changes can be tested
176 and merged in parallel. If a change near the front of the
177 pipeline fails its tests, each change behind it ignores
178 whatever tests have been completed and are tested again
179 without the change in front. This way gate tests may run in
180 parallel but still be tested correctly, exactly as they will
181 appear in the repository when merged.
James E. Blair1761e862017-07-25 16:15:47 -0700182
James E. Blair9fd98ab2017-07-26 14:15:26 -0700183 For more detail on the theory and operation of Zuul's
184 dependent pipeline manager, see: :doc:`gating`.
James E. Blair1de8d402017-05-07 17:08:04 -0700185
James E. Blair94375912017-07-28 17:20:27 -0700186 .. attr:: allow-secrets
James E. Blair88d84242017-07-31 12:05:16 -0700187 :default: false
James E. Blairf17aa9c2017-07-05 13:21:23 -0700188
James E. Blair9fd98ab2017-07-26 14:15:26 -0700189 This is a boolean which can be used to prevent jobs which
190 require secrets from running in this pipeline. Some pipelines
191 run on proposed changes and therefore execute code which has not
192 yet been reviewed. In such a case, allowing a job to use a
193 secret could result in that secret being exposed. The default
James E. Blair88d84242017-07-31 12:05:16 -0700194 is ``false``, meaning that in order to run jobs with secrets,
195 this must be explicitly enabled on each Pipeline where that is
196 safe.
James E. Blairf17aa9c2017-07-05 13:21:23 -0700197
James E. Blair9fd98ab2017-07-26 14:15:26 -0700198 For more information, see :ref:`secret`.
James E. Blair1de8d402017-05-07 17:08:04 -0700199
James E. Blair94375912017-07-28 17:20:27 -0700200 .. attr:: description
James E. Blair1de8d402017-05-07 17:08:04 -0700201
James E. Blair9fd98ab2017-07-26 14:15:26 -0700202 This field may be used to provide a textual description of the
203 pipeline. It may appear in the status page or in documentation.
James E. Blair1de8d402017-05-07 17:08:04 -0700204
James E. Blair94375912017-07-28 17:20:27 -0700205 .. attr:: success-message
James E. Blair88d84242017-07-31 12:05:16 -0700206 :default: Build successful.
James E. Blair1de8d402017-05-07 17:08:04 -0700207
James E. Blair9fd98ab2017-07-26 14:15:26 -0700208 The introductory text in reports when all the voting jobs are
James E. Blair88d84242017-07-31 12:05:16 -0700209 successful.
James E. Blair1de8d402017-05-07 17:08:04 -0700210
James E. Blair94375912017-07-28 17:20:27 -0700211 .. attr:: failure-message
James E. Blair88d84242017-07-31 12:05:16 -0700212 :default: Build failed.
James E. Blair1de8d402017-05-07 17:08:04 -0700213
James E. Blair9fd98ab2017-07-26 14:15:26 -0700214 The introductory text in reports when at least one voting job
James E. Blair88d84242017-07-31 12:05:16 -0700215 fails.
James E. Blair1de8d402017-05-07 17:08:04 -0700216
James E. Blair94375912017-07-28 17:20:27 -0700217 .. attr:: merge-failure-message
James E. Blair88d84242017-07-31 12:05:16 -0700218 :default: Merge failed.
James E. Blair1de8d402017-05-07 17:08:04 -0700219
James E. Blair9fd98ab2017-07-26 14:15:26 -0700220 The introductory text in the message reported when a change
221 fails to merge with the current state of the repository.
222 Defaults to "Merge failed."
James E. Blair1de8d402017-05-07 17:08:04 -0700223
James E. Blair94375912017-07-28 17:20:27 -0700224 .. attr:: footer-message
James E. Blair1de8d402017-05-07 17:08:04 -0700225
James E. Blair9fd98ab2017-07-26 14:15:26 -0700226 Supplies additional information after test results. Useful for
227 adding information about the CI system such as debugging and
228 contact details.
James E. Blair1de8d402017-05-07 17:08:04 -0700229
James E. Blair94375912017-07-28 17:20:27 -0700230 .. attr:: trigger
James E. Blair1de8d402017-05-07 17:08:04 -0700231
James E. Blair9fd98ab2017-07-26 14:15:26 -0700232 At least one trigger source must be supplied for each pipeline.
233 Triggers are not exclusive -- matching events may be placed in
234 multiple pipelines, and they will behave independently in each
235 of the pipelines they match.
James E. Blair1de8d402017-05-07 17:08:04 -0700236
James E. Blair9fd98ab2017-07-26 14:15:26 -0700237 Triggers are loaded from their connection name. The driver type
238 of the connection will dictate which options are available. See
239 :ref:`drivers`.
James E. Blair1de8d402017-05-07 17:08:04 -0700240
James E. Blair94375912017-07-28 17:20:27 -0700241 .. attr:: require
James E. Blair1de8d402017-05-07 17:08:04 -0700242
James E. Blairac3c7ae2017-07-31 09:01:08 -0700243 If this section is present, it establishes prerequisites for
James E. Blair9fd98ab2017-07-26 14:15:26 -0700244 any kind of item entering the Pipeline. Regardless of how the
245 item is to be enqueued (via any trigger or automatic dependency
246 resolution), the conditions specified here must be met or the
James E. Blaird134c6d2017-07-26 16:09:34 -0700247 item will not be enqueued. These requirements may vary
248 depending on the source of the item being enqueued.
James E. Blair1de8d402017-05-07 17:08:04 -0700249
James E. Blaird134c6d2017-07-26 16:09:34 -0700250 Requirements are loaded from their connection name. The driver
251 type of the connection will dictate which options are available.
252 See :ref:`drivers`.
James E. Blair1de8d402017-05-07 17:08:04 -0700253
James E. Blair94375912017-07-28 17:20:27 -0700254 .. attr:: reject
James E. Blair1de8d402017-05-07 17:08:04 -0700255
James E. Blairac3c7ae2017-07-31 09:01:08 -0700256 If this section is present, it establishes prerequisites that
James E. Blair9fd98ab2017-07-26 14:15:26 -0700257 can block an item from being enqueued. It can be considered a
James E. Blairac3c7ae2017-07-31 09:01:08 -0700258 negative version of :attr:`pipeline.require`.
James E. Blair1de8d402017-05-07 17:08:04 -0700259
James E. Blaird134c6d2017-07-26 16:09:34 -0700260 Requirements are loaded from their connection name. The driver
261 type of the connection will dictate which options are available.
262 See :ref:`drivers`.
James E. Blair9fd98ab2017-07-26 14:15:26 -0700263
James E. Blair94375912017-07-28 17:20:27 -0700264 .. attr:: dequeue-on-new-patchset
James E. Blair88d84242017-07-31 12:05:16 -0700265 :default: true
James E. Blair9fd98ab2017-07-26 14:15:26 -0700266
267 Normally, if a new patchset is uploaded to a change that is in a
268 pipeline, the existing entry in the pipeline will be removed
269 (with jobs canceled and any dependent changes that can no longer
270 merge as well. To suppress this behavior (and allow jobs to
James E. Blair88d84242017-07-31 12:05:16 -0700271 continue running), set this to ``false``.
James E. Blair9fd98ab2017-07-26 14:15:26 -0700272
James E. Blair94375912017-07-28 17:20:27 -0700273 .. attr:: ignore-dependencies
James E. Blair88d84242017-07-31 12:05:16 -0700274 :default: false
James E. Blair9fd98ab2017-07-26 14:15:26 -0700275
276 In any kind of pipeline (dependent or independent), Zuul will
277 attempt to enqueue all dependencies ahead of the current change
278 so that they are tested together (independent pipelines report
279 the results of each change regardless of the results of changes
280 ahead). To ignore dependencies completely in an independent
281 pipeline, set this to ``true``. This option is ignored by
James E. Blair88d84242017-07-31 12:05:16 -0700282 dependent pipelines.
James E. Blair9fd98ab2017-07-26 14:15:26 -0700283
James E. Blair94375912017-07-28 17:20:27 -0700284 .. attr:: precedence
James E. Blair88d84242017-07-31 12:05:16 -0700285 :default: normal
James E. Blair9fd98ab2017-07-26 14:15:26 -0700286
287 Indicates how the build scheduler should prioritize jobs for
288 different pipelines. Each pipeline may have one precedence,
289 jobs for pipelines with a higher precedence will be run before
290 ones with lower. The value should be one of ``high``,
291 ``normal``, or ``low``. Default: ``normal``.
292
James E. Blairac3c7ae2017-07-31 09:01:08 -0700293 .. _reporters:
294
295 The following options configure :term:`reporters <reporter>`.
296 Reporters are complementary to triggers; where a trigger is an
297 event on a connection which causes Zuul to enqueue an item, a
298 reporter is the action performed on a connection when an item is
299 dequeued after its jobs complete. The actual syntax for a reporter
300 is defined by the driver which implements it. See :ref:`drivers`
301 for more information.
James E. Blair9fd98ab2017-07-26 14:15:26 -0700302
James E. Blair94375912017-07-28 17:20:27 -0700303 .. attr:: success
James E. Blair9fd98ab2017-07-26 14:15:26 -0700304
305 Describes where Zuul should report to if all the jobs complete
306 successfully. This section is optional; if it is omitted, Zuul
307 will run jobs and do nothing on success -- it will not report at
James E. Blairac3c7ae2017-07-31 09:01:08 -0700308 all. If the section is present, the listed :term:`reporters
309 <reporter>` will be asked to report on the jobs. The reporters
310 are listed by their connection name. The options available
311 depend on the driver for the supplied connection.
James E. Blair9fd98ab2017-07-26 14:15:26 -0700312
James E. Blair94375912017-07-28 17:20:27 -0700313 .. attr:: failure
James E. Blair9fd98ab2017-07-26 14:15:26 -0700314
315 These reporters describe what Zuul should do if at least one job
316 fails.
317
James E. Blair94375912017-07-28 17:20:27 -0700318 .. attr:: merge-failure
James E. Blair9fd98ab2017-07-26 14:15:26 -0700319
320 These reporters describe what Zuul should do if it is unable to
321 merge in the patchset. If no merge-failure reporters are listed
322 then the ``failure`` reporters will be used to notify of
323 unsuccessful merges.
324
James E. Blair94375912017-07-28 17:20:27 -0700325 .. attr:: start
James E. Blair9fd98ab2017-07-26 14:15:26 -0700326
327 These reporters describe what Zuul should do when a change is
328 added to the pipeline. This can be used, for example, to reset
329 a previously reported result.
330
James E. Blair94375912017-07-28 17:20:27 -0700331 .. attr:: disabled
James E. Blair9fd98ab2017-07-26 14:15:26 -0700332
333 These reporters describe what Zuul should do when a pipeline is
334 disabled. See ``disable-after-consecutive-failures``.
335
336 The following options can be used to alter Zuul's behavior to
337 mitigate situations in which jobs are failing frequently (perhaps
338 due to a problem with an external dependency, or unusually high
339 non-deterministic test failures).
340
James E. Blair94375912017-07-28 17:20:27 -0700341 .. attr:: disable-after-consecutive-failures
James E. Blair9fd98ab2017-07-26 14:15:26 -0700342
James E. Blairac3c7ae2017-07-31 09:01:08 -0700343 If set, a pipeline can enter a *disabled* state if too many
James E. Blair9fd98ab2017-07-26 14:15:26 -0700344 changes in a row fail. When this value is exceeded the pipeline
James E. Blairac3c7ae2017-07-31 09:01:08 -0700345 will stop reporting to any of the **success**, **failure** or
346 **merge-failure** reporters and instead only report to the
347 **disabled** reporters. (No **start** reports are made when a
James E. Blair9fd98ab2017-07-26 14:15:26 -0700348 pipeline is disabled).
349
James E. Blair94375912017-07-28 17:20:27 -0700350 .. attr:: window
James E. Blair88d84242017-07-31 12:05:16 -0700351 :default: 20
James E. Blair9fd98ab2017-07-26 14:15:26 -0700352
353 Dependent pipeline managers only. Zuul can rate limit dependent
354 pipelines in a manner similar to TCP flow control. Jobs are
355 only started for items in the queue if they are within the
356 actionable window for the pipeline. The initial length of this
357 window is configurable with this value. The value given should
358 be a positive integer value. A value of ``0`` disables rate
James E. Blairac3c7ae2017-07-31 09:01:08 -0700359 limiting on the :value:`dependent pipeline manager
James E. Blair88d84242017-07-31 12:05:16 -0700360 <pipeline.manager.dependent>`.
James E. Blair9fd98ab2017-07-26 14:15:26 -0700361
James E. Blair94375912017-07-28 17:20:27 -0700362 .. attr:: window-floor
James E. Blair88d84242017-07-31 12:05:16 -0700363 :default: 3
James E. Blair9fd98ab2017-07-26 14:15:26 -0700364
365 Dependent pipeline managers only. This is the minimum value for
366 the window described above. Should be a positive non zero
James E. Blair88d84242017-07-31 12:05:16 -0700367 integer value.
James E. Blair9fd98ab2017-07-26 14:15:26 -0700368
James E. Blair94375912017-07-28 17:20:27 -0700369 .. attr:: window-increase-type
James E. Blair88d84242017-07-31 12:05:16 -0700370 :default: linear
James E. Blair9fd98ab2017-07-26 14:15:26 -0700371
372 Dependent pipeline managers only. This value describes how the
James E. Blairac3c7ae2017-07-31 09:01:08 -0700373 window should grow when changes are successfully merged by zuul.
374
375 .. value:: linear
376
377 Indicates that **window-increase-factor** should be added to
James E. Blair88d84242017-07-31 12:05:16 -0700378 the previous window value.
James E. Blairac3c7ae2017-07-31 09:01:08 -0700379
380 .. value:: exponential
381
382 Indicates that **window-increase-factor** should be
383 multiplied against the previous window value and the result
384 will become the window size.
James E. Blair9fd98ab2017-07-26 14:15:26 -0700385
James E. Blair94375912017-07-28 17:20:27 -0700386 .. attr:: window-increase-factor
James E. Blair88d84242017-07-31 12:05:16 -0700387 :default: 1
James E. Blair9fd98ab2017-07-26 14:15:26 -0700388
389 Dependent pipeline managers only. The value to be added or
390 multiplied against the previous window value to determine the
James E. Blair88d84242017-07-31 12:05:16 -0700391 new window after successful change merges.
James E. Blair9fd98ab2017-07-26 14:15:26 -0700392
James E. Blair94375912017-07-28 17:20:27 -0700393 .. attr:: window-decrease-type
James E. Blair88d84242017-07-31 12:05:16 -0700394 :default: exponential
James E. Blair9fd98ab2017-07-26 14:15:26 -0700395
396 Dependent pipeline managers only. This value describes how the
397 window should shrink when changes are not able to be merged by
James E. Blairac3c7ae2017-07-31 09:01:08 -0700398 Zuul.
399
400 .. value:: linear
401
402 Indicates that **window-decrease-factor** should be
403 subtracted from the previous window value.
404
405 .. value:: exponential
406
407 Indicates that **window-decrease-factor** should be divided
408 against the previous window value and the result will become
James E. Blair88d84242017-07-31 12:05:16 -0700409 the window size.
James E. Blair9fd98ab2017-07-26 14:15:26 -0700410
James E. Blair94375912017-07-28 17:20:27 -0700411 .. attr:: window-decrease-factor
James E. Blair88d84242017-07-31 12:05:16 -0700412 :default: 2
James E. Blair9fd98ab2017-07-26 14:15:26 -0700413
James E. Blairac3c7ae2017-07-31 09:01:08 -0700414 :value:`Dependent pipeline managers
415 <pipeline.manager.dependent>` only. The value to be subtracted
416 or divided against the previous window value to determine the
James E. Blair88d84242017-07-31 12:05:16 -0700417 new window after unsuccessful change merges.
James E. Blair1de8d402017-05-07 17:08:04 -0700418
419
420.. _job:
421
422Job
423~~~
424
425A job is a unit of work performed by Zuul on an item enqueued into a
426pipeline. Items may run any number of jobs (which may depend on each
427other). Each job is an invocation of an Ansible playbook with a
428specific inventory of hosts. The actual tasks that are run by the job
429appear in the playbook for that job while the attributes that appear in the
430Zuul configuration specify information about when, where, and how the
431job should be run.
432
433Jobs in Zuul support inheritance. Any job may specify a single parent
434job, and any attributes not set on the child job are collected from
435the parent job. In this way, a configuration structure may be built
436starting with very basic jobs which describe characteristics that all
437jobs on the system should have, progressing through stages of
438specialization before arriving at a particular job. A job may inherit
439from any other job in any project (however, if the other job is marked
James E. Blairac3c7ae2017-07-31 09:01:08 -0700440as ``final``, some attributes may not be overidden).
James E. Blair1de8d402017-05-07 17:08:04 -0700441
442Jobs also support a concept called variance. The first time a job
443definition appears is called the reference definition of the job.
444Subsequent job definitions with the same name are called variants.
445These may have different selection criteria which indicate to Zuul
446that, for instance, the job should behave differently on a different
447git branch. Unlike inheritance, all job variants must be defined in
448the same project.
449
450When Zuul decides to run a job, it performs a process known as
451freezing the job. Because any number of job variants may be
452applicable, Zuul collects all of the matching variants and applies
453them in the order they appeared in the configuration. The resulting
454frozen job is built from attributes gathered from all of the
455matching variants. In this way, exactly what is run is dependent on
456the pipeline, project, branch, and content of the item.
457
458In addition to the job's main playbook, each job may specify one or
459more pre- and post-playbooks. These are run, in order, before and
460after (respectively) the main playbook. They may be used to set up
461and tear down resources needed by the main playbook. When combined
462with inheritance, they provide powerful tools for job construction. A
463job only has a single main playbook, and when inheriting from a
464parent, the child's main playbook overrides (or replaces) the
465parent's. However, the pre- and post-playbooks are appended and
466prepended in a nesting fashion. So if a parent job and child job both
467specified pre and post playbooks, the sequence of playbooks run would
468be:
469
470* parent pre-run playbook
471* child pre-run playbook
472* child playbook
473* child post-run playbook
474* parent post-run playbook
475
476Further inheritance would nest even deeper.
477
James E. Blair32c52482017-07-29 07:49:03 -0700478Here is an example of two job definitions:
James E. Blair1de8d402017-05-07 17:08:04 -0700479
James E. Blair32c52482017-07-29 07:49:03 -0700480.. code-block:: yaml
James E. Blair1de8d402017-05-07 17:08:04 -0700481
James E. Blair32c52482017-07-29 07:49:03 -0700482 - job:
483 name: base
484 pre-run: copy-git-repos
485 post-run: copy-logs
James E. Blair1de8d402017-05-07 17:08:04 -0700486
James E. Blair32c52482017-07-29 07:49:03 -0700487 - job:
488 name: run-tests
489 parent: base
490 nodes:
491 - name: test-node
492 image: fedora
James E. Blair1de8d402017-05-07 17:08:04 -0700493
James E. Blair32c52482017-07-29 07:49:03 -0700494.. attr:: job
James E. Blair1de8d402017-05-07 17:08:04 -0700495
James E. Blair32c52482017-07-29 07:49:03 -0700496 The following attributes are available on a job; all are optional
497 unless otherwise specified:
James E. Blair1de8d402017-05-07 17:08:04 -0700498
James E. Blair32c52482017-07-29 07:49:03 -0700499 .. attr:: name
500 :required:
James E. Blair1de8d402017-05-07 17:08:04 -0700501
James E. Blair32c52482017-07-29 07:49:03 -0700502 The name of the job. By default, Zuul looks for a playbook with
503 this name to use as the main playbook for the job. This name is
504 also referenced later in a project pipeline configuration.
James E. Blair1de8d402017-05-07 17:08:04 -0700505
James E. Blair32c52482017-07-29 07:49:03 -0700506 .. attr:: parent
James E. Blair1de8d402017-05-07 17:08:04 -0700507
James E. Blairac3c7ae2017-07-31 09:01:08 -0700508 Specifies a job to inherit from. The parent job can be defined
509 in this or any other project. Any attributes not specified on
James E. Blair32c52482017-07-29 07:49:03 -0700510 a job will be collected from its parent.
James E. Blair1de8d402017-05-07 17:08:04 -0700511
James E. Blair32c52482017-07-29 07:49:03 -0700512 .. attr:: description
James E. Blair1de8d402017-05-07 17:08:04 -0700513
James E. Blair32c52482017-07-29 07:49:03 -0700514 A textual description of the job. Not currently used directly
515 by Zuul, but it is used by the zuul-sphinx extension to Sphinx
516 to auto-document Zuul jobs (in which case it is interpreted as
517 ReStructuredText.
James E. Blair1de8d402017-05-07 17:08:04 -0700518
James E. Blair32c52482017-07-29 07:49:03 -0700519 .. attr:: success-message
James E. Blair88d84242017-07-31 12:05:16 -0700520 :default: SUCCESS
James E. Blair1de8d402017-05-07 17:08:04 -0700521
James E. Blairac3c7ae2017-07-31 09:01:08 -0700522 Normally when a job succeeds, the string ``SUCCESS`` is reported
James E. Blair32c52482017-07-29 07:49:03 -0700523 as the result for the job. If set, this option may be used to
James E. Blair88d84242017-07-31 12:05:16 -0700524 supply a different string.
James E. Blair1de8d402017-05-07 17:08:04 -0700525
James E. Blair32c52482017-07-29 07:49:03 -0700526 .. attr:: failure-message
James E. Blair88d84242017-07-31 12:05:16 -0700527 :default: FAILURE
James E. Blair1de8d402017-05-07 17:08:04 -0700528
James E. Blairac3c7ae2017-07-31 09:01:08 -0700529 Normally when a job fails, the string ``FAILURE`` is reported as
James E. Blair32c52482017-07-29 07:49:03 -0700530 the result for the job. If set, this option may be used to
James E. Blair88d84242017-07-31 12:05:16 -0700531 supply a different string.
James E. Blair1de8d402017-05-07 17:08:04 -0700532
James E. Blair32c52482017-07-29 07:49:03 -0700533 .. attr:: success-url
James E. Blair1de8d402017-05-07 17:08:04 -0700534
James E. Blair32c52482017-07-29 07:49:03 -0700535 When a job succeeds, this URL is reported along with the result.
536 If this value is not supplied, Zuul uses the content of the job
537 :ref:`return value <return_values>` **zuul.log_url**. This is
538 recommended as it allows the code which stores the URL to the
539 job artifacts to report exactly where they were stored. To
540 override this value, or if it is not set, supply an absolute URL
541 in this field. If a relative URL is supplied in this field, and
542 **zuul.log_url** is set, then the two will be combined to
543 produce the URL used for the report. This can be used to
544 specify that certain jobs should "deep link" into the stored job
James E. Blair88d84242017-07-31 12:05:16 -0700545 artifacts.
James E. Blair1de8d402017-05-07 17:08:04 -0700546
James E. Blair32c52482017-07-29 07:49:03 -0700547 .. attr:: failure-url
James E. Blair1de8d402017-05-07 17:08:04 -0700548
James E. Blair32c52482017-07-29 07:49:03 -0700549 When a job fails, this URL is reported along with the result.
550 Otherwise behaves the same as **success-url**.
James E. Blair1de8d402017-05-07 17:08:04 -0700551
James E. Blair32c52482017-07-29 07:49:03 -0700552 .. attr:: hold-following-changes
James E. Blair88d84242017-07-31 12:05:16 -0700553 :default: false
James E. Blair1de8d402017-05-07 17:08:04 -0700554
James E. Blair32c52482017-07-29 07:49:03 -0700555 In a dependent pipeline, this option may be used to indicate
556 that no jobs should start on any items which depend on the
557 current item until this job has completed successfully. This
558 may be used to conserve build resources, at the expense of
559 inhibiting the parallelization which speeds the processing of
James E. Blair88d84242017-07-31 12:05:16 -0700560 items in a dependent pipeline.
James E. Blair1de8d402017-05-07 17:08:04 -0700561
James E. Blair32c52482017-07-29 07:49:03 -0700562 .. attr:: voting
James E. Blair88d84242017-07-31 12:05:16 -0700563 :default: true
James E. Blair1de8d402017-05-07 17:08:04 -0700564
James E. Blair32c52482017-07-29 07:49:03 -0700565 Indicates whether the result of this job should be used in
James E. Blair88d84242017-07-31 12:05:16 -0700566 determining the overall result of the item.
James E. Blair1de8d402017-05-07 17:08:04 -0700567
James E. Blair32c52482017-07-29 07:49:03 -0700568 .. attr:: semaphore
James E. Blair1de8d402017-05-07 17:08:04 -0700569
James E. Blair32c52482017-07-29 07:49:03 -0700570 The name of a :ref:`semaphore` which should be acquired and
571 released when the job begins and ends. If the semaphore is at
572 maximum capacity, then Zuul will wait until it can be acquired
James E. Blair88d84242017-07-31 12:05:16 -0700573 before starting the job.
James E. Blair1de8d402017-05-07 17:08:04 -0700574
James E. Blair32c52482017-07-29 07:49:03 -0700575 .. attr:: tags
James E. Blair1de8d402017-05-07 17:08:04 -0700576
James E. Blair32c52482017-07-29 07:49:03 -0700577 Metadata about this job. Tags are units of information attached
578 to the job; they do not affect Zuul's behavior, but they can be
579 used within the job to characterize the job. For example, a job
580 which tests a certain subsystem could be tagged with the name of
581 that subsystem, and if the job's results are reported into a
582 database, then the results of all jobs affecting that subsystem
583 could be queried. This attribute is specified as a list of
584 strings, and when inheriting jobs or applying variants, tags
585 accumulate in a set, so the result is always a set of all the
586 tags from all the jobs and variants used in constructing the
James E. Blair88d84242017-07-31 12:05:16 -0700587 frozen job, with no duplication.
James E. Blair1de8d402017-05-07 17:08:04 -0700588
James E. Blair32c52482017-07-29 07:49:03 -0700589 .. attr:: branches
James E. Blair1de8d402017-05-07 17:08:04 -0700590
James E. Blair32c52482017-07-29 07:49:03 -0700591 A regular expression (or list of regular expressions) which
592 describe on what branches a job should run (or in the case of
593 variants: to alter the behavior of a job for a certain branch).
James E. Blair1de8d402017-05-07 17:08:04 -0700594
James E. Blair32c52482017-07-29 07:49:03 -0700595 If there is no job definition for a given job which matches the
596 branch of an item, then that job is not run for the item.
597 Otherwise, all of the job variants which match that branch (and
598 any other selection criteria) are used when freezing the job.
James E. Blair1de8d402017-05-07 17:08:04 -0700599
James E. Blair32c52482017-07-29 07:49:03 -0700600 This example illustrates a job called *run-tests* which uses a
601 nodeset based on the current release of an operating system to
602 perform its tests, except when testing changes to the stable/2.0
603 branch, in which case it uses an older release:
James E. Blair1de8d402017-05-07 17:08:04 -0700604
James E. Blair32c52482017-07-29 07:49:03 -0700605 .. code-block:: yaml
James E. Blair1de8d402017-05-07 17:08:04 -0700606
James E. Blair32c52482017-07-29 07:49:03 -0700607 - job:
608 name: run-tests
609 nodes: current-release
James E. Blair1de8d402017-05-07 17:08:04 -0700610
James E. Blair32c52482017-07-29 07:49:03 -0700611 - job:
612 name: run-tests
613 branch: stable/2.0
614 nodes: old-release
James E. Blair1de8d402017-05-07 17:08:04 -0700615
James E. Blairac3c7ae2017-07-31 09:01:08 -0700616 In some cases, Zuul uses an implied value for the branch
617 specifier if none is supplied:
James E. Blair1de8d402017-05-07 17:08:04 -0700618
James E. Blairac3c7ae2017-07-31 09:01:08 -0700619 * For a job definition in a :term:`config-project`, no implied
620 branch specifier is used. If no branch specifier appears, the
621 job applies to all branches.
James E. Blair1de8d402017-05-07 17:08:04 -0700622
James E. Blairac3c7ae2017-07-31 09:01:08 -0700623 * In the case of an :term:`untrusted-project`, no implied branch
624 specifier is applied to the reference definition of a job.
625 That is to say, that if the first appearance of the job
626 definition appears without a branch specifier, then it will
627 apply to all branches. Note that when collecting its
628 configuration, Zuul reads the ``master`` branch of a given
629 project first, then other branches in alphabetical order.
James E. Blair1de8d402017-05-07 17:08:04 -0700630
James E. Blairac3c7ae2017-07-31 09:01:08 -0700631 * Any further job variants other than the reference definition
632 in an untrusted-project will, if they do not have a branch
633 specifier, will have an implied branch specifier for the
634 current branch applied.
James E. Blair1de8d402017-05-07 17:08:04 -0700635
James E. Blair32c52482017-07-29 07:49:03 -0700636 This allows for the very simple and expected workflow where if a
James E. Blairac3c7ae2017-07-31 09:01:08 -0700637 project defines a job on the ``master`` branch with no branch
638 specifier, and then creates a new branch based on ``master``,
639 any changes to that job definition within the new branch only
640 affect that branch.
James E. Blair1de8d402017-05-07 17:08:04 -0700641
James E. Blair32c52482017-07-29 07:49:03 -0700642 .. attr:: files
Tobias Henkel2aade262017-07-12 16:09:06 +0200643
James E. Blair32c52482017-07-29 07:49:03 -0700644 This attribute indicates that the job should only run on changes
645 where the specified files are modified. This is a regular
James E. Blair88d84242017-07-31 12:05:16 -0700646 expression or list of regular expressions.
James E. Blair1de8d402017-05-07 17:08:04 -0700647
James E. Blair32c52482017-07-29 07:49:03 -0700648 .. attr:: irrelevant-files
James E. Blair74a82cf2017-07-12 17:23:08 -0700649
James E. Blairac3c7ae2017-07-31 09:01:08 -0700650 This is a negative complement of **files**. It indicates that
651 the job should run unless *all* of the files changed match this
652 list. In other words, if the regular expression ``docs/.*`` is
James E. Blair32c52482017-07-29 07:49:03 -0700653 supplied, then this job will not run if the only files changed
654 are in the docs directory. A regular expression or list of
James E. Blair88d84242017-07-31 12:05:16 -0700655 regular expressions.
James E. Blair1de8d402017-05-07 17:08:04 -0700656
James E. Blair32c52482017-07-29 07:49:03 -0700657 .. attr:: auth
James E. Blair1de8d402017-05-07 17:08:04 -0700658
James E. Blair32c52482017-07-29 07:49:03 -0700659 Authentication information to be made available to the job.
660 This is a dictionary with two potential keys:
James E. Blairbb94dfa2017-07-11 07:45:19 -0700661
James E. Blair32c52482017-07-29 07:49:03 -0700662 .. attr:: inherit
James E. Blair88d84242017-07-31 12:05:16 -0700663 :default: false
James E. Blaireff5a9d2017-06-20 00:00:37 -0700664
James E. Blair32c52482017-07-29 07:49:03 -0700665 A boolean indicating that the authentication information
666 referenced by this job should be able to be inherited by
667 child jobs. Normally when a job inherits from another job,
668 the auth section is not included. This permits jobs to
669 inherit the same basic structure and playbook, but ensures
670 that secret information is unable to be exposed by a child
671 job which may alter the job's behavior. If it is safe for
672 the contents of the authentication section to be used by
James E. Blair88d84242017-07-31 12:05:16 -0700673 child jobs, set this to ``true``.
James E. Blair1de8d402017-05-07 17:08:04 -0700674
James E. Blair32c52482017-07-29 07:49:03 -0700675 .. attr:: secrets
James E. Blair1de8d402017-05-07 17:08:04 -0700676
James E. Blair32c52482017-07-29 07:49:03 -0700677 A list of secrets which may be used by the job. A
678 :ref:`secret` is a named collection of private information
679 defined separately in the configuration. The secrets that
680 appear here must be defined in the same project as this job
681 definition.
James E. Blair1de8d402017-05-07 17:08:04 -0700682
James E. Blair32c52482017-07-29 07:49:03 -0700683 In the future, other types of authentication information may
684 be added.
James E. Blair1de8d402017-05-07 17:08:04 -0700685
James E. Blair32c52482017-07-29 07:49:03 -0700686 .. attr:: nodes
James E. Blair1de8d402017-05-07 17:08:04 -0700687
James E. Blair32c52482017-07-29 07:49:03 -0700688 A list of nodes which should be supplied to the job. This
689 parameter may be supplied either as a string, in which case it
690 references a :ref:`nodeset` definition which appears elsewhere
691 in the configuration, or a list, in which case it is interpreted
692 in the same way as a Nodeset definition (in essence, it is an
693 anonymous Node definition unique to this job). See the
694 :ref:`nodeset` reference for the syntax to use in that case.
James E. Blair1de8d402017-05-07 17:08:04 -0700695
James E. Blair32c52482017-07-29 07:49:03 -0700696 If a job has an empty or no node definition, it will still run
697 and may be able to perform actions on the Zuul executor.
James E. Blair1de8d402017-05-07 17:08:04 -0700698
James E. Blair32c52482017-07-29 07:49:03 -0700699 .. attr:: override-branch
James E. Blair1de8d402017-05-07 17:08:04 -0700700
James E. Blair32c52482017-07-29 07:49:03 -0700701 When Zuul runs jobs for a proposed change, it normally checks
702 out the branch associated with that change on every project
703 present in the job. If jobs are running on a ref (such as a
704 branch tip or tag), then that ref is normally checked out. This
705 attribute is used to override that behavior and indicate that
706 this job should, regardless of the branch for the queue item,
707 use the indicated branch instead. This can be used, for
708 example, to run a previous version of the software (from a
709 stable maintenance branch) under test even if the change being
710 tested applies to a different branch (this is only likely to be
711 useful if there is some cross-branch interaction with some
712 component of the system being tested). See also the
James E. Blairac3c7ae2017-07-31 09:01:08 -0700713 project-specific :attr:`job.required-projects.override-branch`
714 attribute to apply this behavior to a subset of a job's
715 projects.
James E. Blair1de8d402017-05-07 17:08:04 -0700716
James E. Blair32c52482017-07-29 07:49:03 -0700717 .. attr:: timeout
James E. Blair1de8d402017-05-07 17:08:04 -0700718
James E. Blair32c52482017-07-29 07:49:03 -0700719 The time in minutes that the job should be allowed to run before
720 it is automatically aborted and failure is reported. If no
721 timeout is supplied, the job may run indefinitely. Supplying a
722 timeout is highly recommended.
723
724 .. attr:: attempts
James E. Blair88d84242017-07-31 12:05:16 -0700725 :default: 3
James E. Blair32c52482017-07-29 07:49:03 -0700726
727 When Zuul encounters an error running a job's pre-run playbook,
728 Zuul will stop and restart the job. Errors during the main or
729 post-run -playbook phase of a job are not affected by this
730 parameter (they are reported immediately). This parameter
731 controls the number of attempts to make before an error is
James E. Blair88d84242017-07-31 12:05:16 -0700732 reported.
James E. Blair32c52482017-07-29 07:49:03 -0700733
734 .. attr:: pre-run
735
736 The name of a playbook or list of playbooks without file
737 extension to run before the main body of a job. The full path
738 to the playbook in the repo where the job is defined is
739 expected.
740
741 When a job inherits from a parent, the child's pre-run playbooks
742 are run after the parent's. See :ref:`job` for more
743 information.
744
745 .. attr:: post-run
746
747 The name of a playbook or list of playbooks without file
748 extension to run after the main body of a job. The full path to
749 the playbook in the repo where the job is defined is expected.
750
751 When a job inherits from a parent, the child's post-run
752 playbooks are run before the parent's. See :ref:`job` for more
753 information.
754
755 .. attr:: run
756
757 The name of the main playbook for this job. This parameter is
758 not normally necessary, as it defaults to a playbook with the
James E. Blairac3c7ae2017-07-31 09:01:08 -0700759 same name as the job inside of the ``playbooks/`` directory
760 (e.g., the ``foo`` job would default to ``playbooks/foo``.
761 However, if a playbook with a different name is needed, it can
762 be specified here. The file extension is not required, but the
763 full path within the repo is. When a child inherits from a
764 parent, a playbook with the name of the child job is implicitly
765 searched first, before falling back on the playbook used by the
766 parent job (unless the child job specifies a ``run`` attribute,
767 in which case that value is used). Example:
James E. Blair32c52482017-07-29 07:49:03 -0700768
769 .. code-block:: yaml
770
771 run: playbooks/<name of the job>
772
773 .. attr:: roles
774
775 A list of Ansible roles to prepare for the job. Because a job
776 runs an Ansible playbook, any roles which are used by the job
777 must be prepared and installed by Zuul before the job begins.
778 This value is a list of dictionaries, each of which indicates
779 one of two types of roles: a Galaxy role, which is simply a role
780 that is installed from Ansible Galaxy, or a Zuul role, which is
781 a role provided by a project managed by Zuul. Zuul roles are
782 able to benefit from speculative merging and cross-project
783 dependencies when used by playbooks in untrusted projects.
784 Roles are added to the Ansible role path in the order they
785 appear on the job -- roles earlier in the list will take
786 precedence over those which follow.
787
788 In the case of job inheritance or variance, the roles used for
789 each of the playbooks run by the job will be only those which
790 were defined along with that playbook. If a child job inherits
791 from a parent which defines a pre and post playbook, then the
792 pre and post playbooks it inherits from the parent job will run
793 only with the roles that were defined on the parent. If the
794 child adds its own pre and post playbooks, then any roles added
795 by the child will be available to the child's playbooks. This
796 is so that a job which inherits from a parent does not
James E. Blairac3c7ae2017-07-31 09:01:08 -0700797 inadvertently alter the behavior of the parent's playbooks by
James E. Blair32c52482017-07-29 07:49:03 -0700798 the addition of conflicting roles. Roles added by a child will
799 appear before those it inherits from its parent.
800
801 A project which supplies a role may be structured in one of two
802 configurations: a bare role (in which the role exists at the
803 root of the project), or a contained role (in which the role
James E. Blairac3c7ae2017-07-31 09:01:08 -0700804 exists within the ``roles/`` directory of the project, perhaps
James E. Blair32c52482017-07-29 07:49:03 -0700805 along with other roles). In the case of a contained role, the
James E. Blairac3c7ae2017-07-31 09:01:08 -0700806 ``roles/`` directory of the project is added to the role search
James E. Blair32c52482017-07-29 07:49:03 -0700807 path. In the case of a bare role, the project itself is added
808 to the role search path. In case the name of the project is not
809 the name under which the role should be installed (and therefore
James E. Blairac3c7ae2017-07-31 09:01:08 -0700810 referenced from Ansible), the ``name`` attribute may be used to
James E. Blair32c52482017-07-29 07:49:03 -0700811 specify an alternate.
812
813 A job automatically has the project in which it is defined added
814 to the roles path if that project appears to contain a role or
James E. Blairac3c7ae2017-07-31 09:01:08 -0700815 ``roles/`` directory. By default, the project is added to the
James E. Blair32c52482017-07-29 07:49:03 -0700816 path under its own name, however, that may be changed by
817 explicitly listing the project in the roles list in the usual
818 way.
819
820 .. note:: Galaxy roles are not yet implemented.
821
822 .. attr:: galaxy
823
824 The name of the role in Ansible Galaxy. If this attribute is
825 supplied, Zuul will search Ansible Galaxy for a role by this
826 name and install it. Mutually exclusive with ``zuul``;
827 either ``galaxy`` or ``zuul`` must be supplied.
828
829 .. attr:: zuul
830
831 The name of a Zuul project which supplies the role. Mutually
832 exclusive with ``galaxy``; either ``galaxy`` or ``zuul`` must
833 be supplied.
834
835 .. attr:: name
836
837 The installation name of the role. In the case of a bare
838 role, the role will be made available under this name.
839 Ignored in the case of a contained role.
840
841 .. attr:: required-projects
842
843 A list of other projects which are used by this job. Any Zuul
844 projects specified here will also be checked out by Zuul into
845 the working directory for the job. Speculative merging and
846 cross-repo dependencies will be honored.
847
848 The format for this attribute is either a list of strings or
849 dictionaries. Strings are interpreted as project names,
850 dictionaries, if used, may have the following attributes:
851
852 .. attr:: name
853 :required:
854
855 The name of the required project.
856
857 .. attr:: override-branch
858
859 When Zuul runs jobs for a proposed change, it normally checks
860 out the branch associated with that change on every project
861 present in the job. If jobs are running on a ref (such as a
862 branch tip or tag), then that ref is normally checked out.
863 This attribute is used to override that behavior and indicate
864 that this job should, regardless of the branch for the queue
865 item, use the indicated branch instead, for only this
James E. Blairac3c7ae2017-07-31 09:01:08 -0700866 project. See also the :attr:`job.override-branch` attribute
James E. Blair32c52482017-07-29 07:49:03 -0700867 to apply the same behavior to all projects in a job.
868
869 .. attr:: vars
870
871 A dictionary of variables to supply to Ansible. When inheriting
872 from a job (or creating a variant of a job) vars are merged with
873 previous definitions. This means a variable definition with the
874 same name will override a previously defined variable, but new
875 variable names will be added to the set of defined variables.
876
877 .. attr:: dependencies
878
879 A list of other jobs upon which this job depends. Zuul will not
880 start executing this job until all of its dependencies have
881 completed successfully, and if one or more of them fail, this
882 job will not be run.
883
884 .. attr:: allowed-projects
885
886 A list of Zuul projects which may use this job. By default, a
887 job may be used by any other project known to Zuul, however,
888 some jobs use resources or perform actions which are not
889 appropriate for other projects. In these cases, a list of
890 projects which are allowed to use this job may be supplied. If
891 this list is not empty, then it must be an exhaustive list of
892 all projects permitted to use the job. The current project
893 (where the job is defined) is not automatically included, so if
894 it should be able to run this job, then it must be explicitly
James E. Blair88d84242017-07-31 12:05:16 -0700895 listed. By default, all projects may use the job.
James E. Blair1de8d402017-05-07 17:08:04 -0700896
897
898.. _project:
899
900Project
901~~~~~~~
902
903A project corresponds to a source code repository with which Zuul is
James E. Blairac3c7ae2017-07-31 09:01:08 -0700904configured to interact. The main responsibility of the project
James E. Blair1de8d402017-05-07 17:08:04 -0700905configuration item is to specify which jobs should run in which
James E. Blairac3c7ae2017-07-31 09:01:08 -0700906pipelines for a given project. Within each project definition, a
907section for each :ref:`pipeline <pipeline>` may appear. This
908project-pipeline definition is what determines how a project
909participates in a pipeline.
James E. Blair1de8d402017-05-07 17:08:04 -0700910
James E. Blairac3c7ae2017-07-31 09:01:08 -0700911Consider the following project definition::
James E. Blair1de8d402017-05-07 17:08:04 -0700912
913 - project:
914 name: yoyodyne
915 check:
916 jobs:
917 - check-syntax
918 - unit-tests
919 gate:
920 queue: integrated
921 jobs:
922 - unit-tests
923 - integration-tests
924
James E. Blairac3c7ae2017-07-31 09:01:08 -0700925The project has two project-pipeline stanzas, one for the ``check``
926pipeline, and one for ``gate``. Each specifies which jobs should run
927when a change for that project enters the respective pipeline -- when
928a change enters ``check``, the ``check-syntax`` and ``unit-test`` jobs
929are run.
James E. Blair1de8d402017-05-07 17:08:04 -0700930
James E. Blairac3c7ae2017-07-31 09:01:08 -0700931Pipelines which use the dependent pipeline manager (e.g., the ``gate``
James E. Blair1de8d402017-05-07 17:08:04 -0700932example shown earlier) maintain separate queues for groups of
933projects. When Zuul serializes a set of changes which represent
934future potential project states, it must know about all of the
935projects within Zuul which may have an effect on the outcome of the
936jobs it runs. If project *A* uses project *B* as a library, then Zuul
937must be told about that relationship so that it knows to serialize
938changes to A and B together, so that it does not merge a change to B
939while it is testing a change to A.
940
941Zuul could simply assume that all projects are related, or even infer
942relationships by which projects a job indicates it uses, however, in a
943large system that would become unwieldy very quickly, and
944unnecessarily delay changes to unrelated projects. To allow for
945flexibility in the construction of groups of related projects, the
946change queues used by dependent pipeline managers are specified
947manually. To group two or more related projects into a shared queue
948for a dependent pipeline, set the ``queue`` parameter to the same
949value for those projects.
950
James E. Blairac3c7ae2017-07-31 09:01:08 -0700951The ``gate`` project-pipeline definition above specifies that this
952project participates in the ``integrated`` shared queue for that
James E. Blair1de8d402017-05-07 17:08:04 -0700953pipeline.
954
James E. Blair9d4384d2017-08-01 15:54:50 -0700955.. attr:: project
James E. Blair1de8d402017-05-07 17:08:04 -0700956
James E. Blair9d4384d2017-08-01 15:54:50 -0700957 In addition to a project-pipeline definition for one or more
958 pipelines, the following attributes may appear in a project:
James E. Blair1de8d402017-05-07 17:08:04 -0700959
James E. Blair9d4384d2017-08-01 15:54:50 -0700960 .. attr:: name
961 :required:
962
963 The name of the project. If Zuul is configured with two or more
964 unique projects with the same name, the canonical hostname for
965 the project should be included (e.g., `git.example.com/foo`).
966
967 .. attr:: templates
968
969 A list of :ref:`project-template` references; the
970 project-pipeline definitions of each Project Template will be
971 applied to this project. If more than one template includes
972 jobs for a given pipeline, they will be combined, as will any
973 jobs specified in project-pipeline definitions on the project
974 itself.
975
976 .. attr:: merge-mode
977 :default: merge-resolve
978
979 The merge mode which is used by Git for this project. Be sure
980 this matches what the remote system which performs merges (i.e.,
981 Gerrit or GitHub). Must be one of the following values:
982
983 .. value:: merge
984
985 Uses the default git merge strategy (recursive).
986
987 .. value:: merge-resolve
988
989 Uses the resolve git merge strategy. This is a very
990 conservative merge strategy which most closely matches the
991 behavior of Gerrit.
992
993 .. value:: cherry-pick
994
995 Cherry-picks each change onto the branch rather than
996 performing any merges.
997
998 .. attr:: <pipeline>
999
1000 Each pipeline that the project participates in should have an
1001 entry in the project. The value for this key should be a
1002 dictionary with the following format:
1003
1004 .. attr:: jobs
1005 :required:
1006
1007 A list of jobs that should be run when items for this project
1008 are enqueued into the pipeline. Each item of this list may
1009 be a string, in which case it is treated as a job name, or it
1010 may be a dictionary, in which case it is treated as a job
1011 variant local to this project and pipeline. In that case,
1012 the format of the dictionary is the same as the top level
1013 :attr:`job` definition. Any attributes set on the job here
1014 will override previous versions of the job.
1015
1016 .. attr:: queue
1017
1018 If this pipeline is a :value:`dependent
1019 <pipeline.manager.dependent>` pipeline, this specifies the
1020 name of the shared queue this project is in. Any projects
1021 which interact with each other in tests should be part of the
1022 same shared queue in order to ensure that they don't merge
1023 changes which break the others. This is a free-form string;
1024 just set the same value for each group of projects.
James E. Blair1de8d402017-05-07 17:08:04 -07001025
1026.. _project-template:
1027
1028Project Template
1029~~~~~~~~~~~~~~~~
1030
1031A Project Template defines one or more project-pipeline definitions
1032which can be re-used by multiple projects.
1033
1034A Project Template uses the same syntax as a :ref:`project`
James E. Blairaafabe92017-08-02 15:23:19 -07001035definition, however, in the case of a template, the
1036:attr:`project.name` attribute does not refer to the name of a
1037project, but rather names the template so that it can be referenced in
1038a `Project` definition.
James E. Blair1de8d402017-05-07 17:08:04 -07001039
1040.. _secret:
1041
1042Secret
1043~~~~~~
1044
1045A Secret is a collection of private data for use by one or more jobs.
1046In order to maintain the security of the data, the values are usually
1047encrypted, however, data which are not sensitive may be provided
1048unencrypted as well for convenience.
1049
1050A Secret may only be used by jobs defined within the same project. To
1051use a secret, a :ref:`job` must specify the secret within its `auth`
1052section. To protect against jobs in other repositories declaring a
1053job with a secret as a parent and then exposing that secret, jobs
1054which inherit from a job with secrets will not inherit the secrets
1055themselves. To alter that behavior, see the `inherit` job attribute.
1056Further, jobs which do not permit children to inherit secrets (the
1057default) are also automatically marked `final`, meaning that their
1058execution related attributes may not be changed in a project-pipeline
1059stanza. This is to protect against a job with secrets defined in one
1060project being used by another project in a way which might expose the
1061secrets. If a job with secrets is unsafe to be used by other
1062projects, the `allowed-projects` job attribute can be used to restrict
1063the projects which can invoke that job. Finally, pipelines which are
1064used to execute proposed but unreviewed changes can set the
1065`allow-secrets` attribute to indicate that they should not supply
1066secrets at all in order to protect against someone proposing a change
1067which exposes a secret.
1068
James E. Blairaafabe92017-08-02 15:23:19 -07001069.. attr:: secret
James E. Blair1de8d402017-05-07 17:08:04 -07001070
James E. Blairaafabe92017-08-02 15:23:19 -07001071 The following attributes must appear on a secret:
James E. Blair1de8d402017-05-07 17:08:04 -07001072
James E. Blairaafabe92017-08-02 15:23:19 -07001073 .. attr:: name
1074 :required:
1075
1076 The name of the secret, used in a :ref:`Job` definition to
1077 request the secret.
1078
1079 .. attr:: data
1080 :required:
1081
1082 A dictionary which will be added to the Ansible variables
1083 available to the job. The values can either be plain text
1084 strings, or encrypted values. See :ref:`encryption` for more
1085 information.
James E. Blair1de8d402017-05-07 17:08:04 -07001086
1087.. _nodeset:
1088
1089Nodeset
1090~~~~~~~
1091
1092A Nodeset is a named collection of nodes for use by a job. Jobs may
1093specify what nodes they require individually, however, by defining
1094groups of node types once and referring to them by name, job
1095configuration may be simplified.
1096
Tobias Henkeldb686e22017-08-01 09:15:31 +02001097.. code-block:: yaml
James E. Blair1de8d402017-05-07 17:08:04 -07001098
Tobias Henkeldb686e22017-08-01 09:15:31 +02001099 - nodeset:
1100 name: nodeset1
1101 nodes:
1102 - name: controller
1103 label: controller-label
1104 - name: compute1
1105 label: compute-label
1106 - name: compute2
1107 label: compute-label
1108 groups:
1109 - name: ceph-osd
1110 nodes:
1111 - controller
1112 - name: ceph-monitor
1113 nodes:
1114 - controller
1115 - compute1
1116 - compute2
James E. Blair1de8d402017-05-07 17:08:04 -07001117
Tobias Henkeldb686e22017-08-01 09:15:31 +02001118.. attr:: nodeset
James E. Blair1de8d402017-05-07 17:08:04 -07001119
Tobias Henkeldb686e22017-08-01 09:15:31 +02001120 A Nodeset requires two attributes:
James E. Blair1de8d402017-05-07 17:08:04 -07001121
Tobias Henkeldb686e22017-08-01 09:15:31 +02001122 .. attr:: name
1123 :required:
1124
1125 The name of the Nodeset, to be referenced by a :ref:`job`.
1126
1127 .. attr:: nodes
1128 :required:
1129
1130 A list of node definitions, each of which has the following format:
1131
1132 .. attr:: name
1133 :required:
1134
1135 The name of the node. This will appear in the Ansible inventory
1136 for the job.
1137
1138 .. attr:: label
1139 :required:
1140
1141 The Nodepool label for the node. Zuul will request a node with
1142 this label.
1143
1144 .. attr:: groups
1145
1146 Additional groups can be defined which are accessible from the ansible
1147 playbooks.
1148
1149 .. attr:: name
1150 :required:
1151
1152 The name of the group to be referenced by an ansible playbook.
1153
1154 .. attr:: nodes
1155 :required:
1156
1157 The nodes that shall be part of the group. This is specified as a list
1158 of strings.
James E. Blair1de8d402017-05-07 17:08:04 -07001159
1160.. _semaphore:
1161
1162Semaphore
1163~~~~~~~~~
1164
1165Semaphores can be used to restrict the number of certain jobs which
1166are running at the same time. This may be useful for jobs which
1167access shared or limited resources. A semaphore has a value which
1168represents the maximum number of jobs which use that semaphore at the
1169same time.
1170
1171Semaphores are never subject to dynamic reconfiguration. If the value
1172of a semaphore is changed, it will take effect only when the change
Tobias Henkel76832982017-08-01 08:37:40 +02001173where it is updated is merged. An example follows:
James E. Blair1de8d402017-05-07 17:08:04 -07001174
Tobias Henkel76832982017-08-01 08:37:40 +02001175.. code-block:: yaml
James E. Blair1de8d402017-05-07 17:08:04 -07001176
Tobias Henkel76832982017-08-01 08:37:40 +02001177 - semaphore:
1178 name: semaphore-foo
1179 max: 5
1180 - semaphore:
1181 name: semaphore-bar
1182 max: 3
James E. Blair1de8d402017-05-07 17:08:04 -07001183
Tobias Henkel76832982017-08-01 08:37:40 +02001184.. attr:: semaphore
James E. Blair1de8d402017-05-07 17:08:04 -07001185
Tobias Henkel76832982017-08-01 08:37:40 +02001186 The following attributes are available:
1187
1188 .. attr:: name
1189 :required:
1190
1191 The name of the semaphore, referenced by jobs.
1192
1193 .. attr:: max
1194 :default: 1
1195
1196 The maximum number of running jobs which can use this semaphore.