blob: 4151eda11307449fc57f67775dc5fcd7d75f3d6e [file] [log] [blame]
James E. Blair1de8d402017-05-07 17:08:04 -07001:title: Project Configuration
2
3.. _project-config:
4
5Project Configuration
6=====================
7
8The following sections describe the main part of Zuul's configuration.
9All of what follows is found within files inside of the repositories
10that Zuul manages.
11
12Security Contexts
13-----------------
14
15When a system administrator configures Zuul to operate on a project,
16they specify one of two security contexts for that project. A
17*config-project* is one which is primarily tasked with holding
18configuration information and job content for Zuul. Jobs which are
James E. Blairac3c7ae2017-07-31 09:01:08 -070019defined in a config-project are run with elevated privileges, and all
James E. Blair2bab6e72017-08-07 09:52:45 -070020Zuul configuration items are available for use. Base jobs (that is,
21jobs without a parent) may only be defined in config-projects. It is
22expected that changes to config-projects will undergo careful scrutiny
23before being merged.
James E. Blair1de8d402017-05-07 17:08:04 -070024
25An *untrusted-project* is a project whose primary focus is not to
26operate Zuul, but rather it is one of the projects being tested or
27deployed. The Zuul configuration language available to these projects
James E. Blairac3c7ae2017-07-31 09:01:08 -070028is somewhat restricted (as detailed in individual sections below), and
James E. Blair1de8d402017-05-07 17:08:04 -070029jobs defined in these projects run in a restricted execution
30environment since they may be operating on changes which have not yet
31undergone review.
32
33Configuration Loading
34---------------------
35
36When Zuul starts, it examines all of the git repositories which are
James E. Blairac3c7ae2017-07-31 09:01:08 -070037specified by the system administrator in :ref:`tenant-config` and
38searches for files in the root of each repository. Zuul looks first
39for a file named ``zuul.yaml`` or a directory named ``zuul.d``, and if
40they are not found, ``.zuul.yaml`` or ``.zuul.d`` (with a leading
41dot). In the case of an :term:`untrusted-project`, the configuration
42from every branch is included, however, in the case of a
43:term:`config-project`, only the ``master`` branch is examined.
James E. Blair1de8d402017-05-07 17:08:04 -070044
45When a change is proposed to one of these files in an
James E. Blairac3c7ae2017-07-31 09:01:08 -070046untrusted-project, the configuration proposed in the change is merged
47into the running configuration so that any changes to Zuul's
James E. Blair1de8d402017-05-07 17:08:04 -070048configuration are self-testing as part of that change. If there is a
49configuration error, no jobs will be run and the error will be
50reported by any applicable pipelines. In the case of a change to a
James E. Blairac3c7ae2017-07-31 09:01:08 -070051config-project, the new configuration is parsed and examined for
James E. Blair1de8d402017-05-07 17:08:04 -070052errors, but the new configuration is not used in testing the change.
James E. Blairac3c7ae2017-07-31 09:01:08 -070053This is because configuration in config-projects is able to access
James E. Blair1de8d402017-05-07 17:08:04 -070054elevated privileges and should always be reviewed before being merged.
55
56As soon as a change containing a Zuul configuration change merges to
57any Zuul-managed repository, the new configuration takes effect
58immediately.
59
Monty Taylordb39bbb2017-08-23 17:24:00 -040060.. _configuration-items:
61
James E. Blair1de8d402017-05-07 17:08:04 -070062Configuration Items
63-------------------
64
James E. Blairac3c7ae2017-07-31 09:01:08 -070065The ``zuul.yaml`` and ``.zuul.yaml`` configuration files are
James E. Blair1de8d402017-05-07 17:08:04 -070066YAML-formatted and are structured as a series of items, each of which
67is described below.
68
James E. Blairac3c7ae2017-07-31 09:01:08 -070069In the case of a ``zuul.d`` directory, Zuul recurses the directory and
70extends the configuration using all the .yaml files in the sorted path
71order. For example, to keep job's variants in a separate file, it
72needs to be loaded after the main entries, for example using number
73prefixes in file's names::
Tristan Cacqueray4a015832017-07-11 05:18:14 +000074
75* zuul.d/pipelines.yaml
76* zuul.d/projects.yaml
77* zuul.d/01_jobs.yaml
78* zuul.d/02_jobs-variants.yaml
79
James E. Blair1de8d402017-05-07 17:08:04 -070080.. _pipeline:
81
82Pipeline
83~~~~~~~~
84
85A pipeline describes a workflow operation in Zuul. It associates jobs
86for a given project with triggering and reporting events.
87
88Its flexible configuration allows for characterizing any number of
89workflows, and by specifying each as a named configuration, makes it
90easy to apply similar workflow operations to projects or groups of
91projects.
92
93By way of example, one of the primary uses of Zuul is to perform
James E. Blairac3c7ae2017-07-31 09:01:08 -070094project gating. To do so, one can create a :term:`gate` pipeline
95which tells Zuul that when a certain event (such as approval by a code
James E. Blair1de8d402017-05-07 17:08:04 -070096reviewer) occurs, the corresponding change or pull request should be
97enqueued into the pipeline. When that happens, the jobs which have
James E. Blairac3c7ae2017-07-31 09:01:08 -070098been configured to run for that project in the gate pipeline are run,
99and when they complete, the pipeline reports the results to the user.
James E. Blair1de8d402017-05-07 17:08:04 -0700100
James E. Blairac3c7ae2017-07-31 09:01:08 -0700101Pipeline configuration items may only appear in :term:`config-projects
102<config-project>`.
James E. Blair1de8d402017-05-07 17:08:04 -0700103
104Generally, a Zuul administrator would define a small number of
105pipelines which represent the workflow processes used in their
106environment. Each project can then be added to the available
107pipelines as appropriate.
108
James E. Blairac3c7ae2017-07-31 09:01:08 -0700109Here is an example :term:`check` pipeline, which runs whenever a new
James E. Blair1de8d402017-05-07 17:08:04 -0700110patchset is created in Gerrit. If the associated jobs all report
James E. Blairac3c7ae2017-07-31 09:01:08 -0700111success, the pipeline reports back to Gerrit with ``Verified`` vote of
James E. Blair32c52482017-07-29 07:49:03 -0700112+1, or if at least one of them fails, a -1:
James E. Blair1de8d402017-05-07 17:08:04 -0700113
James E. Blair32c52482017-07-29 07:49:03 -0700114.. code-block:: yaml
115
116 - pipeline:
117 name: check
118 manager: independent
119 trigger:
120 my_gerrit:
121 - event: patchset-created
122 success:
123 my_gerrit:
124 Verified: 1
125 failure:
126 my_gerrit
127 Verified: -1
James E. Blair1de8d402017-05-07 17:08:04 -0700128
James E. Blaireff5a9d2017-06-20 00:00:37 -0700129.. TODO: See TODO for more annotated examples of common pipeline configurations.
James E. Blair1de8d402017-05-07 17:08:04 -0700130
James E. Blair94375912017-07-28 17:20:27 -0700131.. attr:: pipeline
James E. Blair7145c582017-07-26 13:30:39 -0700132
James E. Blair9fd98ab2017-07-26 14:15:26 -0700133 The attributes available on a pipeline are as follows (all are
134 optional unless otherwise specified):
James E. Blair1de8d402017-05-07 17:08:04 -0700135
James E. Blair94375912017-07-28 17:20:27 -0700136 .. attr:: name
James E. Blair9fd98ab2017-07-26 14:15:26 -0700137 :required:
James E. Blair1de8d402017-05-07 17:08:04 -0700138
James E. Blair9fd98ab2017-07-26 14:15:26 -0700139 This is used later in the project definition to indicate what jobs
140 should be run for events in the pipeline.
James E. Blair1de8d402017-05-07 17:08:04 -0700141
James E. Blair94375912017-07-28 17:20:27 -0700142 .. attr:: manager
James E. Blair9fd98ab2017-07-26 14:15:26 -0700143 :required:
James E. Blaireff5a9d2017-06-20 00:00:37 -0700144
James E. Blair9fd98ab2017-07-26 14:15:26 -0700145 There are currently two schemes for managing pipelines:
James E. Blair1de8d402017-05-07 17:08:04 -0700146
James E. Blair32c52482017-07-29 07:49:03 -0700147 .. value:: independent
James E. Blaireff5a9d2017-06-20 00:00:37 -0700148
James E. Blair9fd98ab2017-07-26 14:15:26 -0700149 Every event in this pipeline should be treated as independent
150 of other events in the pipeline. This is appropriate when
151 the order of events in the pipeline doesn't matter because
152 the results of the actions this pipeline performs can not
153 affect other events in the pipeline. For example, when a
154 change is first uploaded for review, you may want to run
155 tests on that change to provide early feedback to reviewers.
156 At the end of the tests, the change is not going to be
157 merged, so it is safe to run these tests in parallel without
158 regard to any other changes in the pipeline. They are
159 independent.
James E. Blair1de8d402017-05-07 17:08:04 -0700160
James E. Blair9fd98ab2017-07-26 14:15:26 -0700161 Another type of pipeline that is independent is a post-merge
162 pipeline. In that case, the changes have already merged, so
163 the results can not affect any other events in the pipeline.
James E. Blair1761e862017-07-25 16:15:47 -0700164
James E. Blair32c52482017-07-29 07:49:03 -0700165 .. value:: dependent
James E. Blair1761e862017-07-25 16:15:47 -0700166
James E. Blair9fd98ab2017-07-26 14:15:26 -0700167 The dependent pipeline manager is designed for gating. It
168 ensures that every change is tested exactly as it is going to
169 be merged into the repository. An ideal gating system would
170 test one change at a time, applied to the tip of the
171 repository, and only if that change passed tests would it be
172 merged. Then the next change in line would be tested the
173 same way. In order to achieve parallel testing of changes,
174 the dependent pipeline manager performs speculative execution
175 on changes. It orders changes based on their entry into the
176 pipeline. It begins testing all changes in parallel,
177 assuming that each change ahead in the pipeline will pass its
178 tests. If they all succeed, all the changes can be tested
179 and merged in parallel. If a change near the front of the
180 pipeline fails its tests, each change behind it ignores
181 whatever tests have been completed and are tested again
182 without the change in front. This way gate tests may run in
183 parallel but still be tested correctly, exactly as they will
184 appear in the repository when merged.
James E. Blair1761e862017-07-25 16:15:47 -0700185
James E. Blair9fd98ab2017-07-26 14:15:26 -0700186 For more detail on the theory and operation of Zuul's
187 dependent pipeline manager, see: :doc:`gating`.
James E. Blair1de8d402017-05-07 17:08:04 -0700188
James E. Blair8eb564a2017-08-10 09:21:41 -0700189 .. attr:: post-review
James E. Blair88d84242017-07-31 12:05:16 -0700190 :default: false
James E. Blairf17aa9c2017-07-05 13:21:23 -0700191
James E. Blair8eb564a2017-08-10 09:21:41 -0700192 This is a boolean which indicates that this pipeline executes
193 code that has been reviewed. Some jobs perform actions which
194 should not be permitted with unreviewed code. When this value
195 is ``false`` those jobs will not be permitted to run in the
196 pipeline. If a pipeline is designed only to be used after
197 changes are reviewed or merged, set this value to ``true`` to
198 permit such jobs.
James E. Blairf17aa9c2017-07-05 13:21:23 -0700199
James E. Blair8eb564a2017-08-10 09:21:41 -0700200 For more information, see :ref:`secret` and
201 :attr:`job.post-review`.
James E. Blair1de8d402017-05-07 17:08:04 -0700202
James E. Blair94375912017-07-28 17:20:27 -0700203 .. attr:: description
James E. Blair1de8d402017-05-07 17:08:04 -0700204
James E. Blair9fd98ab2017-07-26 14:15:26 -0700205 This field may be used to provide a textual description of the
206 pipeline. It may appear in the status page or in documentation.
James E. Blair1de8d402017-05-07 17:08:04 -0700207
James E. Blair94375912017-07-28 17:20:27 -0700208 .. attr:: success-message
James E. Blair88d84242017-07-31 12:05:16 -0700209 :default: Build successful.
James E. Blair1de8d402017-05-07 17:08:04 -0700210
James E. Blair9fd98ab2017-07-26 14:15:26 -0700211 The introductory text in reports when all the voting jobs are
James E. Blair88d84242017-07-31 12:05:16 -0700212 successful.
James E. Blair1de8d402017-05-07 17:08:04 -0700213
James E. Blair94375912017-07-28 17:20:27 -0700214 .. attr:: failure-message
James E. Blair88d84242017-07-31 12:05:16 -0700215 :default: Build failed.
James E. Blair1de8d402017-05-07 17:08:04 -0700216
James E. Blair9fd98ab2017-07-26 14:15:26 -0700217 The introductory text in reports when at least one voting job
James E. Blair88d84242017-07-31 12:05:16 -0700218 fails.
James E. Blair1de8d402017-05-07 17:08:04 -0700219
James E. Blair94375912017-07-28 17:20:27 -0700220 .. attr:: merge-failure-message
James E. Blair88d84242017-07-31 12:05:16 -0700221 :default: Merge failed.
James E. Blair1de8d402017-05-07 17:08:04 -0700222
James E. Blair9fd98ab2017-07-26 14:15:26 -0700223 The introductory text in the message reported when a change
224 fails to merge with the current state of the repository.
225 Defaults to "Merge failed."
James E. Blair1de8d402017-05-07 17:08:04 -0700226
James E. Blair94375912017-07-28 17:20:27 -0700227 .. attr:: footer-message
James E. Blair1de8d402017-05-07 17:08:04 -0700228
James E. Blair9fd98ab2017-07-26 14:15:26 -0700229 Supplies additional information after test results. Useful for
230 adding information about the CI system such as debugging and
231 contact details.
James E. Blair1de8d402017-05-07 17:08:04 -0700232
James E. Blair94375912017-07-28 17:20:27 -0700233 .. attr:: trigger
James E. Blair1de8d402017-05-07 17:08:04 -0700234
James E. Blair9fd98ab2017-07-26 14:15:26 -0700235 At least one trigger source must be supplied for each pipeline.
236 Triggers are not exclusive -- matching events may be placed in
237 multiple pipelines, and they will behave independently in each
238 of the pipelines they match.
James E. Blair1de8d402017-05-07 17:08:04 -0700239
James E. Blair9fd98ab2017-07-26 14:15:26 -0700240 Triggers are loaded from their connection name. The driver type
241 of the connection will dictate which options are available. See
242 :ref:`drivers`.
James E. Blair1de8d402017-05-07 17:08:04 -0700243
James E. Blair94375912017-07-28 17:20:27 -0700244 .. attr:: require
James E. Blair1de8d402017-05-07 17:08:04 -0700245
James E. Blairac3c7ae2017-07-31 09:01:08 -0700246 If this section is present, it establishes prerequisites for
James E. Blair9fd98ab2017-07-26 14:15:26 -0700247 any kind of item entering the Pipeline. Regardless of how the
248 item is to be enqueued (via any trigger or automatic dependency
249 resolution), the conditions specified here must be met or the
James E. Blaird134c6d2017-07-26 16:09:34 -0700250 item will not be enqueued. These requirements may vary
251 depending on the source of the item being enqueued.
James E. Blair1de8d402017-05-07 17:08:04 -0700252
James E. Blaird134c6d2017-07-26 16:09:34 -0700253 Requirements are loaded from their connection name. The driver
254 type of the connection will dictate which options are available.
255 See :ref:`drivers`.
James E. Blair1de8d402017-05-07 17:08:04 -0700256
James E. Blair94375912017-07-28 17:20:27 -0700257 .. attr:: reject
James E. Blair1de8d402017-05-07 17:08:04 -0700258
James E. Blairac3c7ae2017-07-31 09:01:08 -0700259 If this section is present, it establishes prerequisites that
James E. Blair9fd98ab2017-07-26 14:15:26 -0700260 can block an item from being enqueued. It can be considered a
James E. Blairac3c7ae2017-07-31 09:01:08 -0700261 negative version of :attr:`pipeline.require`.
James E. Blair1de8d402017-05-07 17:08:04 -0700262
James E. Blaird134c6d2017-07-26 16:09:34 -0700263 Requirements are loaded from their connection name. The driver
264 type of the connection will dictate which options are available.
265 See :ref:`drivers`.
James E. Blair9fd98ab2017-07-26 14:15:26 -0700266
James E. Blair94375912017-07-28 17:20:27 -0700267 .. attr:: dequeue-on-new-patchset
James E. Blair88d84242017-07-31 12:05:16 -0700268 :default: true
James E. Blair9fd98ab2017-07-26 14:15:26 -0700269
270 Normally, if a new patchset is uploaded to a change that is in a
271 pipeline, the existing entry in the pipeline will be removed
272 (with jobs canceled and any dependent changes that can no longer
273 merge as well. To suppress this behavior (and allow jobs to
James E. Blair88d84242017-07-31 12:05:16 -0700274 continue running), set this to ``false``.
James E. Blair9fd98ab2017-07-26 14:15:26 -0700275
James E. Blair94375912017-07-28 17:20:27 -0700276 .. attr:: ignore-dependencies
James E. Blair88d84242017-07-31 12:05:16 -0700277 :default: false
James E. Blair9fd98ab2017-07-26 14:15:26 -0700278
279 In any kind of pipeline (dependent or independent), Zuul will
280 attempt to enqueue all dependencies ahead of the current change
281 so that they are tested together (independent pipelines report
282 the results of each change regardless of the results of changes
283 ahead). To ignore dependencies completely in an independent
284 pipeline, set this to ``true``. This option is ignored by
James E. Blair88d84242017-07-31 12:05:16 -0700285 dependent pipelines.
James E. Blair9fd98ab2017-07-26 14:15:26 -0700286
James E. Blair94375912017-07-28 17:20:27 -0700287 .. attr:: precedence
James E. Blair88d84242017-07-31 12:05:16 -0700288 :default: normal
James E. Blair9fd98ab2017-07-26 14:15:26 -0700289
290 Indicates how the build scheduler should prioritize jobs for
291 different pipelines. Each pipeline may have one precedence,
292 jobs for pipelines with a higher precedence will be run before
293 ones with lower. The value should be one of ``high``,
294 ``normal``, or ``low``. Default: ``normal``.
295
James E. Blairac3c7ae2017-07-31 09:01:08 -0700296 .. _reporters:
297
298 The following options configure :term:`reporters <reporter>`.
299 Reporters are complementary to triggers; where a trigger is an
300 event on a connection which causes Zuul to enqueue an item, a
301 reporter is the action performed on a connection when an item is
302 dequeued after its jobs complete. The actual syntax for a reporter
303 is defined by the driver which implements it. See :ref:`drivers`
304 for more information.
James E. Blair9fd98ab2017-07-26 14:15:26 -0700305
James E. Blair94375912017-07-28 17:20:27 -0700306 .. attr:: success
James E. Blair9fd98ab2017-07-26 14:15:26 -0700307
308 Describes where Zuul should report to if all the jobs complete
309 successfully. This section is optional; if it is omitted, Zuul
310 will run jobs and do nothing on success -- it will not report at
James E. Blairac3c7ae2017-07-31 09:01:08 -0700311 all. If the section is present, the listed :term:`reporters
312 <reporter>` will be asked to report on the jobs. The reporters
313 are listed by their connection name. The options available
314 depend on the driver for the supplied connection.
James E. Blair9fd98ab2017-07-26 14:15:26 -0700315
James E. Blair94375912017-07-28 17:20:27 -0700316 .. attr:: failure
James E. Blair9fd98ab2017-07-26 14:15:26 -0700317
318 These reporters describe what Zuul should do if at least one job
319 fails.
320
James E. Blair94375912017-07-28 17:20:27 -0700321 .. attr:: merge-failure
James E. Blair9fd98ab2017-07-26 14:15:26 -0700322
323 These reporters describe what Zuul should do if it is unable to
324 merge in the patchset. If no merge-failure reporters are listed
325 then the ``failure`` reporters will be used to notify of
326 unsuccessful merges.
327
James E. Blair94375912017-07-28 17:20:27 -0700328 .. attr:: start
James E. Blair9fd98ab2017-07-26 14:15:26 -0700329
330 These reporters describe what Zuul should do when a change is
331 added to the pipeline. This can be used, for example, to reset
332 a previously reported result.
333
James E. Blair94375912017-07-28 17:20:27 -0700334 .. attr:: disabled
James E. Blair9fd98ab2017-07-26 14:15:26 -0700335
336 These reporters describe what Zuul should do when a pipeline is
337 disabled. See ``disable-after-consecutive-failures``.
338
339 The following options can be used to alter Zuul's behavior to
340 mitigate situations in which jobs are failing frequently (perhaps
341 due to a problem with an external dependency, or unusually high
342 non-deterministic test failures).
343
James E. Blair94375912017-07-28 17:20:27 -0700344 .. attr:: disable-after-consecutive-failures
James E. Blair9fd98ab2017-07-26 14:15:26 -0700345
James E. Blairac3c7ae2017-07-31 09:01:08 -0700346 If set, a pipeline can enter a *disabled* state if too many
James E. Blair9fd98ab2017-07-26 14:15:26 -0700347 changes in a row fail. When this value is exceeded the pipeline
James E. Blairac3c7ae2017-07-31 09:01:08 -0700348 will stop reporting to any of the **success**, **failure** or
349 **merge-failure** reporters and instead only report to the
350 **disabled** reporters. (No **start** reports are made when a
James E. Blair9fd98ab2017-07-26 14:15:26 -0700351 pipeline is disabled).
352
James E. Blair94375912017-07-28 17:20:27 -0700353 .. attr:: window
James E. Blair88d84242017-07-31 12:05:16 -0700354 :default: 20
James E. Blair9fd98ab2017-07-26 14:15:26 -0700355
356 Dependent pipeline managers only. Zuul can rate limit dependent
357 pipelines in a manner similar to TCP flow control. Jobs are
358 only started for items in the queue if they are within the
359 actionable window for the pipeline. The initial length of this
360 window is configurable with this value. The value given should
361 be a positive integer value. A value of ``0`` disables rate
James E. Blairac3c7ae2017-07-31 09:01:08 -0700362 limiting on the :value:`dependent pipeline manager
James E. Blair88d84242017-07-31 12:05:16 -0700363 <pipeline.manager.dependent>`.
James E. Blair9fd98ab2017-07-26 14:15:26 -0700364
James E. Blair94375912017-07-28 17:20:27 -0700365 .. attr:: window-floor
James E. Blair88d84242017-07-31 12:05:16 -0700366 :default: 3
James E. Blair9fd98ab2017-07-26 14:15:26 -0700367
368 Dependent pipeline managers only. This is the minimum value for
369 the window described above. Should be a positive non zero
James E. Blair88d84242017-07-31 12:05:16 -0700370 integer value.
James E. Blair9fd98ab2017-07-26 14:15:26 -0700371
James E. Blair94375912017-07-28 17:20:27 -0700372 .. attr:: window-increase-type
James E. Blair88d84242017-07-31 12:05:16 -0700373 :default: linear
James E. Blair9fd98ab2017-07-26 14:15:26 -0700374
375 Dependent pipeline managers only. This value describes how the
James E. Blairac3c7ae2017-07-31 09:01:08 -0700376 window should grow when changes are successfully merged by zuul.
377
378 .. value:: linear
379
380 Indicates that **window-increase-factor** should be added to
James E. Blair88d84242017-07-31 12:05:16 -0700381 the previous window value.
James E. Blairac3c7ae2017-07-31 09:01:08 -0700382
383 .. value:: exponential
384
385 Indicates that **window-increase-factor** should be
386 multiplied against the previous window value and the result
387 will become the window size.
James E. Blair9fd98ab2017-07-26 14:15:26 -0700388
James E. Blair94375912017-07-28 17:20:27 -0700389 .. attr:: window-increase-factor
James E. Blair88d84242017-07-31 12:05:16 -0700390 :default: 1
James E. Blair9fd98ab2017-07-26 14:15:26 -0700391
392 Dependent pipeline managers only. The value to be added or
393 multiplied against the previous window value to determine the
James E. Blair88d84242017-07-31 12:05:16 -0700394 new window after successful change merges.
James E. Blair9fd98ab2017-07-26 14:15:26 -0700395
James E. Blair94375912017-07-28 17:20:27 -0700396 .. attr:: window-decrease-type
James E. Blair88d84242017-07-31 12:05:16 -0700397 :default: exponential
James E. Blair9fd98ab2017-07-26 14:15:26 -0700398
399 Dependent pipeline managers only. This value describes how the
400 window should shrink when changes are not able to be merged by
James E. Blairac3c7ae2017-07-31 09:01:08 -0700401 Zuul.
402
403 .. value:: linear
404
405 Indicates that **window-decrease-factor** should be
406 subtracted from the previous window value.
407
408 .. value:: exponential
409
410 Indicates that **window-decrease-factor** should be divided
411 against the previous window value and the result will become
James E. Blair88d84242017-07-31 12:05:16 -0700412 the window size.
James E. Blair9fd98ab2017-07-26 14:15:26 -0700413
James E. Blair94375912017-07-28 17:20:27 -0700414 .. attr:: window-decrease-factor
James E. Blair88d84242017-07-31 12:05:16 -0700415 :default: 2
James E. Blair9fd98ab2017-07-26 14:15:26 -0700416
James E. Blairac3c7ae2017-07-31 09:01:08 -0700417 :value:`Dependent pipeline managers
418 <pipeline.manager.dependent>` only. The value to be subtracted
419 or divided against the previous window value to determine the
James E. Blair88d84242017-07-31 12:05:16 -0700420 new window after unsuccessful change merges.
James E. Blair1de8d402017-05-07 17:08:04 -0700421
422
423.. _job:
424
425Job
426~~~
427
428A job is a unit of work performed by Zuul on an item enqueued into a
429pipeline. Items may run any number of jobs (which may depend on each
430other). Each job is an invocation of an Ansible playbook with a
431specific inventory of hosts. The actual tasks that are run by the job
432appear in the playbook for that job while the attributes that appear in the
433Zuul configuration specify information about when, where, and how the
434job should be run.
435
436Jobs in Zuul support inheritance. Any job may specify a single parent
437job, and any attributes not set on the child job are collected from
438the parent job. In this way, a configuration structure may be built
439starting with very basic jobs which describe characteristics that all
440jobs on the system should have, progressing through stages of
441specialization before arriving at a particular job. A job may inherit
442from any other job in any project (however, if the other job is marked
Tobias Henkel83167622017-06-30 19:45:03 +0200443as :attr:`job.final`, jobs may not inherit from it).
James E. Blair1de8d402017-05-07 17:08:04 -0700444
James E. Blair2bab6e72017-08-07 09:52:45 -0700445A job with no parent is called a *base job* and may only be defined in
446a :term:`config-project`. Every other job must have a parent, and so
447ultimately, all jobs must have an inheritance path which terminates at
448a base job. Each tenant has a default parent job which will be used
449if no explicit parent is specified.
450
James E. Blairc32a8352017-10-11 16:27:50 -0700451Multiple job definitions with the same name are called variants.
James E. Blair1de8d402017-05-07 17:08:04 -0700452These may have different selection criteria which indicate to Zuul
453that, for instance, the job should behave differently on a different
454git branch. Unlike inheritance, all job variants must be defined in
Tobias Henkel83167622017-06-30 19:45:03 +0200455the same project. Some attributes of jobs marked :attr:`job.final`
456may not be overidden
James E. Blair1de8d402017-05-07 17:08:04 -0700457
458When Zuul decides to run a job, it performs a process known as
459freezing the job. Because any number of job variants may be
460applicable, Zuul collects all of the matching variants and applies
461them in the order they appeared in the configuration. The resulting
462frozen job is built from attributes gathered from all of the
463matching variants. In this way, exactly what is run is dependent on
464the pipeline, project, branch, and content of the item.
465
466In addition to the job's main playbook, each job may specify one or
467more pre- and post-playbooks. These are run, in order, before and
468after (respectively) the main playbook. They may be used to set up
469and tear down resources needed by the main playbook. When combined
470with inheritance, they provide powerful tools for job construction. A
471job only has a single main playbook, and when inheriting from a
472parent, the child's main playbook overrides (or replaces) the
473parent's. However, the pre- and post-playbooks are appended and
474prepended in a nesting fashion. So if a parent job and child job both
475specified pre and post playbooks, the sequence of playbooks run would
476be:
477
478* parent pre-run playbook
479* child pre-run playbook
480* child playbook
481* child post-run playbook
482* parent post-run playbook
483
484Further inheritance would nest even deeper.
485
James E. Blair32c52482017-07-29 07:49:03 -0700486Here is an example of two job definitions:
James E. Blair1de8d402017-05-07 17:08:04 -0700487
James E. Blair32c52482017-07-29 07:49:03 -0700488.. code-block:: yaml
James E. Blair1de8d402017-05-07 17:08:04 -0700489
James E. Blair32c52482017-07-29 07:49:03 -0700490 - job:
491 name: base
492 pre-run: copy-git-repos
493 post-run: copy-logs
James E. Blair1de8d402017-05-07 17:08:04 -0700494
James E. Blair32c52482017-07-29 07:49:03 -0700495 - job:
496 name: run-tests
497 parent: base
James E. Blair7e3e6882017-09-20 15:47:13 -0700498 nodeset:
499 nodes:
500 - name: test-node
501 label: fedora
James E. Blair1de8d402017-05-07 17:08:04 -0700502
James E. Blair32c52482017-07-29 07:49:03 -0700503.. attr:: job
James E. Blair1de8d402017-05-07 17:08:04 -0700504
James E. Blair32c52482017-07-29 07:49:03 -0700505 The following attributes are available on a job; all are optional
506 unless otherwise specified:
James E. Blair1de8d402017-05-07 17:08:04 -0700507
James E. Blair32c52482017-07-29 07:49:03 -0700508 .. attr:: name
509 :required:
James E. Blair1de8d402017-05-07 17:08:04 -0700510
James E. Blair32c52482017-07-29 07:49:03 -0700511 The name of the job. By default, Zuul looks for a playbook with
512 this name to use as the main playbook for the job. This name is
513 also referenced later in a project pipeline configuration.
James E. Blair1de8d402017-05-07 17:08:04 -0700514
James E. Blair2bab6e72017-08-07 09:52:45 -0700515 .. TODO: figure out how to link the parent default to tenant.default.parent
516
James E. Blair32c52482017-07-29 07:49:03 -0700517 .. attr:: parent
James E. Blair2bab6e72017-08-07 09:52:45 -0700518 :default: Tenant default-parent
James E. Blair1de8d402017-05-07 17:08:04 -0700519
James E. Blairac3c7ae2017-07-31 09:01:08 -0700520 Specifies a job to inherit from. The parent job can be defined
James E. Blair2bab6e72017-08-07 09:52:45 -0700521 in this or any other project. Any attributes not specified on a
522 job will be collected from its parent. If no value is supplied
523 here, the job specified by :attr:`tenant.default-parent` will be
524 used. If **parent** is set to ``null`` (which is only valid in
525 a :term:`config-project`), this is a :term:`base job`.
James E. Blair1de8d402017-05-07 17:08:04 -0700526
James E. Blair32c52482017-07-29 07:49:03 -0700527 .. attr:: description
James E. Blair1de8d402017-05-07 17:08:04 -0700528
James E. Blair32c52482017-07-29 07:49:03 -0700529 A textual description of the job. Not currently used directly
530 by Zuul, but it is used by the zuul-sphinx extension to Sphinx
531 to auto-document Zuul jobs (in which case it is interpreted as
532 ReStructuredText.
James E. Blair1de8d402017-05-07 17:08:04 -0700533
Tobias Henkel83167622017-06-30 19:45:03 +0200534 .. attr:: final
535 :default: false
536
537 To prevent other jobs from inheriting from this job, and also to
538 prevent changing execution-related attributes when this job is
539 specified in a project's pipeline, set this attribute to
540 ``true``.
541
James E. Blair32c52482017-07-29 07:49:03 -0700542 .. attr:: success-message
James E. Blair88d84242017-07-31 12:05:16 -0700543 :default: SUCCESS
James E. Blair1de8d402017-05-07 17:08:04 -0700544
James E. Blairac3c7ae2017-07-31 09:01:08 -0700545 Normally when a job succeeds, the string ``SUCCESS`` is reported
James E. Blair32c52482017-07-29 07:49:03 -0700546 as the result for the job. If set, this option may be used to
James E. Blair88d84242017-07-31 12:05:16 -0700547 supply a different string.
James E. Blair1de8d402017-05-07 17:08:04 -0700548
James E. Blair32c52482017-07-29 07:49:03 -0700549 .. attr:: failure-message
James E. Blair88d84242017-07-31 12:05:16 -0700550 :default: FAILURE
James E. Blair1de8d402017-05-07 17:08:04 -0700551
James E. Blairac3c7ae2017-07-31 09:01:08 -0700552 Normally when a job fails, the string ``FAILURE`` is reported as
James E. Blair32c52482017-07-29 07:49:03 -0700553 the result for the job. If set, this option may be used to
James E. Blair88d84242017-07-31 12:05:16 -0700554 supply a different string.
James E. Blair1de8d402017-05-07 17:08:04 -0700555
James E. Blair32c52482017-07-29 07:49:03 -0700556 .. attr:: success-url
James E. Blair1de8d402017-05-07 17:08:04 -0700557
James E. Blair32c52482017-07-29 07:49:03 -0700558 When a job succeeds, this URL is reported along with the result.
559 If this value is not supplied, Zuul uses the content of the job
560 :ref:`return value <return_values>` **zuul.log_url**. This is
561 recommended as it allows the code which stores the URL to the
562 job artifacts to report exactly where they were stored. To
563 override this value, or if it is not set, supply an absolute URL
564 in this field. If a relative URL is supplied in this field, and
565 **zuul.log_url** is set, then the two will be combined to
566 produce the URL used for the report. This can be used to
567 specify that certain jobs should "deep link" into the stored job
James E. Blair88d84242017-07-31 12:05:16 -0700568 artifacts.
James E. Blair1de8d402017-05-07 17:08:04 -0700569
James E. Blair32c52482017-07-29 07:49:03 -0700570 .. attr:: failure-url
James E. Blair1de8d402017-05-07 17:08:04 -0700571
James E. Blair32c52482017-07-29 07:49:03 -0700572 When a job fails, this URL is reported along with the result.
573 Otherwise behaves the same as **success-url**.
James E. Blair1de8d402017-05-07 17:08:04 -0700574
James E. Blair32c52482017-07-29 07:49:03 -0700575 .. attr:: hold-following-changes
James E. Blair88d84242017-07-31 12:05:16 -0700576 :default: false
James E. Blair1de8d402017-05-07 17:08:04 -0700577
James E. Blair32c52482017-07-29 07:49:03 -0700578 In a dependent pipeline, this option may be used to indicate
579 that no jobs should start on any items which depend on the
580 current item until this job has completed successfully. This
581 may be used to conserve build resources, at the expense of
582 inhibiting the parallelization which speeds the processing of
James E. Blair88d84242017-07-31 12:05:16 -0700583 items in a dependent pipeline.
James E. Blair1de8d402017-05-07 17:08:04 -0700584
James E. Blair32c52482017-07-29 07:49:03 -0700585 .. attr:: voting
James E. Blair88d84242017-07-31 12:05:16 -0700586 :default: true
James E. Blair1de8d402017-05-07 17:08:04 -0700587
James E. Blair32c52482017-07-29 07:49:03 -0700588 Indicates whether the result of this job should be used in
James E. Blair88d84242017-07-31 12:05:16 -0700589 determining the overall result of the item.
James E. Blair1de8d402017-05-07 17:08:04 -0700590
James E. Blair32c52482017-07-29 07:49:03 -0700591 .. attr:: semaphore
James E. Blair1de8d402017-05-07 17:08:04 -0700592
James E. Blair32c52482017-07-29 07:49:03 -0700593 The name of a :ref:`semaphore` which should be acquired and
594 released when the job begins and ends. If the semaphore is at
595 maximum capacity, then Zuul will wait until it can be acquired
James E. Blair88d84242017-07-31 12:05:16 -0700596 before starting the job.
James E. Blair1de8d402017-05-07 17:08:04 -0700597
James E. Blair32c52482017-07-29 07:49:03 -0700598 .. attr:: tags
James E. Blair1de8d402017-05-07 17:08:04 -0700599
James E. Blair32c52482017-07-29 07:49:03 -0700600 Metadata about this job. Tags are units of information attached
601 to the job; they do not affect Zuul's behavior, but they can be
602 used within the job to characterize the job. For example, a job
603 which tests a certain subsystem could be tagged with the name of
604 that subsystem, and if the job's results are reported into a
605 database, then the results of all jobs affecting that subsystem
606 could be queried. This attribute is specified as a list of
607 strings, and when inheriting jobs or applying variants, tags
608 accumulate in a set, so the result is always a set of all the
609 tags from all the jobs and variants used in constructing the
James E. Blair88d84242017-07-31 12:05:16 -0700610 frozen job, with no duplication.
James E. Blair1de8d402017-05-07 17:08:04 -0700611
James E. Blair32c52482017-07-29 07:49:03 -0700612 .. attr:: branches
James E. Blair1de8d402017-05-07 17:08:04 -0700613
James E. Blair32c52482017-07-29 07:49:03 -0700614 A regular expression (or list of regular expressions) which
615 describe on what branches a job should run (or in the case of
616 variants: to alter the behavior of a job for a certain branch).
James E. Blair1de8d402017-05-07 17:08:04 -0700617
James E. Blair32c52482017-07-29 07:49:03 -0700618 If there is no job definition for a given job which matches the
619 branch of an item, then that job is not run for the item.
620 Otherwise, all of the job variants which match that branch (and
621 any other selection criteria) are used when freezing the job.
James E. Blair1de8d402017-05-07 17:08:04 -0700622
James E. Blair32c52482017-07-29 07:49:03 -0700623 This example illustrates a job called *run-tests* which uses a
624 nodeset based on the current release of an operating system to
625 perform its tests, except when testing changes to the stable/2.0
626 branch, in which case it uses an older release:
James E. Blair1de8d402017-05-07 17:08:04 -0700627
James E. Blair32c52482017-07-29 07:49:03 -0700628 .. code-block:: yaml
James E. Blair1de8d402017-05-07 17:08:04 -0700629
James E. Blair32c52482017-07-29 07:49:03 -0700630 - job:
631 name: run-tests
James E. Blair7e3e6882017-09-20 15:47:13 -0700632 nodeset: current-release
James E. Blair1de8d402017-05-07 17:08:04 -0700633
James E. Blair32c52482017-07-29 07:49:03 -0700634 - job:
635 name: run-tests
Andreas Jaeger438af862017-10-22 10:17:11 +0200636 branches: stable/2.0
James E. Blair7e3e6882017-09-20 15:47:13 -0700637 nodeset: old-release
James E. Blair1de8d402017-05-07 17:08:04 -0700638
James E. Blairac3c7ae2017-07-31 09:01:08 -0700639 In some cases, Zuul uses an implied value for the branch
640 specifier if none is supplied:
James E. Blair1de8d402017-05-07 17:08:04 -0700641
James E. Blairac3c7ae2017-07-31 09:01:08 -0700642 * For a job definition in a :term:`config-project`, no implied
643 branch specifier is used. If no branch specifier appears, the
644 job applies to all branches.
James E. Blair1de8d402017-05-07 17:08:04 -0700645
James E. Blairdaaf3262017-10-23 13:51:48 -0700646 * In the case of an :term:`untrusted-project`, if the project
647 has only one branch, no implied branch specifier is applied to
648 :ref:`job` definitions. If the project has more than one
649 branch, the branch containing the job definition is used as an
650 implied branch specifier.
James E. Blair1de8d402017-05-07 17:08:04 -0700651
James E. Blaire74f5712017-09-29 15:14:31 -0700652 * In the case of a job variant defined within a :ref:`project`,
653 if the project definition is in a :term:`config-project`, no
654 implied branch specifier is used. If it appears in an
655 :term:`untrusted-project`, with no branch specifier, the
656 branch containing the project definition is used as an implied
657 branch specifier.
658
659 * In the case of a job variant defined within a
660 :ref:`project-template`, if no branch specifier appears, the
James E. Blaire36d1a32017-11-28 13:33:38 -0800661 implied branch containing the project-template definition is
662 used as an implied branch specifier. This means that
663 definitions of the same project-template on different branches
664 may run different jobs.
665
666 When that project-template is used by a :ref:`project`
667 definition within a :term:`untrusted-project`, the branch
668 containing that project definition is combined with the branch
669 specifier of the project-template. This means it is possible
670 for a project to use a template on one branch, but not on
671 another.
James E. Blaire74f5712017-09-29 15:14:31 -0700672
James E. Blair32c52482017-07-29 07:49:03 -0700673 This allows for the very simple and expected workflow where if a
James E. Blairac3c7ae2017-07-31 09:01:08 -0700674 project defines a job on the ``master`` branch with no branch
675 specifier, and then creates a new branch based on ``master``,
676 any changes to that job definition within the new branch only
James E. Blairdaaf3262017-10-23 13:51:48 -0700677 affect that branch, and likewise, changes to the master branch
678 only affect it.
James E. Blair1de8d402017-05-07 17:08:04 -0700679
James E. Blair7edc25f2017-10-26 10:47:14 -0700680 See :attr:`pragma.implied-branch-matchers` for how to override
681 this behavior on a per-file basis.
682
James E. Blair32c52482017-07-29 07:49:03 -0700683 .. attr:: files
Tobias Henkel2aade262017-07-12 16:09:06 +0200684
James E. Blair32c52482017-07-29 07:49:03 -0700685 This attribute indicates that the job should only run on changes
686 where the specified files are modified. This is a regular
James E. Blair88d84242017-07-31 12:05:16 -0700687 expression or list of regular expressions.
James E. Blair1de8d402017-05-07 17:08:04 -0700688
James E. Blair32c52482017-07-29 07:49:03 -0700689 .. attr:: irrelevant-files
James E. Blair74a82cf2017-07-12 17:23:08 -0700690
James E. Blairac3c7ae2017-07-31 09:01:08 -0700691 This is a negative complement of **files**. It indicates that
692 the job should run unless *all* of the files changed match this
693 list. In other words, if the regular expression ``docs/.*`` is
James E. Blair32c52482017-07-29 07:49:03 -0700694 supplied, then this job will not run if the only files changed
695 are in the docs directory. A regular expression or list of
James E. Blair88d84242017-07-31 12:05:16 -0700696 regular expressions.
James E. Blair1de8d402017-05-07 17:08:04 -0700697
James E. Blaire19e88a2017-08-09 15:14:29 -0700698 .. attr:: secrets
James E. Blair1de8d402017-05-07 17:08:04 -0700699
James E. Blaire19e88a2017-08-09 15:14:29 -0700700 A list of secrets which may be used by the job. A
701 :ref:`secret` is a named collection of private information
702 defined separately in the configuration. The secrets that
703 appear here must be defined in the same project as this job
704 definition.
James E. Blair1de8d402017-05-07 17:08:04 -0700705
Monty Tayloraff8b402017-08-16 18:40:41 -0500706 Each item in the list may may be supplied either as a string,
707 in which case it references the name of a :ref:`secret` definition,
708 or as a dict. If an element in this list is given as a dict, it
709 must have the following fields.
710
711 .. attr:: name
712
713 The name to use for the Ansible variable into which the secret
714 content will be placed.
715
716 .. attr:: secret
717
718 The name to use to find the secret's definition in the configuration.
719
720 For example:
721
722 .. code-block:: yaml
723
724 - secret:
725 important-secret:
726 key: encrypted-secret-key-data
727
728 - job:
729 name: amazing-job:
730 secrets:
731 - name: ssh_key
732 secret: important-secret
733
734 will result in the following being passed as a variable to the playbooks
735 in ``amazing-job``:
736
737 .. code-block:: yaml
738
739 ssh_key:
740 key: descrypted-secret-key-data
741
James E. Blair7e3e6882017-09-20 15:47:13 -0700742 .. attr:: nodeset
James E. Blair1de8d402017-05-07 17:08:04 -0700743
James E. Blair7e3e6882017-09-20 15:47:13 -0700744 The nodes which should be supplied to the job. This parameter
745 may be supplied either as a string, in which case it references
746 a :ref:`nodeset` definition which appears elsewhere in the
747 configuration, or a dictionary, in which case it is interpreted
748 in the same way as a Nodeset definition, though the ``name``
749 attribute should be omitted (in essence, it is an anonymous
750 Nodeset definition unique to this job). See the :ref:`nodeset`
751 reference for the syntax to use in that case.
James E. Blair1de8d402017-05-07 17:08:04 -0700752
James E. Blair7e3e6882017-09-20 15:47:13 -0700753 If a job has an empty or no nodeset definition, it will still
754 run and may be able to perform actions on the Zuul executor.
James E. Blair1de8d402017-05-07 17:08:04 -0700755
James E. Blairedff2c22017-10-30 14:04:48 -0700756 .. attr:: override-checkout
James E. Blair1de8d402017-05-07 17:08:04 -0700757
James E. Blair32c52482017-07-29 07:49:03 -0700758 When Zuul runs jobs for a proposed change, it normally checks
759 out the branch associated with that change on every project
760 present in the job. If jobs are running on a ref (such as a
761 branch tip or tag), then that ref is normally checked out. This
762 attribute is used to override that behavior and indicate that
763 this job should, regardless of the branch for the queue item,
James E. Blairedff2c22017-10-30 14:04:48 -0700764 use the indicated ref (i.e., branch or tag) instead. This can
765 be used, for example, to run a previous version of the software
766 (from a stable maintenance branch) under test even if the change
767 being tested applies to a different branch (this is only likely
768 to be useful if there is some cross-branch interaction with some
James E. Blair32c52482017-07-29 07:49:03 -0700769 component of the system being tested). See also the
James E. Blairedff2c22017-10-30 14:04:48 -0700770 project-specific :attr:`job.required-projects.override-checkout`
James E. Blairac3c7ae2017-07-31 09:01:08 -0700771 attribute to apply this behavior to a subset of a job's
772 projects.
James E. Blair1de8d402017-05-07 17:08:04 -0700773
James E. Blair32c52482017-07-29 07:49:03 -0700774 .. attr:: timeout
James E. Blair1de8d402017-05-07 17:08:04 -0700775
James E. Blairb0c8e7e2017-08-28 09:19:49 -0700776 The time in seconds that the job should be allowed to run before
James E. Blair32c52482017-07-29 07:49:03 -0700777 it is automatically aborted and failure is reported. If no
778 timeout is supplied, the job may run indefinitely. Supplying a
779 timeout is highly recommended.
780
781 .. attr:: attempts
James E. Blair88d84242017-07-31 12:05:16 -0700782 :default: 3
James E. Blair32c52482017-07-29 07:49:03 -0700783
784 When Zuul encounters an error running a job's pre-run playbook,
785 Zuul will stop and restart the job. Errors during the main or
786 post-run -playbook phase of a job are not affected by this
787 parameter (they are reported immediately). This parameter
788 controls the number of attempts to make before an error is
James E. Blair88d84242017-07-31 12:05:16 -0700789 reported.
James E. Blair32c52482017-07-29 07:49:03 -0700790
791 .. attr:: pre-run
792
James E. Blair68286562017-10-26 10:55:16 -0700793 The name of a playbook or list of playbooks to run before the
794 main body of a job. The full path to the playbook in the repo
795 where the job is defined is expected.
James E. Blair32c52482017-07-29 07:49:03 -0700796
797 When a job inherits from a parent, the child's pre-run playbooks
798 are run after the parent's. See :ref:`job` for more
799 information.
800
801 .. attr:: post-run
802
James E. Blair68286562017-10-26 10:55:16 -0700803 The name of a playbook or list of playbooks to run after the
804 main body of a job. The full path to the playbook in the repo
805 where the job is defined is expected.
James E. Blair32c52482017-07-29 07:49:03 -0700806
807 When a job inherits from a parent, the child's post-run
808 playbooks are run before the parent's. See :ref:`job` for more
809 information.
810
811 .. attr:: run
812
James E. Blair68286562017-10-26 10:55:16 -0700813 The name of the main playbook for this job. If it is not
814 supplied, the parent's playbook will be used (and likewise up
815 the inheritance chain). The full path within the repo is
816 required. Example:
James E. Blair32c52482017-07-29 07:49:03 -0700817
818 .. code-block:: yaml
819
James E. Blair68286562017-10-26 10:55:16 -0700820 run: playbooks/job-playbook.yaml
821
James E. Blair32c52482017-07-29 07:49:03 -0700822 .. attr:: roles
823
824 A list of Ansible roles to prepare for the job. Because a job
825 runs an Ansible playbook, any roles which are used by the job
826 must be prepared and installed by Zuul before the job begins.
827 This value is a list of dictionaries, each of which indicates
828 one of two types of roles: a Galaxy role, which is simply a role
829 that is installed from Ansible Galaxy, or a Zuul role, which is
830 a role provided by a project managed by Zuul. Zuul roles are
831 able to benefit from speculative merging and cross-project
832 dependencies when used by playbooks in untrusted projects.
833 Roles are added to the Ansible role path in the order they
834 appear on the job -- roles earlier in the list will take
835 precedence over those which follow.
836
837 In the case of job inheritance or variance, the roles used for
838 each of the playbooks run by the job will be only those which
839 were defined along with that playbook. If a child job inherits
840 from a parent which defines a pre and post playbook, then the
841 pre and post playbooks it inherits from the parent job will run
842 only with the roles that were defined on the parent. If the
843 child adds its own pre and post playbooks, then any roles added
844 by the child will be available to the child's playbooks. This
845 is so that a job which inherits from a parent does not
James E. Blairac3c7ae2017-07-31 09:01:08 -0700846 inadvertently alter the behavior of the parent's playbooks by
James E. Blair32c52482017-07-29 07:49:03 -0700847 the addition of conflicting roles. Roles added by a child will
848 appear before those it inherits from its parent.
849
850 A project which supplies a role may be structured in one of two
851 configurations: a bare role (in which the role exists at the
852 root of the project), or a contained role (in which the role
James E. Blairac3c7ae2017-07-31 09:01:08 -0700853 exists within the ``roles/`` directory of the project, perhaps
James E. Blair32c52482017-07-29 07:49:03 -0700854 along with other roles). In the case of a contained role, the
James E. Blairac3c7ae2017-07-31 09:01:08 -0700855 ``roles/`` directory of the project is added to the role search
James E. Blair32c52482017-07-29 07:49:03 -0700856 path. In the case of a bare role, the project itself is added
857 to the role search path. In case the name of the project is not
858 the name under which the role should be installed (and therefore
James E. Blairac3c7ae2017-07-31 09:01:08 -0700859 referenced from Ansible), the ``name`` attribute may be used to
James E. Blair32c52482017-07-29 07:49:03 -0700860 specify an alternate.
861
862 A job automatically has the project in which it is defined added
863 to the roles path if that project appears to contain a role or
James E. Blairac3c7ae2017-07-31 09:01:08 -0700864 ``roles/`` directory. By default, the project is added to the
James E. Blair32c52482017-07-29 07:49:03 -0700865 path under its own name, however, that may be changed by
866 explicitly listing the project in the roles list in the usual
867 way.
868
869 .. note:: Galaxy roles are not yet implemented.
870
871 .. attr:: galaxy
872
873 The name of the role in Ansible Galaxy. If this attribute is
874 supplied, Zuul will search Ansible Galaxy for a role by this
875 name and install it. Mutually exclusive with ``zuul``;
876 either ``galaxy`` or ``zuul`` must be supplied.
877
878 .. attr:: zuul
879
880 The name of a Zuul project which supplies the role. Mutually
881 exclusive with ``galaxy``; either ``galaxy`` or ``zuul`` must
882 be supplied.
883
884 .. attr:: name
885
886 The installation name of the role. In the case of a bare
887 role, the role will be made available under this name.
888 Ignored in the case of a contained role.
889
890 .. attr:: required-projects
891
892 A list of other projects which are used by this job. Any Zuul
893 projects specified here will also be checked out by Zuul into
894 the working directory for the job. Speculative merging and
895 cross-repo dependencies will be honored.
896
897 The format for this attribute is either a list of strings or
898 dictionaries. Strings are interpreted as project names,
899 dictionaries, if used, may have the following attributes:
900
901 .. attr:: name
902 :required:
903
904 The name of the required project.
905
James E. Blairedff2c22017-10-30 14:04:48 -0700906 .. attr:: override-checkout
James E. Blair32c52482017-07-29 07:49:03 -0700907
908 When Zuul runs jobs for a proposed change, it normally checks
909 out the branch associated with that change on every project
910 present in the job. If jobs are running on a ref (such as a
911 branch tip or tag), then that ref is normally checked out.
912 This attribute is used to override that behavior and indicate
913 that this job should, regardless of the branch for the queue
James E. Blairedff2c22017-10-30 14:04:48 -0700914 item, use the indicated ref (i.e., branch or tag) instead,
915 for only this project. See also the
916 :attr:`job.override-checkout` attribute to apply the same
917 behavior to all projects in a job.
James E. Blair32c52482017-07-29 07:49:03 -0700918
919 .. attr:: vars
920
921 A dictionary of variables to supply to Ansible. When inheriting
922 from a job (or creating a variant of a job) vars are merged with
923 previous definitions. This means a variable definition with the
924 same name will override a previously defined variable, but new
925 variable names will be added to the set of defined variables.
926
927 .. attr:: dependencies
928
929 A list of other jobs upon which this job depends. Zuul will not
930 start executing this job until all of its dependencies have
931 completed successfully, and if one or more of them fail, this
932 job will not be run.
933
934 .. attr:: allowed-projects
935
936 A list of Zuul projects which may use this job. By default, a
937 job may be used by any other project known to Zuul, however,
938 some jobs use resources or perform actions which are not
939 appropriate for other projects. In these cases, a list of
940 projects which are allowed to use this job may be supplied. If
941 this list is not empty, then it must be an exhaustive list of
942 all projects permitted to use the job. The current project
943 (where the job is defined) is not automatically included, so if
944 it should be able to run this job, then it must be explicitly
James E. Blair88d84242017-07-31 12:05:16 -0700945 listed. By default, all projects may use the job.
James E. Blair1de8d402017-05-07 17:08:04 -0700946
James E. Blair8eb564a2017-08-10 09:21:41 -0700947 .. attr:: post-review
948 :default: false
James E. Blair892cca62017-08-09 11:36:58 -0700949
James E. Blair8eb564a2017-08-10 09:21:41 -0700950 A boolean value which indicates whether this job may only be
951 used in pipelines where :attr:`pipeline.post-review` is
Monty Taylora49b0ea2017-10-05 16:16:19 -0500952 ``true``. This is automatically set to ``true`` if this job
953 uses a :ref:`secret` and is defined in a :term:`untrusted-project`.
954 It may be explicitly set to obtain the same behavior for jobs
955 defined in :term:`config projects <config-project>`. Once this
956 is set to ``true`` anywhere in the inheritance hierarchy for a job,
957 it will remain set for all child jobs and variants (it can not be
James E. Blair8eb564a2017-08-10 09:21:41 -0700958 set to ``false``).
James E. Blair1de8d402017-05-07 17:08:04 -0700959
960.. _project:
961
962Project
963~~~~~~~
964
965A project corresponds to a source code repository with which Zuul is
James E. Blairac3c7ae2017-07-31 09:01:08 -0700966configured to interact. The main responsibility of the project
James E. Blair1de8d402017-05-07 17:08:04 -0700967configuration item is to specify which jobs should run in which
James E. Blairac3c7ae2017-07-31 09:01:08 -0700968pipelines for a given project. Within each project definition, a
969section for each :ref:`pipeline <pipeline>` may appear. This
970project-pipeline definition is what determines how a project
971participates in a pipeline.
James E. Blair1de8d402017-05-07 17:08:04 -0700972
James E. Blair0b7dc852017-10-10 13:41:03 -0700973Multiple project definitions may appear for the same project (for
Monty Taylor44812322017-10-21 15:35:03 +0200974example, in a central :term:`config projects <config-project>` as well
James E. Blair0b7dc852017-10-10 13:41:03 -0700975as in a repo's own ``.zuul.yaml``). In this case, all of the project
976definitions are combined (the jobs listed in all of the definitions
977will be run).
978
James E. Blairac3c7ae2017-07-31 09:01:08 -0700979Consider the following project definition::
James E. Blair1de8d402017-05-07 17:08:04 -0700980
981 - project:
982 name: yoyodyne
983 check:
984 jobs:
985 - check-syntax
986 - unit-tests
987 gate:
988 queue: integrated
989 jobs:
990 - unit-tests
991 - integration-tests
992
James E. Blairac3c7ae2017-07-31 09:01:08 -0700993The project has two project-pipeline stanzas, one for the ``check``
994pipeline, and one for ``gate``. Each specifies which jobs should run
995when a change for that project enters the respective pipeline -- when
996a change enters ``check``, the ``check-syntax`` and ``unit-test`` jobs
997are run.
James E. Blair1de8d402017-05-07 17:08:04 -0700998
James E. Blairac3c7ae2017-07-31 09:01:08 -0700999Pipelines which use the dependent pipeline manager (e.g., the ``gate``
James E. Blair1de8d402017-05-07 17:08:04 -07001000example shown earlier) maintain separate queues for groups of
1001projects. When Zuul serializes a set of changes which represent
1002future potential project states, it must know about all of the
1003projects within Zuul which may have an effect on the outcome of the
1004jobs it runs. If project *A* uses project *B* as a library, then Zuul
1005must be told about that relationship so that it knows to serialize
1006changes to A and B together, so that it does not merge a change to B
1007while it is testing a change to A.
1008
1009Zuul could simply assume that all projects are related, or even infer
1010relationships by which projects a job indicates it uses, however, in a
1011large system that would become unwieldy very quickly, and
1012unnecessarily delay changes to unrelated projects. To allow for
1013flexibility in the construction of groups of related projects, the
1014change queues used by dependent pipeline managers are specified
1015manually. To group two or more related projects into a shared queue
1016for a dependent pipeline, set the ``queue`` parameter to the same
1017value for those projects.
1018
James E. Blairac3c7ae2017-07-31 09:01:08 -07001019The ``gate`` project-pipeline definition above specifies that this
1020project participates in the ``integrated`` shared queue for that
James E. Blair1de8d402017-05-07 17:08:04 -07001021pipeline.
1022
James E. Blair9d4384d2017-08-01 15:54:50 -07001023.. attr:: project
James E. Blair1de8d402017-05-07 17:08:04 -07001024
James E. Blair0b7dc852017-10-10 13:41:03 -07001025 The following attributes may appear in a project:
James E. Blair1de8d402017-05-07 17:08:04 -07001026
James E. Blair9d4384d2017-08-01 15:54:50 -07001027 .. attr:: name
1028 :required:
1029
1030 The name of the project. If Zuul is configured with two or more
1031 unique projects with the same name, the canonical hostname for
1032 the project should be included (e.g., `git.example.com/foo`).
1033
1034 .. attr:: templates
1035
1036 A list of :ref:`project-template` references; the
1037 project-pipeline definitions of each Project Template will be
1038 applied to this project. If more than one template includes
1039 jobs for a given pipeline, they will be combined, as will any
1040 jobs specified in project-pipeline definitions on the project
1041 itself.
1042
1043 .. attr:: merge-mode
1044 :default: merge-resolve
1045
1046 The merge mode which is used by Git for this project. Be sure
1047 this matches what the remote system which performs merges (i.e.,
1048 Gerrit or GitHub). Must be one of the following values:
1049
1050 .. value:: merge
1051
1052 Uses the default git merge strategy (recursive).
1053
1054 .. value:: merge-resolve
1055
1056 Uses the resolve git merge strategy. This is a very
1057 conservative merge strategy which most closely matches the
1058 behavior of Gerrit.
1059
1060 .. value:: cherry-pick
1061
1062 Cherry-picks each change onto the branch rather than
1063 performing any merges.
1064
1065 .. attr:: <pipeline>
1066
1067 Each pipeline that the project participates in should have an
1068 entry in the project. The value for this key should be a
1069 dictionary with the following format:
1070
1071 .. attr:: jobs
1072 :required:
1073
1074 A list of jobs that should be run when items for this project
1075 are enqueued into the pipeline. Each item of this list may
1076 be a string, in which case it is treated as a job name, or it
1077 may be a dictionary, in which case it is treated as a job
1078 variant local to this project and pipeline. In that case,
1079 the format of the dictionary is the same as the top level
1080 :attr:`job` definition. Any attributes set on the job here
1081 will override previous versions of the job.
1082
1083 .. attr:: queue
1084
1085 If this pipeline is a :value:`dependent
1086 <pipeline.manager.dependent>` pipeline, this specifies the
1087 name of the shared queue this project is in. Any projects
1088 which interact with each other in tests should be part of the
1089 same shared queue in order to ensure that they don't merge
1090 changes which break the others. This is a free-form string;
1091 just set the same value for each group of projects.
James E. Blair1de8d402017-05-07 17:08:04 -07001092
1093.. _project-template:
1094
1095Project Template
1096~~~~~~~~~~~~~~~~
1097
1098A Project Template defines one or more project-pipeline definitions
1099which can be re-used by multiple projects.
1100
1101A Project Template uses the same syntax as a :ref:`project`
James E. Blairaafabe92017-08-02 15:23:19 -07001102definition, however, in the case of a template, the
1103:attr:`project.name` attribute does not refer to the name of a
1104project, but rather names the template so that it can be referenced in
1105a `Project` definition.
James E. Blair1de8d402017-05-07 17:08:04 -07001106
1107.. _secret:
1108
1109Secret
1110~~~~~~
1111
1112A Secret is a collection of private data for use by one or more jobs.
1113In order to maintain the security of the data, the values are usually
1114encrypted, however, data which are not sensitive may be provided
1115unencrypted as well for convenience.
1116
1117A Secret may only be used by jobs defined within the same project. To
James E. Blaire19e88a2017-08-09 15:14:29 -07001118use a secret, a :ref:`job` must specify the secret in
1119:attr:`job.secrets`. Secrets are bound to the playbooks associated
1120with the specific job definition where they were declared. Additional
1121pre or post playbooks which appear in child jobs will not have access
1122to the secrets, nor will playbooks which override the main playbook
1123(if any) of the job which declared the secret. This protects against
1124jobs in other repositories declaring a job with a secret as a parent
1125and then exposing that secret.
James E. Blair892cca62017-08-09 11:36:58 -07001126
1127It is possible to use secrets for jobs defined in :term:`config
1128projects <config-project>` as well as :term:`untrusted projects
1129<untrusted-project>`, however their use differs slightly. Because
1130playbooks in a config project which use secrets run in the
1131:term:`trusted execution context` where proposed changes are not used
1132in executing jobs, it is safe for those secrets to be used in all
1133types of pipelines. However, because playbooks defined in an
1134untrusted project are run in the :term:`untrusted execution context`
1135where proposed changes are used in job execution, it is dangerous to
1136allow those secrets to be used in pipelines which are used to execute
James E. Blair8eb564a2017-08-10 09:21:41 -07001137proposed but unreviewed changes. By default, pipelines are considered
1138`pre-review` and will refuse to run jobs which have playbooks that use
1139secrets in the untrusted execution context to protect against someone
1140proposing a change which exposes a secret. To permit this (for
1141instance, in a pipeline which only runs after code review), the
1142:attr:`pipeline.post-review` attribute may be explicitly set to
1143``true``.
1144
1145In some cases, it may be desirable to prevent a job which is defined
1146in a config project from running in a pre-review pipeline (e.g., a job
1147used to publish an artifact). In these cases, the
1148:attr:`job.post-review` attribute may be explicitly set to ``true`` to
1149indicate the job should only run in post-review pipelines.
James E. Blair892cca62017-08-09 11:36:58 -07001150
1151If a job with secrets is unsafe to be used by other projects, the
1152`allowed-projects` job attribute can be used to restrict the projects
1153which can invoke that job.
James E. Blair1de8d402017-05-07 17:08:04 -07001154
James E. Blairaafabe92017-08-02 15:23:19 -07001155.. attr:: secret
James E. Blair1de8d402017-05-07 17:08:04 -07001156
James E. Blairaafabe92017-08-02 15:23:19 -07001157 The following attributes must appear on a secret:
James E. Blair1de8d402017-05-07 17:08:04 -07001158
James E. Blairaafabe92017-08-02 15:23:19 -07001159 .. attr:: name
1160 :required:
1161
1162 The name of the secret, used in a :ref:`Job` definition to
1163 request the secret.
1164
1165 .. attr:: data
1166 :required:
1167
1168 A dictionary which will be added to the Ansible variables
1169 available to the job. The values can either be plain text
1170 strings, or encrypted values. See :ref:`encryption` for more
1171 information.
James E. Blair1de8d402017-05-07 17:08:04 -07001172
1173.. _nodeset:
1174
1175Nodeset
1176~~~~~~~
1177
1178A Nodeset is a named collection of nodes for use by a job. Jobs may
1179specify what nodes they require individually, however, by defining
1180groups of node types once and referring to them by name, job
1181configuration may be simplified.
1182
Tobias Henkeldb686e22017-08-01 09:15:31 +02001183.. code-block:: yaml
James E. Blair1de8d402017-05-07 17:08:04 -07001184
Tobias Henkeldb686e22017-08-01 09:15:31 +02001185 - nodeset:
1186 name: nodeset1
1187 nodes:
1188 - name: controller
1189 label: controller-label
1190 - name: compute1
1191 label: compute-label
Paul Belangerecb0b842017-11-18 15:23:29 -05001192 - name:
1193 - compute2
1194 - web
Tobias Henkeldb686e22017-08-01 09:15:31 +02001195 label: compute-label
1196 groups:
1197 - name: ceph-osd
1198 nodes:
1199 - controller
1200 - name: ceph-monitor
1201 nodes:
1202 - controller
1203 - compute1
1204 - compute2
Paul Belangerecb0b842017-11-18 15:23:29 -05001205 - name: ceph-web
1206 nodes:
1207 - web
James E. Blair1de8d402017-05-07 17:08:04 -07001208
Tobias Henkeldb686e22017-08-01 09:15:31 +02001209.. attr:: nodeset
James E. Blair1de8d402017-05-07 17:08:04 -07001210
Tobias Henkeldb686e22017-08-01 09:15:31 +02001211 A Nodeset requires two attributes:
James E. Blair1de8d402017-05-07 17:08:04 -07001212
Tobias Henkeldb686e22017-08-01 09:15:31 +02001213 .. attr:: name
1214 :required:
1215
1216 The name of the Nodeset, to be referenced by a :ref:`job`.
1217
1218 .. attr:: nodes
1219 :required:
1220
1221 A list of node definitions, each of which has the following format:
1222
1223 .. attr:: name
1224 :required:
1225
1226 The name of the node. This will appear in the Ansible inventory
1227 for the job.
1228
Paul Belangerecb0b842017-11-18 15:23:29 -05001229 This can also be as a list of strings. If so, then the list of hosts in
1230 the Ansible inventory will share a common ansible_host address.
1231
Tobias Henkeldb686e22017-08-01 09:15:31 +02001232 .. attr:: label
1233 :required:
1234
1235 The Nodepool label for the node. Zuul will request a node with
1236 this label.
1237
1238 .. attr:: groups
1239
1240 Additional groups can be defined which are accessible from the ansible
1241 playbooks.
1242
1243 .. attr:: name
1244 :required:
1245
1246 The name of the group to be referenced by an ansible playbook.
1247
1248 .. attr:: nodes
1249 :required:
1250
1251 The nodes that shall be part of the group. This is specified as a list
1252 of strings.
James E. Blair1de8d402017-05-07 17:08:04 -07001253
1254.. _semaphore:
1255
1256Semaphore
1257~~~~~~~~~
1258
1259Semaphores can be used to restrict the number of certain jobs which
1260are running at the same time. This may be useful for jobs which
1261access shared or limited resources. A semaphore has a value which
1262represents the maximum number of jobs which use that semaphore at the
1263same time.
1264
1265Semaphores are never subject to dynamic reconfiguration. If the value
1266of a semaphore is changed, it will take effect only when the change
Tobias Henkel76832982017-08-01 08:37:40 +02001267where it is updated is merged. An example follows:
James E. Blair1de8d402017-05-07 17:08:04 -07001268
Tobias Henkel76832982017-08-01 08:37:40 +02001269.. code-block:: yaml
James E. Blair1de8d402017-05-07 17:08:04 -07001270
Tobias Henkel76832982017-08-01 08:37:40 +02001271 - semaphore:
1272 name: semaphore-foo
1273 max: 5
1274 - semaphore:
1275 name: semaphore-bar
1276 max: 3
James E. Blair1de8d402017-05-07 17:08:04 -07001277
Tobias Henkel76832982017-08-01 08:37:40 +02001278.. attr:: semaphore
James E. Blair1de8d402017-05-07 17:08:04 -07001279
Tobias Henkel76832982017-08-01 08:37:40 +02001280 The following attributes are available:
1281
1282 .. attr:: name
1283 :required:
1284
1285 The name of the semaphore, referenced by jobs.
1286
1287 .. attr:: max
1288 :default: 1
1289
1290 The maximum number of running jobs which can use this semaphore.
James E. Blair7edc25f2017-10-26 10:47:14 -07001291
1292.. _pragma:
1293
1294Pragma
1295~~~~~~
1296
1297The `pragma` item does not behave like the others. It can not be
1298included or excluded from configuration loading by the administrator,
1299and does not form part of the final configuration itself. It is used
1300to alter how the configuration is processed while loading.
1301
1302A pragma item only affects the current file. The same file in another
1303branch of the same project will not be affected, nor any other files
1304or any other projects. The effect is global within that file --
1305pragma directives may not be set and then unset within the same file.
1306
1307.. code-block:: yaml
1308
1309 - pragma:
1310 implied-branch-matchers: False
1311
1312.. attr:: pragma
1313
1314 The pragma item currently only supports one attribute:
1315
1316 .. attr:: implied-branch-matchers
1317
1318 This is a boolean, which, if set, may be used to enable
1319 (``True``) or disable (``False``) the addition of implied branch
1320 matchers to job definitions. Normally Zuul decides whether to
1321 add these based on heuristics described in :attr:`job.branches`.
1322 This attribute overrides that behavior.
1323
1324 This can be useful if a project has multiple branches, yet the
1325 jobs defined in the master branch should apply to all branches.
1326
1327 Note that if a job contains an explicit branch matcher, it will
1328 be used regardless of the value supplied here.