blob: 4c5a6243a8cae9074a0946bcd6327738202e6807 [file] [log] [blame]
James E. Blaircdd00072012-06-08 19:17:28 -07001:title: Zuul
2
3Zuul
4====
5
6Configuration
7-------------
8
9Zuul has three configuration files:
10
11**zuul.conf**
James E. Blair1f4c2bb2013-04-26 08:40:46 -070012 Connection information for Gerrit and Gearman, locations of the
Łukasz Jernaś048acb42014-03-02 18:49:41 +010013 other config files.
James E. Blaircdd00072012-06-08 19:17:28 -070014**layout.yaml**
Łukasz Jernaś048acb42014-03-02 18:49:41 +010015 Project and pipeline configuration -- what Zuul does.
James E. Blaircdd00072012-06-08 19:17:28 -070016**logging.conf**
Łukasz Jernaś048acb42014-03-02 18:49:41 +010017 Python logging config.
James E. Blaircdd00072012-06-08 19:17:28 -070018
19Examples of each of the three files can be found in the etc/ directory
20of the source distribution.
21
James E. Blair6aea36d2012-12-17 13:03:24 -080022.. _zuulconf:
23
James E. Blaircdd00072012-06-08 19:17:28 -070024zuul.conf
25~~~~~~~~~
26
27Zuul will look for ``/etc/zuul/zuul.conf`` or ``~/zuul.conf`` to
28bootstrap its configuration. Alternately, you may specify ``-c
29/path/to/zuul.conf`` on the command line.
30
James E. Blair1f4c2bb2013-04-26 08:40:46 -070031Gerrit and Gearman connection information are each described in a
32section of zuul.conf. The location of the other two configuration
33files (as well as the location of the PID file when running Zuul as a
34server) are specified in a third section.
James E. Blaircdd00072012-06-08 19:17:28 -070035
Clark Boylan9b670902012-09-28 13:47:56 -070036The three sections of this config and their options are documented below.
37You can also find an example zuul.conf file in the git
38`repository
K Jonathan Harker97e457e2013-02-26 13:29:38 -080039<https://github.com/openstack-infra/zuul/blob/master/etc/zuul.conf-sample>`_
Clark Boylan9b670902012-09-28 13:47:56 -070040
James E. Blair1f4c2bb2013-04-26 08:40:46 -070041gearman
Clark Boylan9b670902012-09-28 13:47:56 -070042"""""""
43
44**server**
James E. Blair1f4c2bb2013-04-26 08:40:46 -070045 Hostname or IP address of the Gearman server.
46 ``server=gearman.example.com``
Clark Boylan9b670902012-09-28 13:47:56 -070047
James E. Blair1f4c2bb2013-04-26 08:40:46 -070048**port**
Łukasz Jernaś048acb42014-03-02 18:49:41 +010049 Port on which the Gearman server is listening.
James E. Blair1f4c2bb2013-04-26 08:40:46 -070050 ``port=4730``
Clark Boylan9b670902012-09-28 13:47:56 -070051
James E. Blair77cc7b82013-07-15 13:22:37 -070052gearman_server
53""""""""""""""
54
55**start**
56 Whether to start the internal Gearman server (default: False).
57 ``start=true``
58
59**log_config**
60 Path to log config file for internal Gearman server.
61 ``log_config=/etc/zuul/gearman-logging.yaml``
62
Clark Boylan9b670902012-09-28 13:47:56 -070063gerrit
64""""""
65
66**server**
67 FQDN of Gerrit server.
68 ``server=review.example.com``
69
Akihiro Motokic7d28a72014-03-22 09:21:38 +090070**port**
71 Optional: Gerrit server port.
72 ``port=29418``
73
Antoine Musso27475012012-11-26 09:53:01 +010074**baseurl**
75 Optional: path to Gerrit web interface. Defaults to ``https://<value
76 of server>/``. ``baseurl=https://review.example.com/review_site/``
77
Clark Boylan9b670902012-09-28 13:47:56 -070078**user**
79 User name to use when logging into above server via ssh.
James E. Blair1f4c2bb2013-04-26 08:40:46 -070080 ``user=zuul``
Clark Boylan9b670902012-09-28 13:47:56 -070081
82**sshkey**
83 Path to SSH key to use when logging into above server.
James E. Blair1f4c2bb2013-04-26 08:40:46 -070084 ``sshkey=/home/zuul/.ssh/id_rsa``
Clark Boylan9b670902012-09-28 13:47:56 -070085
86zuul
87""""
88
89**layout_config**
James E. Blair4076e2b2014-01-28 12:42:20 -080090 Path to layout config file. Used by zuul-server only.
Clark Boylan9b670902012-09-28 13:47:56 -070091 ``layout_config=/etc/zuul/layout.yaml``
92
93**log_config**
James E. Blaira4430132014-02-17 08:32:07 -080094 Path to log config file. Used by zuul-server only.
Clark Boylan9b670902012-09-28 13:47:56 -070095 ``log_config=/etc/zuul/logging.yaml``
96
97**pidfile**
James E. Blaira4430132014-02-17 08:32:07 -080098 Path to PID lock file. Used by zuul-server only.
Clark Boylan9b670902012-09-28 13:47:56 -070099 ``pidfile=/var/run/zuul/zuul.pid``
100
101**state_dir**
James E. Blair4076e2b2014-01-28 12:42:20 -0800102 Path to directory that Zuul should save state to. Used by all Zuul
103 commands.
Clark Boylan9b670902012-09-28 13:47:56 -0700104 ``state_dir=/var/lib/zuul``
105
James E. Blair4076e2b2014-01-28 12:42:20 -0800106**report_times**
107 Boolean value (``true`` or ``false``) that determines if Zuul should
108 include elapsed times for each job in the textual report. Used by
109 zuul-server only.
110 ``report_times=true``
111
112**status_url**
113 URL that will be posted in Zuul comments made to Gerrit changes when
114 starting jobs for a change. Used by zuul-server only.
115 ``status_url=https://zuul.example.com/status``
116
Clark Boylane0b4bdb2014-06-03 17:01:25 -0700117**status_expiry**
118 Zuul will cache the status.json file for this many seconds. This is an
119 optional value and ``1`` is used by default.
120 ``status_expiry=1``
121
James E. Blair4076e2b2014-01-28 12:42:20 -0800122**url_pattern**
123 If you are storing build logs external to the system that originally
124 ran jobs and wish to link to those logs when Zuul makes comments on
125 Gerrit changes for completed jobs this setting configures what the
126 URLs for those links should be. Used by zuul-server only.
127 ``http://logs.example.com/{change.number}/{change.patchset}/{pipeline.name}/{job.name}/{build.number}``
128
129**job_name_in_report**
130 Boolean value (``true`` or ``false``) that indicates whether the
131 job name should be included in the report (normally only the URL
132 is included). Defaults to ``false``. Used by zuul-server only.
133 ``job_name_in_report=true``
134
135merger
136""""""
137
Clark Boylan9b670902012-09-28 13:47:56 -0700138**git_dir**
139 Directory that Zuul should clone local git repositories to.
140 ``git_dir=/var/lib/zuul/git``
141
Paul Belangerb67aba12013-05-13 19:22:14 -0400142**git_user_email**
143 Optional: Value to pass to `git config user.email`.
144 ``git_user_email=zuul@example.com``
145
146**git_user_name**
147 Optional: Value to pass to `git config user.name`.
148 ``git_user_name=zuul``
149
Arx Cruzb1b010d2013-10-28 19:49:59 -0200150**zuul_url**
James E. Blair4076e2b2014-01-28 12:42:20 -0800151 URL of this merger's git repos, accessible to test workers. Usually
152 "http://zuul.example.com/p" or "http://zuul-merger01.example.com/p"
153 depending on whether the merger is co-located with the Zuul server.
Arx Cruzb1b010d2013-10-28 19:49:59 -0200154
James E. Blaira4430132014-02-17 08:32:07 -0800155**log_config**
156 Path to log config file for the merger process.
157 ``log_config=/etc/zuul/logging.yaml``
158
159**pidfile**
160 Path to PID lock file for the merger process.
161 ``pidfile=/var/run/zuul-merger/merger.pid``
162
Joshua Hesketh5fea8672013-08-19 17:32:01 +1000163smtp
164""""
165
166**server**
167 SMTP server hostname or address to use.
168 ``server=localhost``
169
Akihiro Motokic7d28a72014-03-22 09:21:38 +0900170**port**
171 Optional: SMTP server port.
172 ``port=25``
173
Joshua Hesketh5fea8672013-08-19 17:32:01 +1000174**default_from**
175 Who the email should appear to be sent from when emailing the report.
176 This can be overridden by individual pipelines.
177 ``default_from=zuul@example.com``
178
179**default_to**
180 Who the report should be emailed to by default.
181 This can be overridden by individual pipelines.
182 ``default_to=you@example.com``
183
Joshua Hesketh36c3fa52014-01-22 11:40:52 +1100184.. _swift:
185
186swift
187"""""
188
189To send (optional) swift upload instructions this section must be
190present. Multiple destinations can be defined in the :ref:`jobs`
191section of the layout.
192
193**authurl**
194 The (keystone) Auth URL for swift
195 ``For example, https://identity.api.rackspacecloud.com/v2.0/``
196
197Any of the `swiftclient connection parameters`_ can also be defined
198here by the same name. Including the os_options by their key name (
199``for example tenant_id``)
200
201.. _swiftclient connection parameters: http://docs.openstack.org/developer/python-swiftclient/swiftclient.html#module-swiftclient.client
202
203**X-Account-Meta-Temp-Url-Key** (optional)
204 This is the key used to sign the HMAC message. zuul will send the
205 key to swift for you so you only need to define it here. If you do
206 not set a key zuul will generate one automatically.
207
208**region_name** (optional)
209 The region name holding the swift container
210 ``For example, SYD``
211
212Each destination defined by the :ref:`jobs` will have the following
213default values that it may overwrite.
214
215**default_container** (optional)
216 Container name to place the log into
217 ``For example, logs``
218
219**default_expiry** (optional)
220 How long the signed destination should be available for
221 ``default: 7200 (2hrs)``
222
223**default_max_file_size** (optional)
224 The maximum size of an individual file
225 ``default: 104857600 (100MB)``
226
227**default_max_file_count** (optional)
228 The maximum number of separate files to allow
229 ``default: 10``
230
231**default_logserver_prefix**
232 Provide a URL to the CDN or logserver app so that a worker knows
233 what URL to return. The worker should return the logserver_prefix
234 url and the object path.
235 ``For example: http://logs.example.org/server.app?obj=``
236
James E. Blaircdd00072012-06-08 19:17:28 -0700237layout.yaml
238~~~~~~~~~~~
239
Clark Boylan00635dc2012-09-19 14:03:08 -0700240This is the main configuration file for Zuul, where all of the pipelines
James E. Blaircdd00072012-06-08 19:17:28 -0700241and projects are defined, what tests should be run, and what actions
Clark Boylan00635dc2012-09-19 14:03:08 -0700242Zuul should perform. There are three sections: pipelines, jobs, and
James E. Blaircdd00072012-06-08 19:17:28 -0700243projects.
244
James E. Blaire5a847f2012-07-10 15:29:14 -0700245.. _includes:
246
247Includes
248""""""""
249
250Custom functions to be used in Zuul's configuration may be provided
251using the ``includes`` directive. It accepts a list of files to
252include, and currently supports one type of inclusion, a python file::
253
254 includes:
255 - python-file: local_functions.py
256
257**python-file**
258 The path to a python file. The file will be loaded and objects that
259 it defines will be placed in a special environment which can be
260 referenced in the Zuul configuration. Currently only the
261 parameter-function attribute of a Job uses this feature.
262
Clark Boylan00635dc2012-09-19 14:03:08 -0700263Pipelines
264"""""""""
James E. Blaircdd00072012-06-08 19:17:28 -0700265
Clark Boylan00635dc2012-09-19 14:03:08 -0700266Zuul can have any number of independent pipelines. Whenever a matching
267Gerrit event is found for a pipeline, that event is added to the
268pipeline, and the jobs specified for that pipeline are run. When all
269jobs specified for the pipeline that were triggered by an event are
270completed, Zuul reports back to Gerrit the results.
James E. Blaircdd00072012-06-08 19:17:28 -0700271
Clark Boylan00635dc2012-09-19 14:03:08 -0700272There are no pre-defined pipelines in Zuul, rather you can define
273whatever pipelines you need in the layout file. This is a very flexible
274system that can accommodate many kinds of workflows.
James E. Blaircdd00072012-06-08 19:17:28 -0700275
Clark Boylan00635dc2012-09-19 14:03:08 -0700276Here is a quick example of a pipeline definition followed by an
James E. Blaircdd00072012-06-08 19:17:28 -0700277explanation of each of the parameters::
278
279 - name: check
Clark Boylan00635dc2012-09-19 14:03:08 -0700280 manager: IndependentPipelineManager
James E. Blaircdd00072012-06-08 19:17:28 -0700281 trigger:
James E. Blair6c358e72013-07-29 17:06:47 -0700282 gerrit:
283 - event: patchset-created
James E. Blaircdd00072012-06-08 19:17:28 -0700284 success:
285 verified: 1
286 failure:
287 verified: -1
288
289**name**
290 This is used later in the project definition to indicate what jobs
Clark Boylan00635dc2012-09-19 14:03:08 -0700291 should be run for events in the pipeline.
James E. Blaircdd00072012-06-08 19:17:28 -0700292
James E. Blair8dbd56a2012-12-22 10:55:10 -0800293**description**
294 This is an optional field that may be used to provide a textual
295 description of the pipeline.
296
James E. Blair56370192013-01-14 15:47:28 -0800297**success-message**
298 An optional field that supplies the introductory text in message
299 reported back to Gerrit when all the voting builds are successful.
300 Defaults to "Build successful."
301
302**failure-message**
303 An optional field that supplies the introductory text in message
304 reported back to Gerrit when at least one voting build fails.
305 Defaults to "Build failed."
306
Joshua Heskethb7179772014-01-30 23:30:46 +1100307**merge-failure-message**
308 An optional field that supplies the introductory text in message
309 reported back to Gerrit when a change fails to merge with the
310 current state of the repository.
311 Defaults to "Merge failed."
312
Joshua Hesketh3979e3e2014-03-04 11:21:10 +1100313**footer-message**
314 An optional field to supply additional information after test results.
315 Useful for adding information about the CI system such as debugging
316 and contact details.
317
James E. Blaircdd00072012-06-08 19:17:28 -0700318**manager**
Clark Boylan00635dc2012-09-19 14:03:08 -0700319 There are currently two schemes for managing pipelines:
James E. Blaircdd00072012-06-08 19:17:28 -0700320
Clark Boylan00635dc2012-09-19 14:03:08 -0700321 *IndependentPipelineManager*
322 Every event in this pipeline should be treated as independent of
323 other events in the pipeline. This is appropriate when the order of
324 events in the pipeline doesn't matter because the results of the
325 actions this pipeline performs can not affect other events in the
326 pipeline. For example, when a change is first uploaded for review,
James E. Blaircdd00072012-06-08 19:17:28 -0700327 you may want to run tests on that change to provide early feedback
328 to reviewers. At the end of the tests, the change is not going to
329 be merged, so it is safe to run these tests in parallel without
Clark Boylan00635dc2012-09-19 14:03:08 -0700330 regard to any other changes in the pipeline. They are independent.
James E. Blaircdd00072012-06-08 19:17:28 -0700331
Clark Boylan00635dc2012-09-19 14:03:08 -0700332 Another type of pipeline that is independent is a post-merge
333 pipeline. In that case, the changes have already merged, so the
334 results can not affect any other events in the pipeline.
James E. Blaircdd00072012-06-08 19:17:28 -0700335
Clark Boylan00635dc2012-09-19 14:03:08 -0700336 *DependentPipelineManager*
337 The dependent pipeline manager is designed for gating. It ensures
James E. Blaircdd00072012-06-08 19:17:28 -0700338 that every change is tested exactly as it is going to be merged
339 into the repository. An ideal gating system would test one change
340 at a time, applied to the tip of the repository, and only if that
341 change passed tests would it be merged. Then the next change in
342 line would be tested the same way. In order to achieve parallel
Clark Boylan00635dc2012-09-19 14:03:08 -0700343 testing of changes, the dependent pipeline manager performs
James E. Blaircdd00072012-06-08 19:17:28 -0700344 speculative execution on changes. It orders changes based on
Clark Boylan00635dc2012-09-19 14:03:08 -0700345 their entry into the pipeline. It begins testing all changes in
346 parallel, assuming that each change ahead in the pipeline will pass
James E. Blaircdd00072012-06-08 19:17:28 -0700347 its tests. If they all succeed, all the changes can be tested and
Clark Boylan00635dc2012-09-19 14:03:08 -0700348 merged in parallel. If a change near the front of the pipeline
349 fails its tests, each change behind it ignores whatever tests have
350 been completed and are tested again without the change in front.
351 This way gate tests may run in parallel but still be tested
352 correctly, exactly as they will appear in the repository when
353 merged.
James E. Blaircdd00072012-06-08 19:17:28 -0700354
Clark Boylan00635dc2012-09-19 14:03:08 -0700355 One important characteristic of the DependentPipelineManager is that
James E. Blaircdd00072012-06-08 19:17:28 -0700356 it analyzes the jobs that are triggered by different projects, and
357 if those projects have jobs in common, it treats those projects as
358 related, and they share a single virtual queue of changes. Thus,
359 if there is a job that performs integration testing on two
360 projects, those two projects will automatically share a virtual
361 change queue. If a third project does not invoke that job, it
Clark Boylan00635dc2012-09-19 14:03:08 -0700362 will be part of a separate virtual change queue, and changes to
363 it will not depend on changes to the first two jobs.
James E. Blaircdd00072012-06-08 19:17:28 -0700364
365 For more detail on the theory and operation of Zuul's
Clark Boylan00635dc2012-09-19 14:03:08 -0700366 DependentPipelineManager, see: :doc:`gating`.
James E. Blaircdd00072012-06-08 19:17:28 -0700367
368**trigger**
James E. Blair6c358e72013-07-29 17:06:47 -0700369 Exactly one trigger source must be supplied for each pipeline.
James E. Blaircdd00072012-06-08 19:17:28 -0700370 Triggers are not exclusive -- matching events may be placed in
James E. Blair6c358e72013-07-29 17:06:47 -0700371 multiple pipelines, and they will behave independently in each of
372 the pipelines they match. You may select from the following:
James E. Blaircdd00072012-06-08 19:17:28 -0700373
James E. Blair6c358e72013-07-29 17:06:47 -0700374 **gerrit**
375 This describes what Gerrit events should be placed in the
376 pipeline. Multiple gerrit triggers may be listed. Further
377 parameters describe the kind of events that match:
James E. Blaircdd00072012-06-08 19:17:28 -0700378
James E. Blair6c358e72013-07-29 17:06:47 -0700379 *event*
380 The event name from gerrit. Examples: ``patchset-created``,
381 ``comment-added``, ``ref-updated``. This field is treated as a
382 regular expression.
James E. Blaircdd00072012-06-08 19:17:28 -0700383
James E. Blair6c358e72013-07-29 17:06:47 -0700384 *branch*
385 The branch associated with the event. Example: ``master``. This
386 field is treated as a regular expression, and multiple branches may
387 be listed.
James E. Blaircdd00072012-06-08 19:17:28 -0700388
James E. Blair6c358e72013-07-29 17:06:47 -0700389 *ref*
390 On ref-updated events, the branch parameter is not used, instead the
391 ref is provided. Currently Gerrit has the somewhat idiosyncratic
392 behavior of specifying bare refs for branch names (e.g., ``master``),
393 but full ref names for other kinds of refs (e.g., ``refs/tags/foo``).
394 Zuul matches what you put here exactly against what Gerrit
395 provides. This field is treated as a regular expression, and
396 multiple refs may be listed.
James E. Blaircdd00072012-06-08 19:17:28 -0700397
James E. Blair6c358e72013-07-29 17:06:47 -0700398 *approval*
399 This is only used for ``comment-added`` events. It only matches if
400 the event has a matching approval associated with it. Example:
401 ``code-review: 2`` matches a ``+2`` vote on the code review category.
402 Multiple approvals may be listed.
Antoine Mussob4e809e2012-12-06 16:58:06 +0100403
James E. Blair6c358e72013-07-29 17:06:47 -0700404 *email_filter*
405 This is used for any event. It takes a regex applied on the performer
Michael Prokop526926a2013-10-24 16:16:57 +0200406 email, i.e. Gerrit account email address. If you want to specify
James E. Blair6c358e72013-07-29 17:06:47 -0700407 several email filters, you must use a YAML list. Make sure to use non
408 greedy matchers and to escapes dots!
409 Example: ``email_filter: ^.*?@example\.org$``.
410
Joshua Heskethb8a817e2013-12-27 11:21:38 +1100411 *username_filter*
412 This is used for any event. It takes a regex applied on the performer
413 username, i.e. Gerrit account name. If you want to specify several
414 username filters, you must use a YAML list. Make sure to use non greedy
415 matchers and to escapes dots!
416 Example: ``username_filter: ^jenkins$``.
417
James E. Blair6c358e72013-07-29 17:06:47 -0700418 *comment_filter*
419 This is only used for ``comment-added`` events. It accepts a list of
420 regexes that are searched for in the comment string. If any of these
421 regexes matches a portion of the comment string the trigger is
422 matched. ``comment_filter: retrigger`` will match when comments
423 containing 'retrigger' somewhere in the comment text are added to a
424 change.
Clark Boylanb9bcb402012-06-29 17:44:05 -0700425
James E. Blair11041d22014-05-02 14:49:53 -0700426 *require-approval* (deprecated)
James E. Blairc053d022014-01-22 14:57:33 -0800427 This may be used for any event. It requires that a certain kind
428 of approval be present for the current patchset of the change (the
James E. Blair11041d22014-05-02 14:49:53 -0700429 approval could be added by the event in question). It follows the
430 same syntax as the "approval" pipeline requirement below. This
431 form should be considered deprecated and the pipeline requirement
432 used instead.
James E. Blairc053d022014-01-22 14:57:33 -0800433
James E. Blair63bb0ef2013-07-29 17:14:51 -0700434 **timer**
435 This trigger will run based on a cron-style time specification.
436 It will enqueue an event into its pipeline for every project
437 defined in the configuration. Any job associated with the
438 pipeline will run in response to that event.
439
440 *time*
441 The time specification in cron syntax. Only the 5 part syntax is
442 supported, not the symbolic names. Example: ``0 0 * * *`` runs
443 at midnight.
444
James E. Blair11041d22014-05-02 14:49:53 -0700445**require**
446 If this section is present, it established pre-requisites for any
447 kind of item entering the Pipeline. Regardless of how the item is
448 to be enqueued (via any trigger or automatic dependency resolution),
449 the conditions specified here must be met or the item will not be
450 enqueued.
451
452 **approval**
453 This requires that a certain kind of approval be present for the
454 current patchset of the change (the approval could be added by the
455 event in question). It takes several sub-parameters, all of which
456 are optional and are combined together so that there must be an
457 approval matching all specified requirements.
458
459 *username*
460 If present, an approval from this username is required.
461
462 *email-filter*
463 If present, an approval with this email address is required. It
464 is treated as a regular expression as above.
465
466 *older-than*
467 If present, the approval must be older than this amount of time
468 to match. Provide a time interval as a number with a suffix of
469 "w" (weeks), "d" (days), "h" (hours), "m" (minutes), "s"
470 (seconds). Example ``48h`` or ``2d``.
471
472 *newer-than*
473 If present, the approval must be newer than this amount of time
474 to match. Same format as "older-than".
475
476 Any other field is interpreted as a review category and value
477 pair. For example ``verified: 1`` would require that the approval
478 be for a +1 vote in the "Verified" column.
479
480 **open**
481 A boolean value (``true`` or ``false``) that indicates whether the change
482 must be open or closed in order to be enqueued.
483
Clark Boylana9702ad2014-05-08 17:17:24 -0700484 **current-patchset**
485 A boolean value (``true`` or ``false``) that indicates whether the change
486 must be the current patchset in order to be enqueued.
487
James E. Blair11041d22014-05-02 14:49:53 -0700488 **status**
489 A string value that corresponds with the status of the change
490 reported by the trigger. For example, when using the Gerrit
491 trigger, status values such as ``NEW`` or ``MERGED`` may be useful.
James E. Blair63bb0ef2013-07-29 17:14:51 -0700492
James E. Blair2fa50962013-01-30 21:50:41 -0800493**dequeue-on-new-patchset**
494 Normally, if a new patchset is uploaded to a change that is in a
495 pipeline, the existing entry in the pipeline will be removed (with
496 jobs canceled and any dependent changes that can no longer merge as
497 well. To suppress this behavior (and allow jobs to continue
498 running), set this to ``false``. Default: ``true``.
499
James E. Blaircdd00072012-06-08 19:17:28 -0700500**success**
Joshua Hesketh1879cf72013-08-19 14:13:15 +1000501 Describes where Zuul should report to if all the jobs complete
502 successfully.
James E. Blaircdd00072012-06-08 19:17:28 -0700503 This section is optional; if it is omitted, Zuul will run jobs and
504 do nothing on success; it will not even report a message to Gerrit.
Joshua Hesketh1879cf72013-08-19 14:13:15 +1000505 If the section is present, the listed reporter plugins will be
506 asked to report on the jobs.
507 Each reporter's value dictionary is handled by the reporter. See
508 reporters for more details.
James E. Blaircdd00072012-06-08 19:17:28 -0700509
Mathieu Gagnéd6d2a642013-06-11 20:59:58 -0400510**failure**
James E. Blaircdd00072012-06-08 19:17:28 -0700511 Uses the same syntax as **success**, but describes what Zuul should
512 do if at least one job fails.
James E. Blairdc253862012-06-13 17:12:42 -0700513
Joshua Heskethb7179772014-01-30 23:30:46 +1100514**merge-failure**
515 Uses the same syntax as **success**, but describes what Zuul should
516 do if it is unable to merge in the patchset. If no merge-failure
517 reporters are listed then the ``failure`` reporters will be used to
518 notify of unsuccessful merges.
519
Mathieu Gagnéd6d2a642013-06-11 20:59:58 -0400520**start**
James E. Blairdc253862012-06-13 17:12:42 -0700521 Uses the same syntax as **success**, but describes what Zuul should
Clark Boylan00635dc2012-09-19 14:03:08 -0700522 do when a change is added to the pipeline manager. This can be used,
James E. Blairdc253862012-06-13 17:12:42 -0700523 for example, to reset the value of the Verified review category.
Mathieu Gagnéd6d2a642013-06-11 20:59:58 -0400524
James E. Blair64ed6f22013-07-10 14:07:23 -0700525**precedence**
526 Indicates how the build scheduler should prioritize jobs for
527 different pipelines. Each pipeline may have one precedence, jobs
528 for pipelines with a higher precedence will be run before ones with
529 lower. The value should be one of ``high``, ``normal``, or ``low``.
530 Default: ``normal``.
531
Clark Boylanc2d19e42014-01-23 14:08:58 -0800532**window**
533 DependentPipelineManagers only. Zuul can rate limit
534 DependentPipelineManagers in a manner similar to TCP flow control.
535 Jobs are only started for changes in the queue if they sit in the
536 actionable window for the pipeline. The initial length of this window
537 is configurable with this value. The value given should be a positive
538 integer value. A value of ``0`` disables rate limiting on the
539 DependentPipelineManager.
540 Default: ``20``.
541
542**window-floor**
543 DependentPipelineManagers only. This is the minimum value for the
544 window described above. Should be a positive non zero integer value.
545 Default: ``3``.
546
547**window-increase-type**
548 DependentPipelineManagers only. This value describes how the window
549 should grow when changes are successfully merged by zuul. A value of
550 ``linear`` indicates that ``window-increase-factor`` should be added
551 to the previous window value. A value of ``exponential`` indicates
552 that ``window-increase-factor`` should be multiplied against the
553 previous window value and the result will become the window size.
554 Default: ``linear``.
555
556**window-increase-factor**
557 DependentPipelineManagers only. The value to be added or mulitplied
558 against the previous window value to determine the new window after
559 successful change merges.
560 Default: ``1``.
561
562**window-decrease-type**
563 DependentPipelineManagers only. This value describes how the window
564 should shrink when changes are not able to be merged by Zuul. A value
565 of ``linear`` indicates that ``window-decrease-factor`` should be
566 subtracted from the previous window value. A value of ``exponential``
567 indicates that ``window-decrease-factor`` should be divided against
568 the previous window value and the result will become the window size.
569 Default: ``exponential``.
570
571**window-decrease-factor**
572 DependentPipelineManagers only. The value to be subtracted or divided
573 against the previous window value to determine the new window after
574 unsuccessful change merges.
575 Default: ``2``.
576
Clark Boylan00635dc2012-09-19 14:03:08 -0700577Some example pipeline configurations are included in the sample layout
578file. The first is called a *check* pipeline::
James E. Blaircdd00072012-06-08 19:17:28 -0700579
580 - name: check
Clark Boylan00635dc2012-09-19 14:03:08 -0700581 manager: IndependentPipelineManager
James E. Blaircdd00072012-06-08 19:17:28 -0700582 trigger:
583 - event: patchset-created
584 success:
Joshua Hesketh1879cf72013-08-19 14:13:15 +1000585 gerrit:
586 verified: 1
James E. Blaircdd00072012-06-08 19:17:28 -0700587 failure:
Joshua Hesketh1879cf72013-08-19 14:13:15 +1000588 gerrit:
589 verified: -1
James E. Blaircdd00072012-06-08 19:17:28 -0700590
591This will trigger jobs each time a new patchset (or change) is
592uploaded to Gerrit, and report +/-1 values to Gerrit in the
593``verified`` review category. ::
594
595 - name: gate
Clark Boylan00635dc2012-09-19 14:03:08 -0700596 manager: DependentPipelineManager
James E. Blaircdd00072012-06-08 19:17:28 -0700597 trigger:
598 - event: comment-added
599 approval:
600 - approved: 1
601 success:
Joshua Hesketh1879cf72013-08-19 14:13:15 +1000602 gerrit:
603 verified: 2
604 submit: true
James E. Blaircdd00072012-06-08 19:17:28 -0700605 failure:
Joshua Hesketh1879cf72013-08-19 14:13:15 +1000606 gerrit:
607 verified: -2
James E. Blaircdd00072012-06-08 19:17:28 -0700608
609This will trigger jobs whenever a reviewer leaves a vote of ``1`` in the
610``approved`` review category in Gerrit (a non-standard category).
611Changes will be tested in such a way as to guarantee that they will be
612merged exactly as tested, though that will happen in parallel by
613creating a virtual queue of dependent changes and performing
614speculative execution of jobs. ::
615
616 - name: post
Clark Boylan00635dc2012-09-19 14:03:08 -0700617 manager: IndependentPipelineManager
James E. Blaircdd00072012-06-08 19:17:28 -0700618 trigger:
619 - event: ref-updated
620 ref: ^(?!refs/).*$
621
622This will trigger jobs whenever a change is merged to a named branch
623(e.g., ``master``). No output will be reported to Gerrit. This is
624useful for side effects such as creating per-commit tarballs. ::
625
626 - name: silent
Clark Boylan00635dc2012-09-19 14:03:08 -0700627 manager: IndependentPipelineManager
James E. Blaircdd00072012-06-08 19:17:28 -0700628 trigger:
629 - event: patchset-created
630
631This also triggers jobs when changes are uploaded to Gerrit, but no
632results are reported to Gerrit. This is useful for jobs that are in
Antoine Mussoce333842012-10-16 14:42:35 +0200633development and not yet ready to be presented to developers. ::
634
635 pipelines:
636 - name: post-merge
637 manager: IndependentPipelineManager
638 trigger:
639 - event: change-merged
640 success:
Joshua Hesketh1879cf72013-08-19 14:13:15 +1000641 gerrit:
642 force-message: True
Antoine Mussoce333842012-10-16 14:42:35 +0200643 failure:
Joshua Hesketh1879cf72013-08-19 14:13:15 +1000644 gerrit:
645 force-message: True
Antoine Mussoce333842012-10-16 14:42:35 +0200646
647The ``change-merged`` events happen when a change has been merged in the git
648repository. The change is thus closed and Gerrit will not accept modifications
649to the review scoring such as ``code-review`` or ``verified``. By using the
650``force-message: True`` parameter, Zuul will pass ``--force-message`` to the
651``gerrit review`` command, thus making sure the message is actually
652sent back to Gerrit regardless of approval scores.
653That kind of pipeline is nice to run regression or performance tests.
654
655.. note::
656 The ``change-merged`` event does not include the commit sha1 which can be
657 hazardous, it would let you report back to Gerrit though. If you were to
Michael Prokop526926a2013-10-24 16:16:57 +0200658 build a tarball for a specific commit, you should consider instead using
Łukasz Jernaś048acb42014-03-02 18:49:41 +0100659 the ``ref-updated`` event which does include the commit sha1 (but lacks the
Antoine Mussoce333842012-10-16 14:42:35 +0200660 Gerrit change number).
James E. Blaircdd00072012-06-08 19:17:28 -0700661
Joshua Hesketh36c3fa52014-01-22 11:40:52 +1100662
663.. _jobs:
664
James E. Blaircdd00072012-06-08 19:17:28 -0700665Jobs
666""""
667
668The jobs section is optional, and can be used to set attributes of
669jobs that are independent of their association with a project. For
670example, if a job should return a customized message on failure, that
671may be specified here. Otherwise, Zuul does not need to be told about
672each job as it builds a list from the project specification.
673
674**name**
675 The name of the job. This field is treated as a regular expression
676 and will be applied to each job that matches.
677
James E. Blairc8a1e052014-02-25 09:29:26 -0800678**queue-name (optional)**
679 Zuul will automatically combine projects that share a job into
680 shared change queues for dependent pipeline managers. In order to
681 report statistics about these queues, it is convenient for them to
682 have names. Zuul can automatically name change queues, however
683 these can grow quite long and are prone to changing as projects in
684 the queue change. If you assign a queue-name to a job, Zuul will
685 use that as the name for the shared change queue that contains that
686 job instead of the automatically generated one. It is an error for
687 a shared change queue to have more than one job with a queue-name if
688 they are not the same.
689
James E. Blaire5a847f2012-07-10 15:29:14 -0700690**failure-message (optional)**
691 The message that should be reported to Gerrit if the job fails.
James E. Blaircdd00072012-06-08 19:17:28 -0700692
James E. Blaire5a847f2012-07-10 15:29:14 -0700693**success-message (optional)**
694 The message that should be reported to Gerrit if the job fails.
James E. Blaircdd00072012-06-08 19:17:28 -0700695
James E. Blair6aea36d2012-12-17 13:03:24 -0800696**failure-pattern (optional)**
697 The URL that should be reported to Gerrit if the job fails.
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700698 Defaults to the build URL or the url_pattern configured in
James E. Blair6aea36d2012-12-17 13:03:24 -0800699 zuul.conf. May be supplied as a string pattern with substitutions
700 as described in url_pattern in :ref:`zuulconf`.
701
702**success-pattern (optional)**
703 The URL that should be reported to Gerrit if the job succeeds.
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700704 Defaults to the build URL or the url_pattern configured in
James E. Blair6aea36d2012-12-17 13:03:24 -0800705 zuul.conf. May be supplied as a string pattern with substitutions
706 as described in url_pattern in :ref:`zuulconf`.
707
James E. Blair222d4982012-07-16 09:31:19 -0700708**hold-following-changes (optional)**
709 This is a boolean that indicates that changes that follow this
Clark Boylan00635dc2012-09-19 14:03:08 -0700710 change in a dependent change pipeline should wait until this job
James E. Blair222d4982012-07-16 09:31:19 -0700711 succeeds before launching. If this is applied to a very short job
712 that can predict whether longer jobs will fail early, this can be
713 used to reduce the number of jobs that Zuul will launch and
714 ultimately have to cancel. In that case, a small amount of
Mathieu Gagnéd6d2a642013-06-11 20:59:58 -0400715 parallelization of jobs is traded for more efficient use of testing
James E. Blair222d4982012-07-16 09:31:19 -0700716 resources. On the other hand, to apply this to a long running job
717 would largely defeat the parallelization of dependent change testing
Mathieu Gagnéd6d2a642013-06-11 20:59:58 -0400718 that is the main feature of Zuul. Default: ``false``.
James E. Blair222d4982012-07-16 09:31:19 -0700719
James E. Blaire5a847f2012-07-10 15:29:14 -0700720**branch (optional)**
James E. Blaircdd00072012-06-08 19:17:28 -0700721 This job should only be run on matching branches. This field is
722 treated as a regular expression and multiple branches may be
723 listed.
724
James E. Blair70c71582013-03-06 08:50:50 -0800725**files (optional)**
726 This job should only be run if at least one of the files involved in
727 the change (added, deleted, or modified) matches at least one of the
728 file patterns listed here. This field is treated as a regular
729 expression and multiple expressions may be listed.
730
Mathieu Gagnéd6d2a642013-06-11 20:59:58 -0400731**voting (optional)**
732 Boolean value (``true`` or ``false``) that indicates whatever
733 a job is voting or not. Default: ``true``.
734
James E. Blaire5a847f2012-07-10 15:29:14 -0700735**parameter-function (optional)**
736 Specifies a function that should be applied to the parameters before
737 the job is launched. The function should be defined in a python file
738 included with the :ref:`includes` directive. The function
739 should have the following signature:
740
James E. Blaird0470972013-07-29 10:05:43 -0700741 .. function:: parameters(item, job, parameters)
James E. Blaire5a847f2012-07-10 15:29:14 -0700742
743 Manipulate the parameters passed to a job before a build is
744 launched. The ``parameters`` dictionary will already contain the
745 standard Zuul job parameters, and is expected to be modified
746 in-place.
747
James E. Blaird78576a2013-07-09 10:39:17 -0700748 :param item: the current queue item
749 :type item: zuul.model.QueueItem
James E. Blaird0470972013-07-29 10:05:43 -0700750 :param job: the job about to be run
751 :type job: zuul.model.Job
James E. Blaire5a847f2012-07-10 15:29:14 -0700752 :param parameters: parameters to be passed to the job
753 :type parameters: dict
754
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700755 If the parameter **ZUUL_NODE** is set by this function, then it will
756 be used to specify on what node (or class of node) the job should be
757 run.
758
Joshua Hesketh36c3fa52014-01-22 11:40:52 +1100759**swift**
760 If :ref:`swift` is configured then each job can define a destination
761 container for the builder to place logs and/or assets into. Multiple
762 containers can be listed for each job by providing a unique ``name``.
763
764 *name*
765 Set an identifying name for the container. This is used in the
766 parameter key sent to the builder. For example if it ``logs`` then
767 one of the parameters sent will be ``SWIFT_logs_CONTAINER``
768 (case-sensitive).
769
770 Each of the defaults defined in :ref:`swift` can be overwritten as:
771
772 *container* (optional)
773 Container name to place the log into
774 ``For example, logs``
775
776 *expiry* (optional)
777 How long the signed destination should be available for
778
779 *max_file_size** (optional)
780 The maximum size of an individual file
781
782 *max_file_count* (optional)
783 The maximum number of separate files to allow
784
785 *logserver_prefix*
786 Provide a URL to the CDN or logserver app so that a worker knows
787 what URL to return.
788 ``For example: http://logs.example.org/server.app?obj=``
789 The worker should return the logserver_prefix url and the object
790 path as the URL in the results data packet.
791
James E. Blaircdd00072012-06-08 19:17:28 -0700792Here is an example of setting the failure message for jobs that check
793whether a change merges cleanly::
794
795 - name: ^.*-merge$
796 failure-message: This change was unable to be automatically merged
797 with the current state of the repository. Please rebase your
798 change and upload a new patchset.
799
800Projects
801""""""""
802
Clark Boylan00635dc2012-09-19 14:03:08 -0700803The projects section indicates what jobs should be run in each pipeline
James E. Blaircdd00072012-06-08 19:17:28 -0700804for events associated with each project. It contains a list of
805projects. Here is an example::
806
807 - name: example/project
808 check:
809 - project-merge:
810 - project-unittest
Clark Boylan00635dc2012-09-19 14:03:08 -0700811 - project-pep8
812 - project-pyflakes
James E. Blaircdd00072012-06-08 19:17:28 -0700813 gate:
814 - project-merge:
815 - project-unittest
Clark Boylan00635dc2012-09-19 14:03:08 -0700816 - project-pep8
817 - project-pyflakes
James E. Blaircdd00072012-06-08 19:17:28 -0700818 post:
819 - project-publish
820
821**name**
822 The name of the project (as known by Gerrit).
823
James E. Blair19deff22013-08-25 13:17:35 -0700824**merge-mode (optional)**
825 An optional value that indicates what strategy should be used to
826 merge changes to this project. Supported values are:
827
828 ** merge-resolve **
829 Equivalent to 'git merge -s resolve'. This corresponds closely to
830 what Gerrit performs (using JGit) for a project if the "Merge if
831 necessary" merge mode is selected and "Automatically resolve
832 conflicts" is checked. This is the default.
833
834 ** merge **
835 Equivalent to 'git merge'.
836
837 ** cherry-pick **
838 Equivalent to 'git cherry-pick'.
839
Clark Boylan00635dc2012-09-19 14:03:08 -0700840This is followed by a section for each of the pipelines defined above.
841Pipelines may be omitted if no jobs should run for this project in a
842given pipeline. Within the pipeline section, the jobs that should be
James E. Blaircdd00072012-06-08 19:17:28 -0700843executed are listed. If a job is entered as a dictionary key, then
844jobs contained within that key are only executed if the key job
845succeeds. In the above example, project-unittest, project-pep8, and
846project-pyflakes are only executed if project-merge succeeds. This
847can help avoid running unnecessary jobs.
848
James E. Blair18c64442014-03-18 10:14:45 -0700849The special job named ``noop`` is internal to Zuul and will always
850return ``SUCCESS`` immediately. This can be useful if you require
851that all changes be processed by a pipeline but a project has no jobs
852that can be run on it.
853
Paul Belangerffef9e42013-02-11 22:15:18 -0500854.. seealso:: The OpenStack Zuul configuration for a comprehensive example: https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/zuul/layout.yaml
James E. Blaircdd00072012-06-08 19:17:28 -0700855
Antoine Musso80edd5a2013-02-13 15:37:53 +0100856Project Templates
857"""""""""""""""""
858
Michael Prokop526926a2013-10-24 16:16:57 +0200859Whenever you have lot of similar projects (such as plugins for a project) you
Antoine Musso80edd5a2013-02-13 15:37:53 +0100860will most probably want to use the same pipeline configurations. The
861project templates let you define pipelines and job name templates to trigger.
862One can then just apply the template on its project which make it easier to
Michael Prokop526926a2013-10-24 16:16:57 +0200863update several similar projects. As an example::
Antoine Musso80edd5a2013-02-13 15:37:53 +0100864
865 project-templates:
866 # Name of the template
867 - name: plugin-triggering
868 # Definition of pipelines just like for a `project`
869 check:
870 - '{jobprefix}-merge':
871 - '{jobprefix}-pep8'
872 - '{jobprefix}-pyflakes'
873 gate:
874 - '{jobprefix}-merge':
875 - '{jobprefix}-unittest'
876 - '{jobprefix}-pep8'
877 - '{jobprefix}-pyflakes'
878
879In your projects definition, you will then apply the template using the template
880key::
881
882 projects:
883 - name: plugin/foobar
884 template:
885 - name: plugin-triggering
886 jobprefix: plugin-foobar
887
James E. Blairaea6cf62013-12-16 15:38:12 -0800888You can pass several parameters to a template. A ``parameter`` value
889will be used for expansion of ``{parameter}`` in the template
890strings. The parameter ``name`` will be automatically provided and
891will contain the short name of the project, that is the portion of the
892project name after the last ``/`` character.
James E. Blaircdd00072012-06-08 19:17:28 -0700893
James E. Blair3e98c022013-12-16 15:25:38 -0800894Multiple templates can be combined in a project, and the jobs from all
895of those templates will be added to the project. Individual jobs may
896also be added::
897
898 projects:
899 - name: plugin/foobar
900 template:
901 - name: plugin-triggering
902 jobprefix: plugin-foobar
903 - name: plugin-extras
904 jobprefix: plugin-foobar
905 check:
906 - foobar-extra-special-job
907
908The order of the jobs listed in the project (which only affects the
909order of jobs listed on the report) will be the jobs from each
910template in the order listed, followed by any jobs individually listed
911for the project.
912
913Note that if multiple templates are used for a project and one
914template specifies a job that is also specified in another template,
James E. Blair12a92b12014-03-26 11:54:53 -0700915or specified in the project itself, the configuration defined by
916either the last template or the project itself will take priority.
James E. Blair3e98c022013-12-16 15:25:38 -0800917
James E. Blaircdd00072012-06-08 19:17:28 -0700918logging.conf
919~~~~~~~~~~~~
920This file is optional. If provided, it should be a standard
921:mod:`logging.config` module configuration file. If not present, Zuul will
922output all log messages of DEBUG level or higher to the console.
923
924Starting Zuul
925-------------
926
927To start Zuul, run **zuul-server**::
928
Antoine Mussob3aa8282013-04-19 15:16:59 +0200929 usage: zuul-server [-h] [-c CONFIG] [-l LAYOUT] [-d] [-t] [--version]
James E. Blaircdd00072012-06-08 19:17:28 -0700930
931 Project gating system.
932
933 optional arguments:
934 -h, --help show this help message and exit
935 -c CONFIG specify the config file
Antoine Mussob3aa8282013-04-19 15:16:59 +0200936 -l LAYOUT specify the layout file
James E. Blaircdd00072012-06-08 19:17:28 -0700937 -d do not run as a daemon
Antoine Mussob3aa8282013-04-19 15:16:59 +0200938 -t validate layout file syntax
939 --version show zuul version
James E. Blaircdd00072012-06-08 19:17:28 -0700940
941You may want to use the ``-d`` argument while you are initially setting
942up Zuul so you can detect any configuration errors quickly. Under
943normal operation, omit ``-d`` and let Zuul run as a daemon.
944
945If you send signal 1 (SIGHUP) to the zuul-server process, Zuul will
946stop executing new jobs, wait until all executing jobs are finished,
947reload its configuration, and resume. Any values in any of the
948configuration files may be changed, except the location of Zuul's PID
949file (a change to that will be ignored until Zuul is restarted).
Clark Boylanf231fa22013-02-08 12:28:53 -0800950
951If you send a SIGUSR1 to the zuul-server process, Zuul will stop
952executing new jobs, wait until all executing jobs are finished,
953then exit. While waiting to exit Zuul will queue Gerrit events and
954save these events prior to exiting. When Zuul starts again it will
955read these saved events and act on them.
Jeremy Stanley93e05f42013-03-08 17:29:17 +0000956
Michael Prokop526926a2013-10-24 16:16:57 +0200957If you need to abort Zuul and intend to manually requeue changes for
Jeremy Stanley93e05f42013-03-08 17:29:17 +0000958jobs which were running in its pipelines, prior to terminating you can
959use the zuul-changes.py tool script to simplify the process. For
960example, this would give you a list of Gerrit commands to reverify or
961recheck changes for the gate and check pipelines respectively::
962
963 ./tools/zuul-changes.py --review-host=review.openstack.org \
964 http://zuul.openstack.org/ gate 'reverify no bug'
965 ./tools/zuul-changes.py --review-host=review.openstack.org \
966 http://zuul.openstack.org/ check 'recheck no bug'
Clark Boylanfba9b242013-08-20 10:11:17 -0700967
968If you send a SIGUSR2 to the zuul-server process, Zuul will dump a stack
969trace for each running thread into its debug log. This is useful for
970tracking down deadlock or otherwise slow threads.