blob: ef6259c4e39fcf493a84df4512b3b5afe8838bac [file] [log] [blame]
James E. Blaircdd00072012-06-08 19:17:28 -07001:title: Zuul
2
3Zuul
4====
5
6Configuration
7-------------
8
9Zuul has three configuration files:
10
11**zuul.conf**
James E. Blair1f4c2bb2013-04-26 08:40:46 -070012 Connection information for Gerrit and Gearman, locations of the
Łukasz Jernaś048acb42014-03-02 18:49:41 +010013 other config files.
James E. Blaircdd00072012-06-08 19:17:28 -070014**layout.yaml**
Łukasz Jernaś048acb42014-03-02 18:49:41 +010015 Project and pipeline configuration -- what Zuul does.
James E. Blaircdd00072012-06-08 19:17:28 -070016**logging.conf**
Łukasz Jernaś048acb42014-03-02 18:49:41 +010017 Python logging config.
James E. Blaircdd00072012-06-08 19:17:28 -070018
19Examples of each of the three files can be found in the etc/ directory
20of the source distribution.
21
James E. Blair6aea36d2012-12-17 13:03:24 -080022.. _zuulconf:
23
James E. Blaircdd00072012-06-08 19:17:28 -070024zuul.conf
25~~~~~~~~~
26
27Zuul will look for ``/etc/zuul/zuul.conf`` or ``~/zuul.conf`` to
28bootstrap its configuration. Alternately, you may specify ``-c
29/path/to/zuul.conf`` on the command line.
30
James E. Blair1f4c2bb2013-04-26 08:40:46 -070031Gerrit and Gearman connection information are each described in a
32section of zuul.conf. The location of the other two configuration
33files (as well as the location of the PID file when running Zuul as a
34server) are specified in a third section.
James E. Blaircdd00072012-06-08 19:17:28 -070035
Clark Boylan9b670902012-09-28 13:47:56 -070036The three sections of this config and their options are documented below.
37You can also find an example zuul.conf file in the git
38`repository
K Jonathan Harker97e457e2013-02-26 13:29:38 -080039<https://github.com/openstack-infra/zuul/blob/master/etc/zuul.conf-sample>`_
Clark Boylan9b670902012-09-28 13:47:56 -070040
James E. Blair1f4c2bb2013-04-26 08:40:46 -070041gearman
Clark Boylan9b670902012-09-28 13:47:56 -070042"""""""
43
44**server**
James E. Blair1f4c2bb2013-04-26 08:40:46 -070045 Hostname or IP address of the Gearman server.
46 ``server=gearman.example.com``
Clark Boylan9b670902012-09-28 13:47:56 -070047
James E. Blair1f4c2bb2013-04-26 08:40:46 -070048**port**
Łukasz Jernaś048acb42014-03-02 18:49:41 +010049 Port on which the Gearman server is listening.
James E. Blair1f4c2bb2013-04-26 08:40:46 -070050 ``port=4730``
Clark Boylan9b670902012-09-28 13:47:56 -070051
James E. Blair77cc7b82013-07-15 13:22:37 -070052gearman_server
53""""""""""""""
54
55**start**
56 Whether to start the internal Gearman server (default: False).
57 ``start=true``
58
59**log_config**
60 Path to log config file for internal Gearman server.
61 ``log_config=/etc/zuul/gearman-logging.yaml``
62
Clark Boylan9b670902012-09-28 13:47:56 -070063gerrit
64""""""
65
66**server**
67 FQDN of Gerrit server.
68 ``server=review.example.com``
69
Akihiro Motokic7d28a72014-03-22 09:21:38 +090070**port**
71 Optional: Gerrit server port.
72 ``port=29418``
73
Antoine Musso27475012012-11-26 09:53:01 +010074**baseurl**
75 Optional: path to Gerrit web interface. Defaults to ``https://<value
76 of server>/``. ``baseurl=https://review.example.com/review_site/``
77
Clark Boylan9b670902012-09-28 13:47:56 -070078**user**
79 User name to use when logging into above server via ssh.
James E. Blair1f4c2bb2013-04-26 08:40:46 -070080 ``user=zuul``
Clark Boylan9b670902012-09-28 13:47:56 -070081
82**sshkey**
83 Path to SSH key to use when logging into above server.
James E. Blair1f4c2bb2013-04-26 08:40:46 -070084 ``sshkey=/home/zuul/.ssh/id_rsa``
Clark Boylan9b670902012-09-28 13:47:56 -070085
86zuul
87""""
88
89**layout_config**
James E. Blair4076e2b2014-01-28 12:42:20 -080090 Path to layout config file. Used by zuul-server only.
Clark Boylan9b670902012-09-28 13:47:56 -070091 ``layout_config=/etc/zuul/layout.yaml``
92
93**log_config**
James E. Blaira4430132014-02-17 08:32:07 -080094 Path to log config file. Used by zuul-server only.
Clark Boylan9b670902012-09-28 13:47:56 -070095 ``log_config=/etc/zuul/logging.yaml``
96
97**pidfile**
James E. Blaira4430132014-02-17 08:32:07 -080098 Path to PID lock file. Used by zuul-server only.
Clark Boylan9b670902012-09-28 13:47:56 -070099 ``pidfile=/var/run/zuul/zuul.pid``
100
101**state_dir**
James E. Blair4076e2b2014-01-28 12:42:20 -0800102 Path to directory that Zuul should save state to. Used by all Zuul
103 commands.
Clark Boylan9b670902012-09-28 13:47:56 -0700104 ``state_dir=/var/lib/zuul``
105
James E. Blair4076e2b2014-01-28 12:42:20 -0800106**report_times**
107 Boolean value (``true`` or ``false``) that determines if Zuul should
108 include elapsed times for each job in the textual report. Used by
109 zuul-server only.
110 ``report_times=true``
111
112**status_url**
113 URL that will be posted in Zuul comments made to Gerrit changes when
114 starting jobs for a change. Used by zuul-server only.
115 ``status_url=https://zuul.example.com/status``
116
117**url_pattern**
118 If you are storing build logs external to the system that originally
119 ran jobs and wish to link to those logs when Zuul makes comments on
120 Gerrit changes for completed jobs this setting configures what the
121 URLs for those links should be. Used by zuul-server only.
122 ``http://logs.example.com/{change.number}/{change.patchset}/{pipeline.name}/{job.name}/{build.number}``
123
124**job_name_in_report**
125 Boolean value (``true`` or ``false``) that indicates whether the
126 job name should be included in the report (normally only the URL
127 is included). Defaults to ``false``. Used by zuul-server only.
128 ``job_name_in_report=true``
129
130merger
131""""""
132
Clark Boylan9b670902012-09-28 13:47:56 -0700133**git_dir**
134 Directory that Zuul should clone local git repositories to.
135 ``git_dir=/var/lib/zuul/git``
136
Paul Belangerb67aba12013-05-13 19:22:14 -0400137**git_user_email**
138 Optional: Value to pass to `git config user.email`.
139 ``git_user_email=zuul@example.com``
140
141**git_user_name**
142 Optional: Value to pass to `git config user.name`.
143 ``git_user_name=zuul``
144
Arx Cruzb1b010d2013-10-28 19:49:59 -0200145**zuul_url**
James E. Blair4076e2b2014-01-28 12:42:20 -0800146 URL of this merger's git repos, accessible to test workers. Usually
147 "http://zuul.example.com/p" or "http://zuul-merger01.example.com/p"
148 depending on whether the merger is co-located with the Zuul server.
Arx Cruzb1b010d2013-10-28 19:49:59 -0200149
James E. Blaira4430132014-02-17 08:32:07 -0800150**log_config**
151 Path to log config file for the merger process.
152 ``log_config=/etc/zuul/logging.yaml``
153
154**pidfile**
155 Path to PID lock file for the merger process.
156 ``pidfile=/var/run/zuul-merger/merger.pid``
157
Joshua Hesketh5fea8672013-08-19 17:32:01 +1000158smtp
159""""
160
161**server**
162 SMTP server hostname or address to use.
163 ``server=localhost``
164
Akihiro Motokic7d28a72014-03-22 09:21:38 +0900165**port**
166 Optional: SMTP server port.
167 ``port=25``
168
Joshua Hesketh5fea8672013-08-19 17:32:01 +1000169**default_from**
170 Who the email should appear to be sent from when emailing the report.
171 This can be overridden by individual pipelines.
172 ``default_from=zuul@example.com``
173
174**default_to**
175 Who the report should be emailed to by default.
176 This can be overridden by individual pipelines.
177 ``default_to=you@example.com``
178
Joshua Hesketh36c3fa52014-01-22 11:40:52 +1100179.. _swift:
180
181swift
182"""""
183
184To send (optional) swift upload instructions this section must be
185present. Multiple destinations can be defined in the :ref:`jobs`
186section of the layout.
187
188**authurl**
189 The (keystone) Auth URL for swift
190 ``For example, https://identity.api.rackspacecloud.com/v2.0/``
191
192Any of the `swiftclient connection parameters`_ can also be defined
193here by the same name. Including the os_options by their key name (
194``for example tenant_id``)
195
196.. _swiftclient connection parameters: http://docs.openstack.org/developer/python-swiftclient/swiftclient.html#module-swiftclient.client
197
198**X-Account-Meta-Temp-Url-Key** (optional)
199 This is the key used to sign the HMAC message. zuul will send the
200 key to swift for you so you only need to define it here. If you do
201 not set a key zuul will generate one automatically.
202
203**region_name** (optional)
204 The region name holding the swift container
205 ``For example, SYD``
206
207Each destination defined by the :ref:`jobs` will have the following
208default values that it may overwrite.
209
210**default_container** (optional)
211 Container name to place the log into
212 ``For example, logs``
213
214**default_expiry** (optional)
215 How long the signed destination should be available for
216 ``default: 7200 (2hrs)``
217
218**default_max_file_size** (optional)
219 The maximum size of an individual file
220 ``default: 104857600 (100MB)``
221
222**default_max_file_count** (optional)
223 The maximum number of separate files to allow
224 ``default: 10``
225
226**default_logserver_prefix**
227 Provide a URL to the CDN or logserver app so that a worker knows
228 what URL to return. The worker should return the logserver_prefix
229 url and the object path.
230 ``For example: http://logs.example.org/server.app?obj=``
231
James E. Blaircdd00072012-06-08 19:17:28 -0700232layout.yaml
233~~~~~~~~~~~
234
Clark Boylan00635dc2012-09-19 14:03:08 -0700235This is the main configuration file for Zuul, where all of the pipelines
James E. Blaircdd00072012-06-08 19:17:28 -0700236and projects are defined, what tests should be run, and what actions
Clark Boylan00635dc2012-09-19 14:03:08 -0700237Zuul should perform. There are three sections: pipelines, jobs, and
James E. Blaircdd00072012-06-08 19:17:28 -0700238projects.
239
James E. Blaire5a847f2012-07-10 15:29:14 -0700240.. _includes:
241
242Includes
243""""""""
244
245Custom functions to be used in Zuul's configuration may be provided
246using the ``includes`` directive. It accepts a list of files to
247include, and currently supports one type of inclusion, a python file::
248
249 includes:
250 - python-file: local_functions.py
251
252**python-file**
253 The path to a python file. The file will be loaded and objects that
254 it defines will be placed in a special environment which can be
255 referenced in the Zuul configuration. Currently only the
256 parameter-function attribute of a Job uses this feature.
257
Clark Boylan00635dc2012-09-19 14:03:08 -0700258Pipelines
259"""""""""
James E. Blaircdd00072012-06-08 19:17:28 -0700260
Clark Boylan00635dc2012-09-19 14:03:08 -0700261Zuul can have any number of independent pipelines. Whenever a matching
262Gerrit event is found for a pipeline, that event is added to the
263pipeline, and the jobs specified for that pipeline are run. When all
264jobs specified for the pipeline that were triggered by an event are
265completed, Zuul reports back to Gerrit the results.
James E. Blaircdd00072012-06-08 19:17:28 -0700266
Clark Boylan00635dc2012-09-19 14:03:08 -0700267There are no pre-defined pipelines in Zuul, rather you can define
268whatever pipelines you need in the layout file. This is a very flexible
269system that can accommodate many kinds of workflows.
James E. Blaircdd00072012-06-08 19:17:28 -0700270
Clark Boylan00635dc2012-09-19 14:03:08 -0700271Here is a quick example of a pipeline definition followed by an
James E. Blaircdd00072012-06-08 19:17:28 -0700272explanation of each of the parameters::
273
274 - name: check
Clark Boylan00635dc2012-09-19 14:03:08 -0700275 manager: IndependentPipelineManager
James E. Blaircdd00072012-06-08 19:17:28 -0700276 trigger:
James E. Blair6c358e72013-07-29 17:06:47 -0700277 gerrit:
278 - event: patchset-created
James E. Blaircdd00072012-06-08 19:17:28 -0700279 success:
280 verified: 1
281 failure:
282 verified: -1
283
284**name**
285 This is used later in the project definition to indicate what jobs
Clark Boylan00635dc2012-09-19 14:03:08 -0700286 should be run for events in the pipeline.
James E. Blaircdd00072012-06-08 19:17:28 -0700287
James E. Blair8dbd56a2012-12-22 10:55:10 -0800288**description**
289 This is an optional field that may be used to provide a textual
290 description of the pipeline.
291
James E. Blair56370192013-01-14 15:47:28 -0800292**success-message**
293 An optional field that supplies the introductory text in message
294 reported back to Gerrit when all the voting builds are successful.
295 Defaults to "Build successful."
296
297**failure-message**
298 An optional field that supplies the introductory text in message
299 reported back to Gerrit when at least one voting build fails.
300 Defaults to "Build failed."
301
Joshua Heskethb7179772014-01-30 23:30:46 +1100302**merge-failure-message**
303 An optional field that supplies the introductory text in message
304 reported back to Gerrit when a change fails to merge with the
305 current state of the repository.
306 Defaults to "Merge failed."
307
Joshua Hesketh3979e3e2014-03-04 11:21:10 +1100308**footer-message**
309 An optional field to supply additional information after test results.
310 Useful for adding information about the CI system such as debugging
311 and contact details.
312
James E. Blaircdd00072012-06-08 19:17:28 -0700313**manager**
Clark Boylan00635dc2012-09-19 14:03:08 -0700314 There are currently two schemes for managing pipelines:
James E. Blaircdd00072012-06-08 19:17:28 -0700315
Clark Boylan00635dc2012-09-19 14:03:08 -0700316 *IndependentPipelineManager*
317 Every event in this pipeline should be treated as independent of
318 other events in the pipeline. This is appropriate when the order of
319 events in the pipeline doesn't matter because the results of the
320 actions this pipeline performs can not affect other events in the
321 pipeline. For example, when a change is first uploaded for review,
James E. Blaircdd00072012-06-08 19:17:28 -0700322 you may want to run tests on that change to provide early feedback
323 to reviewers. At the end of the tests, the change is not going to
324 be merged, so it is safe to run these tests in parallel without
Clark Boylan00635dc2012-09-19 14:03:08 -0700325 regard to any other changes in the pipeline. They are independent.
James E. Blaircdd00072012-06-08 19:17:28 -0700326
Clark Boylan00635dc2012-09-19 14:03:08 -0700327 Another type of pipeline that is independent is a post-merge
328 pipeline. In that case, the changes have already merged, so the
329 results can not affect any other events in the pipeline.
James E. Blaircdd00072012-06-08 19:17:28 -0700330
Clark Boylan00635dc2012-09-19 14:03:08 -0700331 *DependentPipelineManager*
332 The dependent pipeline manager is designed for gating. It ensures
James E. Blaircdd00072012-06-08 19:17:28 -0700333 that every change is tested exactly as it is going to be merged
334 into the repository. An ideal gating system would test one change
335 at a time, applied to the tip of the repository, and only if that
336 change passed tests would it be merged. Then the next change in
337 line would be tested the same way. In order to achieve parallel
Clark Boylan00635dc2012-09-19 14:03:08 -0700338 testing of changes, the dependent pipeline manager performs
James E. Blaircdd00072012-06-08 19:17:28 -0700339 speculative execution on changes. It orders changes based on
Clark Boylan00635dc2012-09-19 14:03:08 -0700340 their entry into the pipeline. It begins testing all changes in
341 parallel, assuming that each change ahead in the pipeline will pass
James E. Blaircdd00072012-06-08 19:17:28 -0700342 its tests. If they all succeed, all the changes can be tested and
Clark Boylan00635dc2012-09-19 14:03:08 -0700343 merged in parallel. If a change near the front of the pipeline
344 fails its tests, each change behind it ignores whatever tests have
345 been completed and are tested again without the change in front.
346 This way gate tests may run in parallel but still be tested
347 correctly, exactly as they will appear in the repository when
348 merged.
James E. Blaircdd00072012-06-08 19:17:28 -0700349
Clark Boylan00635dc2012-09-19 14:03:08 -0700350 One important characteristic of the DependentPipelineManager is that
James E. Blaircdd00072012-06-08 19:17:28 -0700351 it analyzes the jobs that are triggered by different projects, and
352 if those projects have jobs in common, it treats those projects as
353 related, and they share a single virtual queue of changes. Thus,
354 if there is a job that performs integration testing on two
355 projects, those two projects will automatically share a virtual
356 change queue. If a third project does not invoke that job, it
Clark Boylan00635dc2012-09-19 14:03:08 -0700357 will be part of a separate virtual change queue, and changes to
358 it will not depend on changes to the first two jobs.
James E. Blaircdd00072012-06-08 19:17:28 -0700359
360 For more detail on the theory and operation of Zuul's
Clark Boylan00635dc2012-09-19 14:03:08 -0700361 DependentPipelineManager, see: :doc:`gating`.
James E. Blaircdd00072012-06-08 19:17:28 -0700362
363**trigger**
James E. Blair6c358e72013-07-29 17:06:47 -0700364 Exactly one trigger source must be supplied for each pipeline.
James E. Blaircdd00072012-06-08 19:17:28 -0700365 Triggers are not exclusive -- matching events may be placed in
James E. Blair6c358e72013-07-29 17:06:47 -0700366 multiple pipelines, and they will behave independently in each of
367 the pipelines they match. You may select from the following:
James E. Blaircdd00072012-06-08 19:17:28 -0700368
James E. Blair6c358e72013-07-29 17:06:47 -0700369 **gerrit**
370 This describes what Gerrit events should be placed in the
371 pipeline. Multiple gerrit triggers may be listed. Further
372 parameters describe the kind of events that match:
James E. Blaircdd00072012-06-08 19:17:28 -0700373
James E. Blair6c358e72013-07-29 17:06:47 -0700374 *event*
375 The event name from gerrit. Examples: ``patchset-created``,
376 ``comment-added``, ``ref-updated``. This field is treated as a
377 regular expression.
James E. Blaircdd00072012-06-08 19:17:28 -0700378
James E. Blair6c358e72013-07-29 17:06:47 -0700379 *branch*
380 The branch associated with the event. Example: ``master``. This
381 field is treated as a regular expression, and multiple branches may
382 be listed.
James E. Blaircdd00072012-06-08 19:17:28 -0700383
James E. Blair6c358e72013-07-29 17:06:47 -0700384 *ref*
385 On ref-updated events, the branch parameter is not used, instead the
386 ref is provided. Currently Gerrit has the somewhat idiosyncratic
387 behavior of specifying bare refs for branch names (e.g., ``master``),
388 but full ref names for other kinds of refs (e.g., ``refs/tags/foo``).
389 Zuul matches what you put here exactly against what Gerrit
390 provides. This field is treated as a regular expression, and
391 multiple refs may be listed.
James E. Blaircdd00072012-06-08 19:17:28 -0700392
James E. Blair6c358e72013-07-29 17:06:47 -0700393 *approval*
394 This is only used for ``comment-added`` events. It only matches if
395 the event has a matching approval associated with it. Example:
396 ``code-review: 2`` matches a ``+2`` vote on the code review category.
397 Multiple approvals may be listed.
Antoine Mussob4e809e2012-12-06 16:58:06 +0100398
James E. Blair6c358e72013-07-29 17:06:47 -0700399 *email_filter*
400 This is used for any event. It takes a regex applied on the performer
Michael Prokop526926a2013-10-24 16:16:57 +0200401 email, i.e. Gerrit account email address. If you want to specify
James E. Blair6c358e72013-07-29 17:06:47 -0700402 several email filters, you must use a YAML list. Make sure to use non
403 greedy matchers and to escapes dots!
404 Example: ``email_filter: ^.*?@example\.org$``.
405
Joshua Heskethb8a817e2013-12-27 11:21:38 +1100406 *username_filter*
407 This is used for any event. It takes a regex applied on the performer
408 username, i.e. Gerrit account name. If you want to specify several
409 username filters, you must use a YAML list. Make sure to use non greedy
410 matchers and to escapes dots!
411 Example: ``username_filter: ^jenkins$``.
412
James E. Blair6c358e72013-07-29 17:06:47 -0700413 *comment_filter*
414 This is only used for ``comment-added`` events. It accepts a list of
415 regexes that are searched for in the comment string. If any of these
416 regexes matches a portion of the comment string the trigger is
417 matched. ``comment_filter: retrigger`` will match when comments
418 containing 'retrigger' somewhere in the comment text are added to a
419 change.
Clark Boylanb9bcb402012-06-29 17:44:05 -0700420
James E. Blair11041d22014-05-02 14:49:53 -0700421 *require-approval* (deprecated)
James E. Blairc053d022014-01-22 14:57:33 -0800422 This may be used for any event. It requires that a certain kind
423 of approval be present for the current patchset of the change (the
James E. Blair11041d22014-05-02 14:49:53 -0700424 approval could be added by the event in question). It follows the
425 same syntax as the "approval" pipeline requirement below. This
426 form should be considered deprecated and the pipeline requirement
427 used instead.
James E. Blairc053d022014-01-22 14:57:33 -0800428
James E. Blair63bb0ef2013-07-29 17:14:51 -0700429 **timer**
430 This trigger will run based on a cron-style time specification.
431 It will enqueue an event into its pipeline for every project
432 defined in the configuration. Any job associated with the
433 pipeline will run in response to that event.
434
435 *time*
436 The time specification in cron syntax. Only the 5 part syntax is
437 supported, not the symbolic names. Example: ``0 0 * * *`` runs
438 at midnight.
439
James E. Blair11041d22014-05-02 14:49:53 -0700440**require**
441 If this section is present, it established pre-requisites for any
442 kind of item entering the Pipeline. Regardless of how the item is
443 to be enqueued (via any trigger or automatic dependency resolution),
444 the conditions specified here must be met or the item will not be
445 enqueued.
446
447 **approval**
448 This requires that a certain kind of approval be present for the
449 current patchset of the change (the approval could be added by the
450 event in question). It takes several sub-parameters, all of which
451 are optional and are combined together so that there must be an
452 approval matching all specified requirements.
453
454 *username*
455 If present, an approval from this username is required.
456
457 *email-filter*
458 If present, an approval with this email address is required. It
459 is treated as a regular expression as above.
460
461 *older-than*
462 If present, the approval must be older than this amount of time
463 to match. Provide a time interval as a number with a suffix of
464 "w" (weeks), "d" (days), "h" (hours), "m" (minutes), "s"
465 (seconds). Example ``48h`` or ``2d``.
466
467 *newer-than*
468 If present, the approval must be newer than this amount of time
469 to match. Same format as "older-than".
470
471 Any other field is interpreted as a review category and value
472 pair. For example ``verified: 1`` would require that the approval
473 be for a +1 vote in the "Verified" column.
474
475 **open**
476 A boolean value (``true`` or ``false``) that indicates whether the change
477 must be open or closed in order to be enqueued.
478
479 **status**
480 A string value that corresponds with the status of the change
481 reported by the trigger. For example, when using the Gerrit
482 trigger, status values such as ``NEW`` or ``MERGED`` may be useful.
James E. Blair63bb0ef2013-07-29 17:14:51 -0700483
James E. Blair2fa50962013-01-30 21:50:41 -0800484**dequeue-on-new-patchset**
485 Normally, if a new patchset is uploaded to a change that is in a
486 pipeline, the existing entry in the pipeline will be removed (with
487 jobs canceled and any dependent changes that can no longer merge as
488 well. To suppress this behavior (and allow jobs to continue
489 running), set this to ``false``. Default: ``true``.
490
James E. Blaircdd00072012-06-08 19:17:28 -0700491**success**
Joshua Hesketh1879cf72013-08-19 14:13:15 +1000492 Describes where Zuul should report to if all the jobs complete
493 successfully.
James E. Blaircdd00072012-06-08 19:17:28 -0700494 This section is optional; if it is omitted, Zuul will run jobs and
495 do nothing on success; it will not even report a message to Gerrit.
Joshua Hesketh1879cf72013-08-19 14:13:15 +1000496 If the section is present, the listed reporter plugins will be
497 asked to report on the jobs.
498 Each reporter's value dictionary is handled by the reporter. See
499 reporters for more details.
James E. Blaircdd00072012-06-08 19:17:28 -0700500
Mathieu Gagnéd6d2a642013-06-11 20:59:58 -0400501**failure**
James E. Blaircdd00072012-06-08 19:17:28 -0700502 Uses the same syntax as **success**, but describes what Zuul should
503 do if at least one job fails.
James E. Blairdc253862012-06-13 17:12:42 -0700504
Joshua Heskethb7179772014-01-30 23:30:46 +1100505**merge-failure**
506 Uses the same syntax as **success**, but describes what Zuul should
507 do if it is unable to merge in the patchset. If no merge-failure
508 reporters are listed then the ``failure`` reporters will be used to
509 notify of unsuccessful merges.
510
Mathieu Gagnéd6d2a642013-06-11 20:59:58 -0400511**start**
James E. Blairdc253862012-06-13 17:12:42 -0700512 Uses the same syntax as **success**, but describes what Zuul should
Clark Boylan00635dc2012-09-19 14:03:08 -0700513 do when a change is added to the pipeline manager. This can be used,
James E. Blairdc253862012-06-13 17:12:42 -0700514 for example, to reset the value of the Verified review category.
Mathieu Gagnéd6d2a642013-06-11 20:59:58 -0400515
James E. Blair64ed6f22013-07-10 14:07:23 -0700516**precedence**
517 Indicates how the build scheduler should prioritize jobs for
518 different pipelines. Each pipeline may have one precedence, jobs
519 for pipelines with a higher precedence will be run before ones with
520 lower. The value should be one of ``high``, ``normal``, or ``low``.
521 Default: ``normal``.
522
Clark Boylanc2d19e42014-01-23 14:08:58 -0800523**window**
524 DependentPipelineManagers only. Zuul can rate limit
525 DependentPipelineManagers in a manner similar to TCP flow control.
526 Jobs are only started for changes in the queue if they sit in the
527 actionable window for the pipeline. The initial length of this window
528 is configurable with this value. The value given should be a positive
529 integer value. A value of ``0`` disables rate limiting on the
530 DependentPipelineManager.
531 Default: ``20``.
532
533**window-floor**
534 DependentPipelineManagers only. This is the minimum value for the
535 window described above. Should be a positive non zero integer value.
536 Default: ``3``.
537
538**window-increase-type**
539 DependentPipelineManagers only. This value describes how the window
540 should grow when changes are successfully merged by zuul. A value of
541 ``linear`` indicates that ``window-increase-factor`` should be added
542 to the previous window value. A value of ``exponential`` indicates
543 that ``window-increase-factor`` should be multiplied against the
544 previous window value and the result will become the window size.
545 Default: ``linear``.
546
547**window-increase-factor**
548 DependentPipelineManagers only. The value to be added or mulitplied
549 against the previous window value to determine the new window after
550 successful change merges.
551 Default: ``1``.
552
553**window-decrease-type**
554 DependentPipelineManagers only. This value describes how the window
555 should shrink when changes are not able to be merged by Zuul. A value
556 of ``linear`` indicates that ``window-decrease-factor`` should be
557 subtracted from the previous window value. A value of ``exponential``
558 indicates that ``window-decrease-factor`` should be divided against
559 the previous window value and the result will become the window size.
560 Default: ``exponential``.
561
562**window-decrease-factor**
563 DependentPipelineManagers only. The value to be subtracted or divided
564 against the previous window value to determine the new window after
565 unsuccessful change merges.
566 Default: ``2``.
567
Clark Boylan00635dc2012-09-19 14:03:08 -0700568Some example pipeline configurations are included in the sample layout
569file. The first is called a *check* pipeline::
James E. Blaircdd00072012-06-08 19:17:28 -0700570
571 - name: check
Clark Boylan00635dc2012-09-19 14:03:08 -0700572 manager: IndependentPipelineManager
James E. Blaircdd00072012-06-08 19:17:28 -0700573 trigger:
574 - event: patchset-created
575 success:
Joshua Hesketh1879cf72013-08-19 14:13:15 +1000576 gerrit:
577 verified: 1
James E. Blaircdd00072012-06-08 19:17:28 -0700578 failure:
Joshua Hesketh1879cf72013-08-19 14:13:15 +1000579 gerrit:
580 verified: -1
James E. Blaircdd00072012-06-08 19:17:28 -0700581
582This will trigger jobs each time a new patchset (or change) is
583uploaded to Gerrit, and report +/-1 values to Gerrit in the
584``verified`` review category. ::
585
586 - name: gate
Clark Boylan00635dc2012-09-19 14:03:08 -0700587 manager: DependentPipelineManager
James E. Blaircdd00072012-06-08 19:17:28 -0700588 trigger:
589 - event: comment-added
590 approval:
591 - approved: 1
592 success:
Joshua Hesketh1879cf72013-08-19 14:13:15 +1000593 gerrit:
594 verified: 2
595 submit: true
James E. Blaircdd00072012-06-08 19:17:28 -0700596 failure:
Joshua Hesketh1879cf72013-08-19 14:13:15 +1000597 gerrit:
598 verified: -2
James E. Blaircdd00072012-06-08 19:17:28 -0700599
600This will trigger jobs whenever a reviewer leaves a vote of ``1`` in the
601``approved`` review category in Gerrit (a non-standard category).
602Changes will be tested in such a way as to guarantee that they will be
603merged exactly as tested, though that will happen in parallel by
604creating a virtual queue of dependent changes and performing
605speculative execution of jobs. ::
606
607 - name: post
Clark Boylan00635dc2012-09-19 14:03:08 -0700608 manager: IndependentPipelineManager
James E. Blaircdd00072012-06-08 19:17:28 -0700609 trigger:
610 - event: ref-updated
611 ref: ^(?!refs/).*$
612
613This will trigger jobs whenever a change is merged to a named branch
614(e.g., ``master``). No output will be reported to Gerrit. This is
615useful for side effects such as creating per-commit tarballs. ::
616
617 - name: silent
Clark Boylan00635dc2012-09-19 14:03:08 -0700618 manager: IndependentPipelineManager
James E. Blaircdd00072012-06-08 19:17:28 -0700619 trigger:
620 - event: patchset-created
621
622This also triggers jobs when changes are uploaded to Gerrit, but no
623results are reported to Gerrit. This is useful for jobs that are in
Antoine Mussoce333842012-10-16 14:42:35 +0200624development and not yet ready to be presented to developers. ::
625
626 pipelines:
627 - name: post-merge
628 manager: IndependentPipelineManager
629 trigger:
630 - event: change-merged
631 success:
Joshua Hesketh1879cf72013-08-19 14:13:15 +1000632 gerrit:
633 force-message: True
Antoine Mussoce333842012-10-16 14:42:35 +0200634 failure:
Joshua Hesketh1879cf72013-08-19 14:13:15 +1000635 gerrit:
636 force-message: True
Antoine Mussoce333842012-10-16 14:42:35 +0200637
638The ``change-merged`` events happen when a change has been merged in the git
639repository. The change is thus closed and Gerrit will not accept modifications
640to the review scoring such as ``code-review`` or ``verified``. By using the
641``force-message: True`` parameter, Zuul will pass ``--force-message`` to the
642``gerrit review`` command, thus making sure the message is actually
643sent back to Gerrit regardless of approval scores.
644That kind of pipeline is nice to run regression or performance tests.
645
646.. note::
647 The ``change-merged`` event does not include the commit sha1 which can be
648 hazardous, it would let you report back to Gerrit though. If you were to
Michael Prokop526926a2013-10-24 16:16:57 +0200649 build a tarball for a specific commit, you should consider instead using
Łukasz Jernaś048acb42014-03-02 18:49:41 +0100650 the ``ref-updated`` event which does include the commit sha1 (but lacks the
Antoine Mussoce333842012-10-16 14:42:35 +0200651 Gerrit change number).
James E. Blaircdd00072012-06-08 19:17:28 -0700652
Joshua Hesketh36c3fa52014-01-22 11:40:52 +1100653
654.. _jobs:
655
James E. Blaircdd00072012-06-08 19:17:28 -0700656Jobs
657""""
658
659The jobs section is optional, and can be used to set attributes of
660jobs that are independent of their association with a project. For
661example, if a job should return a customized message on failure, that
662may be specified here. Otherwise, Zuul does not need to be told about
663each job as it builds a list from the project specification.
664
665**name**
666 The name of the job. This field is treated as a regular expression
667 and will be applied to each job that matches.
668
James E. Blairc8a1e052014-02-25 09:29:26 -0800669**queue-name (optional)**
670 Zuul will automatically combine projects that share a job into
671 shared change queues for dependent pipeline managers. In order to
672 report statistics about these queues, it is convenient for them to
673 have names. Zuul can automatically name change queues, however
674 these can grow quite long and are prone to changing as projects in
675 the queue change. If you assign a queue-name to a job, Zuul will
676 use that as the name for the shared change queue that contains that
677 job instead of the automatically generated one. It is an error for
678 a shared change queue to have more than one job with a queue-name if
679 they are not the same.
680
James E. Blaire5a847f2012-07-10 15:29:14 -0700681**failure-message (optional)**
682 The message that should be reported to Gerrit if the job fails.
James E. Blaircdd00072012-06-08 19:17:28 -0700683
James E. Blaire5a847f2012-07-10 15:29:14 -0700684**success-message (optional)**
685 The message that should be reported to Gerrit if the job fails.
James E. Blaircdd00072012-06-08 19:17:28 -0700686
James E. Blair6aea36d2012-12-17 13:03:24 -0800687**failure-pattern (optional)**
688 The URL that should be reported to Gerrit if the job fails.
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700689 Defaults to the build URL or the url_pattern configured in
James E. Blair6aea36d2012-12-17 13:03:24 -0800690 zuul.conf. May be supplied as a string pattern with substitutions
691 as described in url_pattern in :ref:`zuulconf`.
692
693**success-pattern (optional)**
694 The URL that should be reported to Gerrit if the job succeeds.
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700695 Defaults to the build URL or the url_pattern configured in
James E. Blair6aea36d2012-12-17 13:03:24 -0800696 zuul.conf. May be supplied as a string pattern with substitutions
697 as described in url_pattern in :ref:`zuulconf`.
698
James E. Blair222d4982012-07-16 09:31:19 -0700699**hold-following-changes (optional)**
700 This is a boolean that indicates that changes that follow this
Clark Boylan00635dc2012-09-19 14:03:08 -0700701 change in a dependent change pipeline should wait until this job
James E. Blair222d4982012-07-16 09:31:19 -0700702 succeeds before launching. If this is applied to a very short job
703 that can predict whether longer jobs will fail early, this can be
704 used to reduce the number of jobs that Zuul will launch and
705 ultimately have to cancel. In that case, a small amount of
Mathieu Gagnéd6d2a642013-06-11 20:59:58 -0400706 parallelization of jobs is traded for more efficient use of testing
James E. Blair222d4982012-07-16 09:31:19 -0700707 resources. On the other hand, to apply this to a long running job
708 would largely defeat the parallelization of dependent change testing
Mathieu Gagnéd6d2a642013-06-11 20:59:58 -0400709 that is the main feature of Zuul. Default: ``false``.
James E. Blair222d4982012-07-16 09:31:19 -0700710
James E. Blaire5a847f2012-07-10 15:29:14 -0700711**branch (optional)**
James E. Blaircdd00072012-06-08 19:17:28 -0700712 This job should only be run on matching branches. This field is
713 treated as a regular expression and multiple branches may be
714 listed.
715
James E. Blair70c71582013-03-06 08:50:50 -0800716**files (optional)**
717 This job should only be run if at least one of the files involved in
718 the change (added, deleted, or modified) matches at least one of the
719 file patterns listed here. This field is treated as a regular
720 expression and multiple expressions may be listed.
721
Mathieu Gagnéd6d2a642013-06-11 20:59:58 -0400722**voting (optional)**
723 Boolean value (``true`` or ``false``) that indicates whatever
724 a job is voting or not. Default: ``true``.
725
James E. Blaire5a847f2012-07-10 15:29:14 -0700726**parameter-function (optional)**
727 Specifies a function that should be applied to the parameters before
728 the job is launched. The function should be defined in a python file
729 included with the :ref:`includes` directive. The function
730 should have the following signature:
731
James E. Blaird0470972013-07-29 10:05:43 -0700732 .. function:: parameters(item, job, parameters)
James E. Blaire5a847f2012-07-10 15:29:14 -0700733
734 Manipulate the parameters passed to a job before a build is
735 launched. The ``parameters`` dictionary will already contain the
736 standard Zuul job parameters, and is expected to be modified
737 in-place.
738
James E. Blaird78576a2013-07-09 10:39:17 -0700739 :param item: the current queue item
740 :type item: zuul.model.QueueItem
James E. Blaird0470972013-07-29 10:05:43 -0700741 :param job: the job about to be run
742 :type job: zuul.model.Job
James E. Blaire5a847f2012-07-10 15:29:14 -0700743 :param parameters: parameters to be passed to the job
744 :type parameters: dict
745
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700746 If the parameter **ZUUL_NODE** is set by this function, then it will
747 be used to specify on what node (or class of node) the job should be
748 run.
749
Joshua Hesketh36c3fa52014-01-22 11:40:52 +1100750**swift**
751 If :ref:`swift` is configured then each job can define a destination
752 container for the builder to place logs and/or assets into. Multiple
753 containers can be listed for each job by providing a unique ``name``.
754
755 *name*
756 Set an identifying name for the container. This is used in the
757 parameter key sent to the builder. For example if it ``logs`` then
758 one of the parameters sent will be ``SWIFT_logs_CONTAINER``
759 (case-sensitive).
760
761 Each of the defaults defined in :ref:`swift` can be overwritten as:
762
763 *container* (optional)
764 Container name to place the log into
765 ``For example, logs``
766
767 *expiry* (optional)
768 How long the signed destination should be available for
769
770 *max_file_size** (optional)
771 The maximum size of an individual file
772
773 *max_file_count* (optional)
774 The maximum number of separate files to allow
775
776 *logserver_prefix*
777 Provide a URL to the CDN or logserver app so that a worker knows
778 what URL to return.
779 ``For example: http://logs.example.org/server.app?obj=``
780 The worker should return the logserver_prefix url and the object
781 path as the URL in the results data packet.
782
James E. Blaircdd00072012-06-08 19:17:28 -0700783Here is an example of setting the failure message for jobs that check
784whether a change merges cleanly::
785
786 - name: ^.*-merge$
787 failure-message: This change was unable to be automatically merged
788 with the current state of the repository. Please rebase your
789 change and upload a new patchset.
790
791Projects
792""""""""
793
Clark Boylan00635dc2012-09-19 14:03:08 -0700794The projects section indicates what jobs should be run in each pipeline
James E. Blaircdd00072012-06-08 19:17:28 -0700795for events associated with each project. It contains a list of
796projects. Here is an example::
797
798 - name: example/project
799 check:
800 - project-merge:
801 - project-unittest
Clark Boylan00635dc2012-09-19 14:03:08 -0700802 - project-pep8
803 - project-pyflakes
James E. Blaircdd00072012-06-08 19:17:28 -0700804 gate:
805 - project-merge:
806 - project-unittest
Clark Boylan00635dc2012-09-19 14:03:08 -0700807 - project-pep8
808 - project-pyflakes
James E. Blaircdd00072012-06-08 19:17:28 -0700809 post:
810 - project-publish
811
812**name**
813 The name of the project (as known by Gerrit).
814
James E. Blair19deff22013-08-25 13:17:35 -0700815**merge-mode (optional)**
816 An optional value that indicates what strategy should be used to
817 merge changes to this project. Supported values are:
818
819 ** merge-resolve **
820 Equivalent to 'git merge -s resolve'. This corresponds closely to
821 what Gerrit performs (using JGit) for a project if the "Merge if
822 necessary" merge mode is selected and "Automatically resolve
823 conflicts" is checked. This is the default.
824
825 ** merge **
826 Equivalent to 'git merge'.
827
828 ** cherry-pick **
829 Equivalent to 'git cherry-pick'.
830
Clark Boylan00635dc2012-09-19 14:03:08 -0700831This is followed by a section for each of the pipelines defined above.
832Pipelines may be omitted if no jobs should run for this project in a
833given pipeline. Within the pipeline section, the jobs that should be
James E. Blaircdd00072012-06-08 19:17:28 -0700834executed are listed. If a job is entered as a dictionary key, then
835jobs contained within that key are only executed if the key job
836succeeds. In the above example, project-unittest, project-pep8, and
837project-pyflakes are only executed if project-merge succeeds. This
838can help avoid running unnecessary jobs.
839
James E. Blair18c64442014-03-18 10:14:45 -0700840The special job named ``noop`` is internal to Zuul and will always
841return ``SUCCESS`` immediately. This can be useful if you require
842that all changes be processed by a pipeline but a project has no jobs
843that can be run on it.
844
Paul Belangerffef9e42013-02-11 22:15:18 -0500845.. seealso:: The OpenStack Zuul configuration for a comprehensive example: https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/zuul/layout.yaml
James E. Blaircdd00072012-06-08 19:17:28 -0700846
Antoine Musso80edd5a2013-02-13 15:37:53 +0100847Project Templates
848"""""""""""""""""
849
Michael Prokop526926a2013-10-24 16:16:57 +0200850Whenever you have lot of similar projects (such as plugins for a project) you
Antoine Musso80edd5a2013-02-13 15:37:53 +0100851will most probably want to use the same pipeline configurations. The
852project templates let you define pipelines and job name templates to trigger.
853One can then just apply the template on its project which make it easier to
Michael Prokop526926a2013-10-24 16:16:57 +0200854update several similar projects. As an example::
Antoine Musso80edd5a2013-02-13 15:37:53 +0100855
856 project-templates:
857 # Name of the template
858 - name: plugin-triggering
859 # Definition of pipelines just like for a `project`
860 check:
861 - '{jobprefix}-merge':
862 - '{jobprefix}-pep8'
863 - '{jobprefix}-pyflakes'
864 gate:
865 - '{jobprefix}-merge':
866 - '{jobprefix}-unittest'
867 - '{jobprefix}-pep8'
868 - '{jobprefix}-pyflakes'
869
870In your projects definition, you will then apply the template using the template
871key::
872
873 projects:
874 - name: plugin/foobar
875 template:
876 - name: plugin-triggering
877 jobprefix: plugin-foobar
878
James E. Blairaea6cf62013-12-16 15:38:12 -0800879You can pass several parameters to a template. A ``parameter`` value
880will be used for expansion of ``{parameter}`` in the template
881strings. The parameter ``name`` will be automatically provided and
882will contain the short name of the project, that is the portion of the
883project name after the last ``/`` character.
James E. Blaircdd00072012-06-08 19:17:28 -0700884
James E. Blair3e98c022013-12-16 15:25:38 -0800885Multiple templates can be combined in a project, and the jobs from all
886of those templates will be added to the project. Individual jobs may
887also be added::
888
889 projects:
890 - name: plugin/foobar
891 template:
892 - name: plugin-triggering
893 jobprefix: plugin-foobar
894 - name: plugin-extras
895 jobprefix: plugin-foobar
896 check:
897 - foobar-extra-special-job
898
899The order of the jobs listed in the project (which only affects the
900order of jobs listed on the report) will be the jobs from each
901template in the order listed, followed by any jobs individually listed
902for the project.
903
904Note that if multiple templates are used for a project and one
905template specifies a job that is also specified in another template,
James E. Blair12a92b12014-03-26 11:54:53 -0700906or specified in the project itself, the configuration defined by
907either the last template or the project itself will take priority.
James E. Blair3e98c022013-12-16 15:25:38 -0800908
James E. Blaircdd00072012-06-08 19:17:28 -0700909logging.conf
910~~~~~~~~~~~~
911This file is optional. If provided, it should be a standard
912:mod:`logging.config` module configuration file. If not present, Zuul will
913output all log messages of DEBUG level or higher to the console.
914
915Starting Zuul
916-------------
917
918To start Zuul, run **zuul-server**::
919
Antoine Mussob3aa8282013-04-19 15:16:59 +0200920 usage: zuul-server [-h] [-c CONFIG] [-l LAYOUT] [-d] [-t] [--version]
James E. Blaircdd00072012-06-08 19:17:28 -0700921
922 Project gating system.
923
924 optional arguments:
925 -h, --help show this help message and exit
926 -c CONFIG specify the config file
Antoine Mussob3aa8282013-04-19 15:16:59 +0200927 -l LAYOUT specify the layout file
James E. Blaircdd00072012-06-08 19:17:28 -0700928 -d do not run as a daemon
Antoine Mussob3aa8282013-04-19 15:16:59 +0200929 -t validate layout file syntax
930 --version show zuul version
James E. Blaircdd00072012-06-08 19:17:28 -0700931
932You may want to use the ``-d`` argument while you are initially setting
933up Zuul so you can detect any configuration errors quickly. Under
934normal operation, omit ``-d`` and let Zuul run as a daemon.
935
936If you send signal 1 (SIGHUP) to the zuul-server process, Zuul will
937stop executing new jobs, wait until all executing jobs are finished,
938reload its configuration, and resume. Any values in any of the
939configuration files may be changed, except the location of Zuul's PID
940file (a change to that will be ignored until Zuul is restarted).
Clark Boylanf231fa22013-02-08 12:28:53 -0800941
942If you send a SIGUSR1 to the zuul-server process, Zuul will stop
943executing new jobs, wait until all executing jobs are finished,
944then exit. While waiting to exit Zuul will queue Gerrit events and
945save these events prior to exiting. When Zuul starts again it will
946read these saved events and act on them.
Jeremy Stanley93e05f42013-03-08 17:29:17 +0000947
Michael Prokop526926a2013-10-24 16:16:57 +0200948If you need to abort Zuul and intend to manually requeue changes for
Jeremy Stanley93e05f42013-03-08 17:29:17 +0000949jobs which were running in its pipelines, prior to terminating you can
950use the zuul-changes.py tool script to simplify the process. For
951example, this would give you a list of Gerrit commands to reverify or
952recheck changes for the gate and check pipelines respectively::
953
954 ./tools/zuul-changes.py --review-host=review.openstack.org \
955 http://zuul.openstack.org/ gate 'reverify no bug'
956 ./tools/zuul-changes.py --review-host=review.openstack.org \
957 http://zuul.openstack.org/ check 'recheck no bug'
Clark Boylanfba9b242013-08-20 10:11:17 -0700958
959If you send a SIGUSR2 to the zuul-server process, Zuul will dump a stack
960trace for each running thread into its debug log. This is useful for
961tracking down deadlock or otherwise slow threads.