blob: 8d7bbeff99e558e14c47ac8e2bda07c7375e6a02 [file] [log] [blame]
James E. Blaircdd00072012-06-08 19:17:28 -07001:title: Zuul
2
3Zuul
4====
5
6Configuration
7-------------
8
9Zuul has three configuration files:
10
11**zuul.conf**
12 Credentials for Gerrit and Jenkins, locations of the other config files
13**layout.yaml**
14 Project and queue configuration -- what Zuul does
15**logging.conf**
16 Python logging config
17
18Examples of each of the three files can be found in the etc/ directory
19of the source distribution.
20
21zuul.conf
22~~~~~~~~~
23
24Zuul will look for ``/etc/zuul/zuul.conf`` or ``~/zuul.conf`` to
25bootstrap its configuration. Alternately, you may specify ``-c
26/path/to/zuul.conf`` on the command line.
27
28Gerrit and Jenkins credentials are each described in a section of
29zuul.conf. The location of the other two configuration files (as well
30as the location of the PID file when running Zuul as a server) are
31specified in a third section.
32
33layout.yaml
34~~~~~~~~~~~
35
36This is the main configuration file for Zuul, where all of the queues
37and projects are defined, what tests should be run, and what actions
38Zuul should perform. There are three sections: queues, jobs, and
39projects.
40
James E. Blaire5a847f2012-07-10 15:29:14 -070041.. _includes:
42
43Includes
44""""""""
45
46Custom functions to be used in Zuul's configuration may be provided
47using the ``includes`` directive. It accepts a list of files to
48include, and currently supports one type of inclusion, a python file::
49
50 includes:
51 - python-file: local_functions.py
52
53**python-file**
54 The path to a python file. The file will be loaded and objects that
55 it defines will be placed in a special environment which can be
56 referenced in the Zuul configuration. Currently only the
57 parameter-function attribute of a Job uses this feature.
58
James E. Blaircdd00072012-06-08 19:17:28 -070059Queues
60""""""
61
62Zuul can have any number of independent queues. Whenever a matching
63Gerrit event is found for a queue, that event is added to the queue,
64and the jobs specified for that queue are run. When all jobs
65specified for the queue that were triggered by an event are completed,
66Zuul reports back to Gerrit the results.
67
68There are no pre-defined queues in Zuul, rather you can define
69whatever queues you need in the layout file. This is a very flexible
70system that can accommodate many kinds of workflows.
71
72Here is a quick example of a queue definition followed by an
73explanation of each of the parameters::
74
75 - name: check
76 manager: IndependentQueueManager
77 trigger:
78 - event: patchset-created
79 success:
80 verified: 1
81 failure:
82 verified: -1
83
84**name**
85 This is used later in the project definition to indicate what jobs
86 should be run for events in the queue.
87
88**manager**
89 There are currently two schemes for managing queues:
90
91 *IndependentQueueManager*
92 Every event in this queue should be treated as independent of
93 other events in the queue. This is appropriate when the order of
94 events in the queue doesn't matter because the results of the
95 actions this queue performs can not affect other events in the
96 queue. For example, when a change is first uploaded for review,
97 you may want to run tests on that change to provide early feedback
98 to reviewers. At the end of the tests, the change is not going to
99 be merged, so it is safe to run these tests in parallel without
100 regard to any other changes in the queue. They are independent.
101
102 Another type of queue that is independent is a post-merge queue.
103 In that case, the changes have already merged, so the results can
104 not affect any other events in the queue.
105
106 *DependentQueueManager*
107 The dependent queue manager is designed for gating. It ensures
108 that every change is tested exactly as it is going to be merged
109 into the repository. An ideal gating system would test one change
110 at a time, applied to the tip of the repository, and only if that
111 change passed tests would it be merged. Then the next change in
112 line would be tested the same way. In order to achieve parallel
113 testing of changes, the dependent queue manager performs
114 speculative execution on changes. It orders changes based on
115 their entry into the queue. It begins testing all changes in
116 parallel, assuming that each change ahead in the queue will pass
117 its tests. If they all succeed, all the changes can be tested and
118 merged in parallel. If a change near the front of the queue fails
119 its tests, each change behind it ignores whatever tests have been
120 completed and are tested again without the change in front. This
121 way gate tests may run in parallel but still be tested correctly,
122 exactly as they will appear in the repository when merged.
123
124 One important characteristic of the DependentQueueManager is that
125 it analyzes the jobs that are triggered by different projects, and
126 if those projects have jobs in common, it treats those projects as
127 related, and they share a single virtual queue of changes. Thus,
128 if there is a job that performs integration testing on two
129 projects, those two projects will automatically share a virtual
130 change queue. If a third project does not invoke that job, it
131 will be part of a separate virtual change queue, and changes to it
132 will not depend on changes to the first two jobs.
133
134 For more detail on the theory and operation of Zuul's
135 DependentQueueManager, see: :doc:`gating`.
136
137**trigger**
138 This describes what Gerrit events should be placed in the queue.
139 Triggers are not exclusive -- matching events may be placed in
140 multiple queues, and they will behave independently in each of the
141 queues they match. Multiple triggers may be listed. Further
142 parameters describe the kind of events that match:
143
144 *event*
145 The event name from gerrit. Examples: ``patchset-created``,
146 ``comment-added``, ``ref-updated``. This field is treated as a
147 regular expression.
148
149 *branch*
150 The branch associated with the event. Example: ``master``. This
151 field is treated as a regular expression, and multiple branches may
152 be listed.
153
154 *ref*
155 On ref-updated events, the branch parameter is not used, instead the
156 ref is provided. Currently Gerrit has the somewhat idiosyncratic
157 behavior of specifying bare refs for branch names (e.g., ``master``),
158 but full ref names for other kinds of refs (e.g., ``refs/tags/foo``).
159 Zuul matches what you put here exactly against what Gerrit
160 provides. This field is treated as a regular expression, and
161 multiple refs may be listed.
162
163 *approval*
164 This is only used for ``comment-added`` events. It only matches if
165 the event has a matching approval associated with it. Example:
166 ``code-review: 2`` matches a ``+2`` vote on the code review category.
167 Multiple approvals may be listed.
168
Clark Boylanb9bcb402012-06-29 17:44:05 -0700169 *comment_filter*
170 This is only used for ``comment-added`` events. It accepts a list of
171 regexes that are searched for in the comment string. If any of these
172 regexes matches a portion of the comment string the trigger is
173 matched. ``comment_filter: retrigger`` will match when comments
174 containing 'retrigger' somewhere in the comment text are added to a
175 change.
176
James E. Blaircdd00072012-06-08 19:17:28 -0700177**success**
178 Describes what Zuul should do if all the jobs complete successfully.
179 This section is optional; if it is omitted, Zuul will run jobs and
180 do nothing on success; it will not even report a message to Gerrit.
181 If the section is present, it will leave a message on the Gerrit
182 review. Each additional argument is assumed to be an argument to
183 ``gerrit review``, with the boolean value of ``true`` simply
184 indicating that the argument should be present without following it
185 with a value. For example, ``verified: 1`` becomes ``gerrit
186 review --verified 1`` and ``submit: true`` becomes ``gerrit review
187 --submit``.
188
189**failure**
190 Uses the same syntax as **success**, but describes what Zuul should
191 do if at least one job fails.
James E. Blairdc253862012-06-13 17:12:42 -0700192
193**start**
194 Uses the same syntax as **success**, but describes what Zuul should
195 do when a change is added to the queue manager. This can be used,
196 for example, to reset the value of the Verified review category.
James E. Blaircdd00072012-06-08 19:17:28 -0700197
198Some example queue configurations are included in the sample layout
199file. The first is called a *check* queue::
200
201 - name: check
202 manager: IndependentQueueManager
203 trigger:
204 - event: patchset-created
205 success:
206 verified: 1
207 failure:
208 verified: -1
209
210This will trigger jobs each time a new patchset (or change) is
211uploaded to Gerrit, and report +/-1 values to Gerrit in the
212``verified`` review category. ::
213
214 - name: gate
215 manager: DependentQueueManager
216 trigger:
217 - event: comment-added
218 approval:
219 - approved: 1
220 success:
221 verified: 2
222 submit: true
223 failure:
224 verified: -2
225
226This will trigger jobs whenever a reviewer leaves a vote of ``1`` in the
227``approved`` review category in Gerrit (a non-standard category).
228Changes will be tested in such a way as to guarantee that they will be
229merged exactly as tested, though that will happen in parallel by
230creating a virtual queue of dependent changes and performing
231speculative execution of jobs. ::
232
233 - name: post
234 manager: IndependentQueueManager
235 trigger:
236 - event: ref-updated
237 ref: ^(?!refs/).*$
238
239This will trigger jobs whenever a change is merged to a named branch
240(e.g., ``master``). No output will be reported to Gerrit. This is
241useful for side effects such as creating per-commit tarballs. ::
242
243 - name: silent
244 manager: IndependentQueueManager
245 trigger:
246 - event: patchset-created
247
248This also triggers jobs when changes are uploaded to Gerrit, but no
249results are reported to Gerrit. This is useful for jobs that are in
250development and not yet ready to be presented to developers.
251
252Jobs
253""""
254
255The jobs section is optional, and can be used to set attributes of
256jobs that are independent of their association with a project. For
257example, if a job should return a customized message on failure, that
258may be specified here. Otherwise, Zuul does not need to be told about
259each job as it builds a list from the project specification.
260
261**name**
262 The name of the job. This field is treated as a regular expression
263 and will be applied to each job that matches.
264
James E. Blaire5a847f2012-07-10 15:29:14 -0700265**failure-message (optional)**
266 The message that should be reported to Gerrit if the job fails.
James E. Blaircdd00072012-06-08 19:17:28 -0700267
James E. Blaire5a847f2012-07-10 15:29:14 -0700268**success-message (optional)**
269 The message that should be reported to Gerrit if the job fails.
James E. Blaircdd00072012-06-08 19:17:28 -0700270
James E. Blair222d4982012-07-16 09:31:19 -0700271**hold-following-changes (optional)**
272 This is a boolean that indicates that changes that follow this
273 change in a dependent change queue should wait until this job
274 succeeds before launching. If this is applied to a very short job
275 that can predict whether longer jobs will fail early, this can be
276 used to reduce the number of jobs that Zuul will launch and
277 ultimately have to cancel. In that case, a small amount of
278 paralellization of jobs is traded for more efficient use of testing
279 resources. On the other hand, to apply this to a long running job
280 would largely defeat the parallelization of dependent change testing
281 that is the main feature of Zuul. The default is False.
282
James E. Blaire5a847f2012-07-10 15:29:14 -0700283**branch (optional)**
James E. Blaircdd00072012-06-08 19:17:28 -0700284 This job should only be run on matching branches. This field is
285 treated as a regular expression and multiple branches may be
286 listed.
287
James E. Blaire5a847f2012-07-10 15:29:14 -0700288**parameter-function (optional)**
289 Specifies a function that should be applied to the parameters before
290 the job is launched. The function should be defined in a python file
291 included with the :ref:`includes` directive. The function
292 should have the following signature:
293
294 .. function:: parameters(change, parameters)
295
296 Manipulate the parameters passed to a job before a build is
297 launched. The ``parameters`` dictionary will already contain the
298 standard Zuul job parameters, and is expected to be modified
299 in-place.
300
301 :param change: the current change
302 :type change: zuul.model.Change
303 :param parameters: parameters to be passed to the job
304 :type parameters: dict
305
James E. Blaircdd00072012-06-08 19:17:28 -0700306Here is an example of setting the failure message for jobs that check
307whether a change merges cleanly::
308
309 - name: ^.*-merge$
310 failure-message: This change was unable to be automatically merged
311 with the current state of the repository. Please rebase your
312 change and upload a new patchset.
313
314Projects
315""""""""
316
317The projects section indicates what jobs should be run in each queue
318for events associated with each project. It contains a list of
319projects. Here is an example::
320
321 - name: example/project
322 check:
323 - project-merge:
324 - project-unittest
325 - project-pep8
326 - project-pyflakes
327 gate:
328 - project-merge:
329 - project-unittest
330 - project-pep8
331 - project-pyflakes
332 post:
333 - project-publish
334
335**name**
336 The name of the project (as known by Gerrit).
337
338This is followed by a section for each of the queues defined above.
339Queues may be omitted if no jobs should run for this project in a
340given queue. Within the queue section, the jobs that should be
341executed are listed. If a job is entered as a dictionary key, then
342jobs contained within that key are only executed if the key job
343succeeds. In the above example, project-unittest, project-pep8, and
344project-pyflakes are only executed if project-merge succeeds. This
345can help avoid running unnecessary jobs.
346
347.. seealso:: The OpenStack Zuul configuration for a comprehensive example: https://github.com/openstack/openstack-ci-puppet/blob/master/modules/openstack-ci-config/files/zuul/layout.yaml
348
349
350logging.conf
351~~~~~~~~~~~~
352This file is optional. If provided, it should be a standard
353:mod:`logging.config` module configuration file. If not present, Zuul will
354output all log messages of DEBUG level or higher to the console.
355
356Starting Zuul
357-------------
358
359To start Zuul, run **zuul-server**::
360
361 usage: zuul-server [-h] [-c CONFIG] [-d]
362
363 Project gating system.
364
365 optional arguments:
366 -h, --help show this help message and exit
367 -c CONFIG specify the config file
368 -d do not run as a daemon
369
370You may want to use the ``-d`` argument while you are initially setting
371up Zuul so you can detect any configuration errors quickly. Under
372normal operation, omit ``-d`` and let Zuul run as a daemon.
373
374If you send signal 1 (SIGHUP) to the zuul-server process, Zuul will
375stop executing new jobs, wait until all executing jobs are finished,
376reload its configuration, and resume. Any values in any of the
377configuration files may be changed, except the location of Zuul's PID
378file (a change to that will be ignored until Zuul is restarted).