James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 1 | :title: Zuul |
| 2 | |
| 3 | Zuul |
| 4 | ==== |
| 5 | |
| 6 | Configuration |
| 7 | ------------- |
| 8 | |
| 9 | Zuul has three configuration files: |
| 10 | |
| 11 | **zuul.conf** |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 12 | Connection information for Gerrit and Gearman, locations of the |
Łukasz Jernaś | 048acb4 | 2014-03-02 18:49:41 +0100 | [diff] [blame] | 13 | other config files. |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 14 | **layout.yaml** |
Łukasz Jernaś | 048acb4 | 2014-03-02 18:49:41 +0100 | [diff] [blame] | 15 | Project and pipeline configuration -- what Zuul does. |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 16 | **logging.conf** |
Łukasz Jernaś | 048acb4 | 2014-03-02 18:49:41 +0100 | [diff] [blame] | 17 | Python logging config. |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 18 | |
| 19 | Examples of each of the three files can be found in the etc/ directory |
| 20 | of the source distribution. |
| 21 | |
James E. Blair | 6aea36d | 2012-12-17 13:03:24 -0800 | [diff] [blame] | 22 | .. _zuulconf: |
| 23 | |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 24 | zuul.conf |
| 25 | ~~~~~~~~~ |
| 26 | |
| 27 | Zuul will look for ``/etc/zuul/zuul.conf`` or ``~/zuul.conf`` to |
| 28 | bootstrap its configuration. Alternately, you may specify ``-c |
| 29 | /path/to/zuul.conf`` on the command line. |
| 30 | |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 31 | Gerrit and Gearman connection information are each described in a |
| 32 | section of zuul.conf. The location of the other two configuration |
| 33 | files (as well as the location of the PID file when running Zuul as a |
| 34 | server) are specified in a third section. |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 35 | |
Clark Boylan | 9b67090 | 2012-09-28 13:47:56 -0700 | [diff] [blame] | 36 | The three sections of this config and their options are documented below. |
| 37 | You can also find an example zuul.conf file in the git |
| 38 | `repository |
K Jonathan Harker | 97e457e | 2013-02-26 13:29:38 -0800 | [diff] [blame] | 39 | <https://github.com/openstack-infra/zuul/blob/master/etc/zuul.conf-sample>`_ |
Clark Boylan | 9b67090 | 2012-09-28 13:47:56 -0700 | [diff] [blame] | 40 | |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 41 | gearman |
Clark Boylan | 9b67090 | 2012-09-28 13:47:56 -0700 | [diff] [blame] | 42 | """"""" |
| 43 | |
| 44 | **server** |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 45 | Hostname or IP address of the Gearman server. |
| 46 | ``server=gearman.example.com`` |
Clark Boylan | 9b67090 | 2012-09-28 13:47:56 -0700 | [diff] [blame] | 47 | |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 48 | **port** |
Łukasz Jernaś | 048acb4 | 2014-03-02 18:49:41 +0100 | [diff] [blame] | 49 | Port on which the Gearman server is listening. |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 50 | ``port=4730`` |
Clark Boylan | 9b67090 | 2012-09-28 13:47:56 -0700 | [diff] [blame] | 51 | |
James E. Blair | 77cc7b8 | 2013-07-15 13:22:37 -0700 | [diff] [blame] | 52 | gearman_server |
| 53 | """""""""""""" |
| 54 | |
| 55 | **start** |
| 56 | Whether to start the internal Gearman server (default: False). |
| 57 | ``start=true`` |
| 58 | |
| 59 | **log_config** |
| 60 | Path to log config file for internal Gearman server. |
| 61 | ``log_config=/etc/zuul/gearman-logging.yaml`` |
| 62 | |
Clark Boylan | 9b67090 | 2012-09-28 13:47:56 -0700 | [diff] [blame] | 63 | gerrit |
| 64 | """""" |
| 65 | |
| 66 | **server** |
| 67 | FQDN of Gerrit server. |
| 68 | ``server=review.example.com`` |
| 69 | |
Akihiro Motoki | c7d28a7 | 2014-03-22 09:21:38 +0900 | [diff] [blame] | 70 | **port** |
| 71 | Optional: Gerrit server port. |
| 72 | ``port=29418`` |
| 73 | |
Antoine Musso | 2747501 | 2012-11-26 09:53:01 +0100 | [diff] [blame] | 74 | **baseurl** |
| 75 | Optional: path to Gerrit web interface. Defaults to ``https://<value |
| 76 | of server>/``. ``baseurl=https://review.example.com/review_site/`` |
| 77 | |
Clark Boylan | 9b67090 | 2012-09-28 13:47:56 -0700 | [diff] [blame] | 78 | **user** |
| 79 | User name to use when logging into above server via ssh. |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 80 | ``user=zuul`` |
Clark Boylan | 9b67090 | 2012-09-28 13:47:56 -0700 | [diff] [blame] | 81 | |
| 82 | **sshkey** |
| 83 | Path to SSH key to use when logging into above server. |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 84 | ``sshkey=/home/zuul/.ssh/id_rsa`` |
Clark Boylan | 9b67090 | 2012-09-28 13:47:56 -0700 | [diff] [blame] | 85 | |
| 86 | zuul |
| 87 | """" |
| 88 | |
| 89 | **layout_config** |
James E. Blair | 4076e2b | 2014-01-28 12:42:20 -0800 | [diff] [blame] | 90 | Path to layout config file. Used by zuul-server only. |
Clark Boylan | 9b67090 | 2012-09-28 13:47:56 -0700 | [diff] [blame] | 91 | ``layout_config=/etc/zuul/layout.yaml`` |
| 92 | |
| 93 | **log_config** |
James E. Blair | a443013 | 2014-02-17 08:32:07 -0800 | [diff] [blame] | 94 | Path to log config file. Used by zuul-server only. |
Clark Boylan | 9b67090 | 2012-09-28 13:47:56 -0700 | [diff] [blame] | 95 | ``log_config=/etc/zuul/logging.yaml`` |
| 96 | |
| 97 | **pidfile** |
James E. Blair | a443013 | 2014-02-17 08:32:07 -0800 | [diff] [blame] | 98 | Path to PID lock file. Used by zuul-server only. |
Clark Boylan | 9b67090 | 2012-09-28 13:47:56 -0700 | [diff] [blame] | 99 | ``pidfile=/var/run/zuul/zuul.pid`` |
| 100 | |
| 101 | **state_dir** |
James E. Blair | 4076e2b | 2014-01-28 12:42:20 -0800 | [diff] [blame] | 102 | Path to directory that Zuul should save state to. Used by all Zuul |
| 103 | commands. |
Clark Boylan | 9b67090 | 2012-09-28 13:47:56 -0700 | [diff] [blame] | 104 | ``state_dir=/var/lib/zuul`` |
| 105 | |
James E. Blair | 4076e2b | 2014-01-28 12:42:20 -0800 | [diff] [blame] | 106 | **report_times** |
| 107 | Boolean value (``true`` or ``false``) that determines if Zuul should |
| 108 | include elapsed times for each job in the textual report. Used by |
| 109 | zuul-server only. |
| 110 | ``report_times=true`` |
| 111 | |
| 112 | **status_url** |
| 113 | URL that will be posted in Zuul comments made to Gerrit changes when |
| 114 | starting jobs for a change. Used by zuul-server only. |
| 115 | ``status_url=https://zuul.example.com/status`` |
| 116 | |
Clark Boylan | e0b4bdb | 2014-06-03 17:01:25 -0700 | [diff] [blame] | 117 | **status_expiry** |
| 118 | Zuul will cache the status.json file for this many seconds. This is an |
| 119 | optional value and ``1`` is used by default. |
| 120 | ``status_expiry=1`` |
| 121 | |
James E. Blair | 4076e2b | 2014-01-28 12:42:20 -0800 | [diff] [blame] | 122 | **url_pattern** |
| 123 | If you are storing build logs external to the system that originally |
| 124 | ran jobs and wish to link to those logs when Zuul makes comments on |
| 125 | Gerrit changes for completed jobs this setting configures what the |
| 126 | URLs for those links should be. Used by zuul-server only. |
| 127 | ``http://logs.example.com/{change.number}/{change.patchset}/{pipeline.name}/{job.name}/{build.number}`` |
| 128 | |
| 129 | **job_name_in_report** |
| 130 | Boolean value (``true`` or ``false``) that indicates whether the |
| 131 | job name should be included in the report (normally only the URL |
| 132 | is included). Defaults to ``false``. Used by zuul-server only. |
| 133 | ``job_name_in_report=true`` |
| 134 | |
| 135 | merger |
| 136 | """""" |
| 137 | |
Clark Boylan | 9b67090 | 2012-09-28 13:47:56 -0700 | [diff] [blame] | 138 | **git_dir** |
| 139 | Directory that Zuul should clone local git repositories to. |
| 140 | ``git_dir=/var/lib/zuul/git`` |
| 141 | |
Paul Belanger | b67aba1 | 2013-05-13 19:22:14 -0400 | [diff] [blame] | 142 | **git_user_email** |
| 143 | Optional: Value to pass to `git config user.email`. |
| 144 | ``git_user_email=zuul@example.com`` |
| 145 | |
| 146 | **git_user_name** |
| 147 | Optional: Value to pass to `git config user.name`. |
| 148 | ``git_user_name=zuul`` |
| 149 | |
Arx Cruz | b1b010d | 2013-10-28 19:49:59 -0200 | [diff] [blame] | 150 | **zuul_url** |
James E. Blair | 4076e2b | 2014-01-28 12:42:20 -0800 | [diff] [blame] | 151 | URL of this merger's git repos, accessible to test workers. Usually |
| 152 | "http://zuul.example.com/p" or "http://zuul-merger01.example.com/p" |
| 153 | depending on whether the merger is co-located with the Zuul server. |
Arx Cruz | b1b010d | 2013-10-28 19:49:59 -0200 | [diff] [blame] | 154 | |
James E. Blair | a443013 | 2014-02-17 08:32:07 -0800 | [diff] [blame] | 155 | **log_config** |
| 156 | Path to log config file for the merger process. |
| 157 | ``log_config=/etc/zuul/logging.yaml`` |
| 158 | |
| 159 | **pidfile** |
| 160 | Path to PID lock file for the merger process. |
| 161 | ``pidfile=/var/run/zuul-merger/merger.pid`` |
| 162 | |
Joshua Hesketh | 5fea867 | 2013-08-19 17:32:01 +1000 | [diff] [blame] | 163 | smtp |
| 164 | """" |
| 165 | |
| 166 | **server** |
| 167 | SMTP server hostname or address to use. |
| 168 | ``server=localhost`` |
| 169 | |
Akihiro Motoki | c7d28a7 | 2014-03-22 09:21:38 +0900 | [diff] [blame] | 170 | **port** |
| 171 | Optional: SMTP server port. |
| 172 | ``port=25`` |
| 173 | |
Joshua Hesketh | 5fea867 | 2013-08-19 17:32:01 +1000 | [diff] [blame] | 174 | **default_from** |
| 175 | Who the email should appear to be sent from when emailing the report. |
| 176 | This can be overridden by individual pipelines. |
| 177 | ``default_from=zuul@example.com`` |
| 178 | |
| 179 | **default_to** |
| 180 | Who the report should be emailed to by default. |
| 181 | This can be overridden by individual pipelines. |
| 182 | ``default_to=you@example.com`` |
| 183 | |
Joshua Hesketh | 36c3fa5 | 2014-01-22 11:40:52 +1100 | [diff] [blame] | 184 | .. _swift: |
| 185 | |
| 186 | swift |
| 187 | """"" |
| 188 | |
| 189 | To send (optional) swift upload instructions this section must be |
| 190 | present. Multiple destinations can be defined in the :ref:`jobs` |
| 191 | section of the layout. |
| 192 | |
| 193 | **authurl** |
| 194 | The (keystone) Auth URL for swift |
| 195 | ``For example, https://identity.api.rackspacecloud.com/v2.0/`` |
| 196 | |
| 197 | Any of the `swiftclient connection parameters`_ can also be defined |
| 198 | here by the same name. Including the os_options by their key name ( |
| 199 | ``for example tenant_id``) |
| 200 | |
| 201 | .. _swiftclient connection parameters: http://docs.openstack.org/developer/python-swiftclient/swiftclient.html#module-swiftclient.client |
| 202 | |
| 203 | **X-Account-Meta-Temp-Url-Key** (optional) |
| 204 | This is the key used to sign the HMAC message. zuul will send the |
| 205 | key to swift for you so you only need to define it here. If you do |
| 206 | not set a key zuul will generate one automatically. |
| 207 | |
| 208 | **region_name** (optional) |
| 209 | The region name holding the swift container |
| 210 | ``For example, SYD`` |
| 211 | |
| 212 | Each destination defined by the :ref:`jobs` will have the following |
| 213 | default values that it may overwrite. |
| 214 | |
| 215 | **default_container** (optional) |
| 216 | Container name to place the log into |
| 217 | ``For example, logs`` |
| 218 | |
| 219 | **default_expiry** (optional) |
| 220 | How long the signed destination should be available for |
| 221 | ``default: 7200 (2hrs)`` |
| 222 | |
| 223 | **default_max_file_size** (optional) |
| 224 | The maximum size of an individual file |
| 225 | ``default: 104857600 (100MB)`` |
| 226 | |
| 227 | **default_max_file_count** (optional) |
| 228 | The maximum number of separate files to allow |
| 229 | ``default: 10`` |
| 230 | |
| 231 | **default_logserver_prefix** |
| 232 | Provide a URL to the CDN or logserver app so that a worker knows |
| 233 | what URL to return. The worker should return the logserver_prefix |
| 234 | url and the object path. |
| 235 | ``For example: http://logs.example.org/server.app?obj=`` |
| 236 | |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 237 | layout.yaml |
| 238 | ~~~~~~~~~~~ |
| 239 | |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 240 | This is the main configuration file for Zuul, where all of the pipelines |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 241 | and projects are defined, what tests should be run, and what actions |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 242 | Zuul should perform. There are three sections: pipelines, jobs, and |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 243 | projects. |
| 244 | |
James E. Blair | e5a847f | 2012-07-10 15:29:14 -0700 | [diff] [blame] | 245 | .. _includes: |
| 246 | |
| 247 | Includes |
| 248 | """""""" |
| 249 | |
| 250 | Custom functions to be used in Zuul's configuration may be provided |
| 251 | using the ``includes`` directive. It accepts a list of files to |
| 252 | include, and currently supports one type of inclusion, a python file:: |
| 253 | |
| 254 | includes: |
| 255 | - python-file: local_functions.py |
| 256 | |
| 257 | **python-file** |
| 258 | The path to a python file. The file will be loaded and objects that |
| 259 | it defines will be placed in a special environment which can be |
| 260 | referenced in the Zuul configuration. Currently only the |
| 261 | parameter-function attribute of a Job uses this feature. |
| 262 | |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 263 | Pipelines |
| 264 | """"""""" |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 265 | |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 266 | Zuul can have any number of independent pipelines. Whenever a matching |
| 267 | Gerrit event is found for a pipeline, that event is added to the |
| 268 | pipeline, and the jobs specified for that pipeline are run. When all |
| 269 | jobs specified for the pipeline that were triggered by an event are |
| 270 | completed, Zuul reports back to Gerrit the results. |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 271 | |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 272 | There are no pre-defined pipelines in Zuul, rather you can define |
| 273 | whatever pipelines you need in the layout file. This is a very flexible |
| 274 | system that can accommodate many kinds of workflows. |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 275 | |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 276 | Here is a quick example of a pipeline definition followed by an |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 277 | explanation of each of the parameters:: |
| 278 | |
| 279 | - name: check |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 280 | manager: IndependentPipelineManager |
James E. Blair | c0dedf8 | 2014-08-06 09:37:52 -0700 | [diff] [blame^] | 281 | source: gerrit |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 282 | trigger: |
James E. Blair | 6c358e7 | 2013-07-29 17:06:47 -0700 | [diff] [blame] | 283 | gerrit: |
| 284 | - event: patchset-created |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 285 | success: |
| 286 | verified: 1 |
| 287 | failure: |
| 288 | verified: -1 |
| 289 | |
| 290 | **name** |
| 291 | This is used later in the project definition to indicate what jobs |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 292 | should be run for events in the pipeline. |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 293 | |
James E. Blair | 8dbd56a | 2012-12-22 10:55:10 -0800 | [diff] [blame] | 294 | **description** |
| 295 | This is an optional field that may be used to provide a textual |
| 296 | description of the pipeline. |
| 297 | |
James E. Blair | c0dedf8 | 2014-08-06 09:37:52 -0700 | [diff] [blame^] | 298 | **source** |
| 299 | A required field that specifies a trigger that provides access to |
| 300 | the change objects that this pipeline operates on. Currently only |
| 301 | the value ``gerrit`` is supported. |
| 302 | |
James E. Blair | 5637019 | 2013-01-14 15:47:28 -0800 | [diff] [blame] | 303 | **success-message** |
| 304 | An optional field that supplies the introductory text in message |
| 305 | reported back to Gerrit when all the voting builds are successful. |
| 306 | Defaults to "Build successful." |
| 307 | |
| 308 | **failure-message** |
| 309 | An optional field that supplies the introductory text in message |
| 310 | reported back to Gerrit when at least one voting build fails. |
| 311 | Defaults to "Build failed." |
| 312 | |
Joshua Hesketh | b717977 | 2014-01-30 23:30:46 +1100 | [diff] [blame] | 313 | **merge-failure-message** |
| 314 | An optional field that supplies the introductory text in message |
| 315 | reported back to Gerrit when a change fails to merge with the |
| 316 | current state of the repository. |
| 317 | Defaults to "Merge failed." |
| 318 | |
Joshua Hesketh | 3979e3e | 2014-03-04 11:21:10 +1100 | [diff] [blame] | 319 | **footer-message** |
| 320 | An optional field to supply additional information after test results. |
| 321 | Useful for adding information about the CI system such as debugging |
| 322 | and contact details. |
| 323 | |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 324 | **manager** |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 325 | There are currently two schemes for managing pipelines: |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 326 | |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 327 | *IndependentPipelineManager* |
| 328 | Every event in this pipeline should be treated as independent of |
| 329 | other events in the pipeline. This is appropriate when the order of |
| 330 | events in the pipeline doesn't matter because the results of the |
| 331 | actions this pipeline performs can not affect other events in the |
| 332 | pipeline. For example, when a change is first uploaded for review, |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 333 | you may want to run tests on that change to provide early feedback |
| 334 | to reviewers. At the end of the tests, the change is not going to |
| 335 | be merged, so it is safe to run these tests in parallel without |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 336 | regard to any other changes in the pipeline. They are independent. |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 337 | |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 338 | Another type of pipeline that is independent is a post-merge |
| 339 | pipeline. In that case, the changes have already merged, so the |
| 340 | results can not affect any other events in the pipeline. |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 341 | |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 342 | *DependentPipelineManager* |
| 343 | The dependent pipeline manager is designed for gating. It ensures |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 344 | that every change is tested exactly as it is going to be merged |
| 345 | into the repository. An ideal gating system would test one change |
| 346 | at a time, applied to the tip of the repository, and only if that |
| 347 | change passed tests would it be merged. Then the next change in |
| 348 | line would be tested the same way. In order to achieve parallel |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 349 | testing of changes, the dependent pipeline manager performs |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 350 | speculative execution on changes. It orders changes based on |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 351 | their entry into the pipeline. It begins testing all changes in |
| 352 | parallel, assuming that each change ahead in the pipeline will pass |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 353 | its tests. If they all succeed, all the changes can be tested and |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 354 | merged in parallel. If a change near the front of the pipeline |
| 355 | fails its tests, each change behind it ignores whatever tests have |
| 356 | been completed and are tested again without the change in front. |
| 357 | This way gate tests may run in parallel but still be tested |
| 358 | correctly, exactly as they will appear in the repository when |
| 359 | merged. |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 360 | |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 361 | One important characteristic of the DependentPipelineManager is that |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 362 | it analyzes the jobs that are triggered by different projects, and |
| 363 | if those projects have jobs in common, it treats those projects as |
| 364 | related, and they share a single virtual queue of changes. Thus, |
| 365 | if there is a job that performs integration testing on two |
| 366 | projects, those two projects will automatically share a virtual |
| 367 | change queue. If a third project does not invoke that job, it |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 368 | will be part of a separate virtual change queue, and changes to |
| 369 | it will not depend on changes to the first two jobs. |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 370 | |
| 371 | For more detail on the theory and operation of Zuul's |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 372 | DependentPipelineManager, see: :doc:`gating`. |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 373 | |
| 374 | **trigger** |
James E. Blair | 6c358e7 | 2013-07-29 17:06:47 -0700 | [diff] [blame] | 375 | Exactly one trigger source must be supplied for each pipeline. |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 376 | Triggers are not exclusive -- matching events may be placed in |
James E. Blair | 6c358e7 | 2013-07-29 17:06:47 -0700 | [diff] [blame] | 377 | multiple pipelines, and they will behave independently in each of |
| 378 | the pipelines they match. You may select from the following: |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 379 | |
James E. Blair | 6c358e7 | 2013-07-29 17:06:47 -0700 | [diff] [blame] | 380 | **gerrit** |
| 381 | This describes what Gerrit events should be placed in the |
| 382 | pipeline. Multiple gerrit triggers may be listed. Further |
| 383 | parameters describe the kind of events that match: |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 384 | |
James E. Blair | 6c358e7 | 2013-07-29 17:06:47 -0700 | [diff] [blame] | 385 | *event* |
| 386 | The event name from gerrit. Examples: ``patchset-created``, |
| 387 | ``comment-added``, ``ref-updated``. This field is treated as a |
| 388 | regular expression. |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 389 | |
James E. Blair | 6c358e7 | 2013-07-29 17:06:47 -0700 | [diff] [blame] | 390 | *branch* |
| 391 | The branch associated with the event. Example: ``master``. This |
| 392 | field is treated as a regular expression, and multiple branches may |
| 393 | be listed. |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 394 | |
James E. Blair | 6c358e7 | 2013-07-29 17:06:47 -0700 | [diff] [blame] | 395 | *ref* |
| 396 | On ref-updated events, the branch parameter is not used, instead the |
| 397 | ref is provided. Currently Gerrit has the somewhat idiosyncratic |
| 398 | behavior of specifying bare refs for branch names (e.g., ``master``), |
| 399 | but full ref names for other kinds of refs (e.g., ``refs/tags/foo``). |
| 400 | Zuul matches what you put here exactly against what Gerrit |
| 401 | provides. This field is treated as a regular expression, and |
| 402 | multiple refs may be listed. |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 403 | |
James E. Blair | 6c358e7 | 2013-07-29 17:06:47 -0700 | [diff] [blame] | 404 | *approval* |
| 405 | This is only used for ``comment-added`` events. It only matches if |
| 406 | the event has a matching approval associated with it. Example: |
| 407 | ``code-review: 2`` matches a ``+2`` vote on the code review category. |
| 408 | Multiple approvals may be listed. |
Antoine Musso | b4e809e | 2012-12-06 16:58:06 +0100 | [diff] [blame] | 409 | |
James E. Blair | 1fbfceb | 2014-06-23 14:42:53 -0700 | [diff] [blame] | 410 | *email* |
James E. Blair | 6c358e7 | 2013-07-29 17:06:47 -0700 | [diff] [blame] | 411 | This is used for any event. It takes a regex applied on the performer |
Michael Prokop | 526926a | 2013-10-24 16:16:57 +0200 | [diff] [blame] | 412 | email, i.e. Gerrit account email address. If you want to specify |
James E. Blair | 6c358e7 | 2013-07-29 17:06:47 -0700 | [diff] [blame] | 413 | several email filters, you must use a YAML list. Make sure to use non |
| 414 | greedy matchers and to escapes dots! |
James E. Blair | 1fbfceb | 2014-06-23 14:42:53 -0700 | [diff] [blame] | 415 | Example: ``email: ^.*?@example\.org$``. |
James E. Blair | 6c358e7 | 2013-07-29 17:06:47 -0700 | [diff] [blame] | 416 | |
James E. Blair | 1fbfceb | 2014-06-23 14:42:53 -0700 | [diff] [blame] | 417 | *email_filter* (deprecated) |
| 418 | A deprecated alternate spelling of *email*. Only one of *email* or |
| 419 | *email_filter* should be used. |
| 420 | |
| 421 | *username* |
Joshua Hesketh | b8a817e | 2013-12-27 11:21:38 +1100 | [diff] [blame] | 422 | This is used for any event. It takes a regex applied on the performer |
| 423 | username, i.e. Gerrit account name. If you want to specify several |
| 424 | username filters, you must use a YAML list. Make sure to use non greedy |
| 425 | matchers and to escapes dots! |
James E. Blair | 1fbfceb | 2014-06-23 14:42:53 -0700 | [diff] [blame] | 426 | Example: ``username: ^jenkins$``. |
Joshua Hesketh | b8a817e | 2013-12-27 11:21:38 +1100 | [diff] [blame] | 427 | |
James E. Blair | 1fbfceb | 2014-06-23 14:42:53 -0700 | [diff] [blame] | 428 | *username_filter* (deprecated) |
| 429 | A deprecated alternate spelling of *username*. Only one of *username* or |
| 430 | *username_filter* should be used. |
| 431 | |
| 432 | *comment* |
James E. Blair | 6c358e7 | 2013-07-29 17:06:47 -0700 | [diff] [blame] | 433 | This is only used for ``comment-added`` events. It accepts a list of |
| 434 | regexes that are searched for in the comment string. If any of these |
| 435 | regexes matches a portion of the comment string the trigger is |
James E. Blair | 1fbfceb | 2014-06-23 14:42:53 -0700 | [diff] [blame] | 436 | matched. ``comment: retrigger`` will match when comments |
James E. Blair | 6c358e7 | 2013-07-29 17:06:47 -0700 | [diff] [blame] | 437 | containing 'retrigger' somewhere in the comment text are added to a |
| 438 | change. |
Clark Boylan | b9bcb40 | 2012-06-29 17:44:05 -0700 | [diff] [blame] | 439 | |
James E. Blair | 1fbfceb | 2014-06-23 14:42:53 -0700 | [diff] [blame] | 440 | *comment_filter* (deprecated) |
| 441 | A deprecated alternate spelling of *comment*. Only one of *comment* or |
| 442 | *comment_filter* should be used. |
| 443 | |
James E. Blair | 9c17dbf | 2014-06-23 14:21:58 -0700 | [diff] [blame] | 444 | *require-approval* |
James E. Blair | c053d02 | 2014-01-22 14:57:33 -0800 | [diff] [blame] | 445 | This may be used for any event. It requires that a certain kind |
| 446 | of approval be present for the current patchset of the change (the |
James E. Blair | 11041d2 | 2014-05-02 14:49:53 -0700 | [diff] [blame] | 447 | approval could be added by the event in question). It follows the |
James E. Blair | 9c17dbf | 2014-06-23 14:21:58 -0700 | [diff] [blame] | 448 | same syntax as the "approval" pipeline requirement below. |
James E. Blair | c053d02 | 2014-01-22 14:57:33 -0800 | [diff] [blame] | 449 | |
James E. Blair | 63bb0ef | 2013-07-29 17:14:51 -0700 | [diff] [blame] | 450 | **timer** |
| 451 | This trigger will run based on a cron-style time specification. |
| 452 | It will enqueue an event into its pipeline for every project |
| 453 | defined in the configuration. Any job associated with the |
| 454 | pipeline will run in response to that event. |
| 455 | |
| 456 | *time* |
| 457 | The time specification in cron syntax. Only the 5 part syntax is |
| 458 | supported, not the symbolic names. Example: ``0 0 * * *`` runs |
| 459 | at midnight. |
| 460 | |
James E. Blair | 11041d2 | 2014-05-02 14:49:53 -0700 | [diff] [blame] | 461 | **require** |
| 462 | If this section is present, it established pre-requisites for any |
| 463 | kind of item entering the Pipeline. Regardless of how the item is |
| 464 | to be enqueued (via any trigger or automatic dependency resolution), |
| 465 | the conditions specified here must be met or the item will not be |
| 466 | enqueued. |
| 467 | |
| 468 | **approval** |
| 469 | This requires that a certain kind of approval be present for the |
| 470 | current patchset of the change (the approval could be added by the |
| 471 | event in question). It takes several sub-parameters, all of which |
| 472 | are optional and are combined together so that there must be an |
| 473 | approval matching all specified requirements. |
| 474 | |
| 475 | *username* |
| 476 | If present, an approval from this username is required. |
| 477 | |
James E. Blair | 1fbfceb | 2014-06-23 14:42:53 -0700 | [diff] [blame] | 478 | *email* |
James E. Blair | 11041d2 | 2014-05-02 14:49:53 -0700 | [diff] [blame] | 479 | If present, an approval with this email address is required. It |
| 480 | is treated as a regular expression as above. |
| 481 | |
James E. Blair | 1fbfceb | 2014-06-23 14:42:53 -0700 | [diff] [blame] | 482 | *email-filter* (deprecated) |
| 483 | A deprecated alternate spelling of *email*. Only one of *email* or |
| 484 | *email_filter* should be used. |
| 485 | |
James E. Blair | 11041d2 | 2014-05-02 14:49:53 -0700 | [diff] [blame] | 486 | *older-than* |
| 487 | If present, the approval must be older than this amount of time |
| 488 | to match. Provide a time interval as a number with a suffix of |
| 489 | "w" (weeks), "d" (days), "h" (hours), "m" (minutes), "s" |
| 490 | (seconds). Example ``48h`` or ``2d``. |
| 491 | |
| 492 | *newer-than* |
| 493 | If present, the approval must be newer than this amount of time |
| 494 | to match. Same format as "older-than". |
| 495 | |
| 496 | Any other field is interpreted as a review category and value |
| 497 | pair. For example ``verified: 1`` would require that the approval |
James E. Blair | 9c17dbf | 2014-06-23 14:21:58 -0700 | [diff] [blame] | 498 | be for a +1 vote in the "Verified" column. The value may either |
| 499 | be a single value or a list: ``verified: [1, 2]`` would match |
| 500 | either a +1 or +2 vote. |
James E. Blair | 11041d2 | 2014-05-02 14:49:53 -0700 | [diff] [blame] | 501 | |
| 502 | **open** |
| 503 | A boolean value (``true`` or ``false``) that indicates whether the change |
| 504 | must be open or closed in order to be enqueued. |
| 505 | |
Clark Boylan | a9702ad | 2014-05-08 17:17:24 -0700 | [diff] [blame] | 506 | **current-patchset** |
| 507 | A boolean value (``true`` or ``false``) that indicates whether the change |
| 508 | must be the current patchset in order to be enqueued. |
| 509 | |
James E. Blair | 11041d2 | 2014-05-02 14:49:53 -0700 | [diff] [blame] | 510 | **status** |
| 511 | A string value that corresponds with the status of the change |
| 512 | reported by the trigger. For example, when using the Gerrit |
| 513 | trigger, status values such as ``NEW`` or ``MERGED`` may be useful. |
James E. Blair | 63bb0ef | 2013-07-29 17:14:51 -0700 | [diff] [blame] | 514 | |
James E. Blair | 2fa5096 | 2013-01-30 21:50:41 -0800 | [diff] [blame] | 515 | **dequeue-on-new-patchset** |
| 516 | Normally, if a new patchset is uploaded to a change that is in a |
| 517 | pipeline, the existing entry in the pipeline will be removed (with |
| 518 | jobs canceled and any dependent changes that can no longer merge as |
| 519 | well. To suppress this behavior (and allow jobs to continue |
| 520 | running), set this to ``false``. Default: ``true``. |
| 521 | |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 522 | **success** |
Joshua Hesketh | 1879cf7 | 2013-08-19 14:13:15 +1000 | [diff] [blame] | 523 | Describes where Zuul should report to if all the jobs complete |
| 524 | successfully. |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 525 | This section is optional; if it is omitted, Zuul will run jobs and |
| 526 | do nothing on success; it will not even report a message to Gerrit. |
Joshua Hesketh | 1879cf7 | 2013-08-19 14:13:15 +1000 | [diff] [blame] | 527 | If the section is present, the listed reporter plugins will be |
| 528 | asked to report on the jobs. |
| 529 | Each reporter's value dictionary is handled by the reporter. See |
| 530 | reporters for more details. |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 531 | |
Mathieu Gagné | d6d2a64 | 2013-06-11 20:59:58 -0400 | [diff] [blame] | 532 | **failure** |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 533 | Uses the same syntax as **success**, but describes what Zuul should |
| 534 | do if at least one job fails. |
James E. Blair | dc25386 | 2012-06-13 17:12:42 -0700 | [diff] [blame] | 535 | |
Joshua Hesketh | b717977 | 2014-01-30 23:30:46 +1100 | [diff] [blame] | 536 | **merge-failure** |
| 537 | Uses the same syntax as **success**, but describes what Zuul should |
| 538 | do if it is unable to merge in the patchset. If no merge-failure |
| 539 | reporters are listed then the ``failure`` reporters will be used to |
| 540 | notify of unsuccessful merges. |
| 541 | |
Mathieu Gagné | d6d2a64 | 2013-06-11 20:59:58 -0400 | [diff] [blame] | 542 | **start** |
James E. Blair | dc25386 | 2012-06-13 17:12:42 -0700 | [diff] [blame] | 543 | Uses the same syntax as **success**, but describes what Zuul should |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 544 | do when a change is added to the pipeline manager. This can be used, |
James E. Blair | dc25386 | 2012-06-13 17:12:42 -0700 | [diff] [blame] | 545 | for example, to reset the value of the Verified review category. |
Mathieu Gagné | d6d2a64 | 2013-06-11 20:59:58 -0400 | [diff] [blame] | 546 | |
James E. Blair | 64ed6f2 | 2013-07-10 14:07:23 -0700 | [diff] [blame] | 547 | **precedence** |
| 548 | Indicates how the build scheduler should prioritize jobs for |
| 549 | different pipelines. Each pipeline may have one precedence, jobs |
| 550 | for pipelines with a higher precedence will be run before ones with |
| 551 | lower. The value should be one of ``high``, ``normal``, or ``low``. |
| 552 | Default: ``normal``. |
| 553 | |
Clark Boylan | c2d19e4 | 2014-01-23 14:08:58 -0800 | [diff] [blame] | 554 | **window** |
| 555 | DependentPipelineManagers only. Zuul can rate limit |
| 556 | DependentPipelineManagers in a manner similar to TCP flow control. |
| 557 | Jobs are only started for changes in the queue if they sit in the |
| 558 | actionable window for the pipeline. The initial length of this window |
| 559 | is configurable with this value. The value given should be a positive |
| 560 | integer value. A value of ``0`` disables rate limiting on the |
| 561 | DependentPipelineManager. |
| 562 | Default: ``20``. |
| 563 | |
| 564 | **window-floor** |
| 565 | DependentPipelineManagers only. This is the minimum value for the |
| 566 | window described above. Should be a positive non zero integer value. |
| 567 | Default: ``3``. |
| 568 | |
| 569 | **window-increase-type** |
| 570 | DependentPipelineManagers only. This value describes how the window |
| 571 | should grow when changes are successfully merged by zuul. A value of |
| 572 | ``linear`` indicates that ``window-increase-factor`` should be added |
| 573 | to the previous window value. A value of ``exponential`` indicates |
| 574 | that ``window-increase-factor`` should be multiplied against the |
| 575 | previous window value and the result will become the window size. |
| 576 | Default: ``linear``. |
| 577 | |
| 578 | **window-increase-factor** |
| 579 | DependentPipelineManagers only. The value to be added or mulitplied |
| 580 | against the previous window value to determine the new window after |
| 581 | successful change merges. |
| 582 | Default: ``1``. |
| 583 | |
| 584 | **window-decrease-type** |
| 585 | DependentPipelineManagers only. This value describes how the window |
| 586 | should shrink when changes are not able to be merged by Zuul. A value |
| 587 | of ``linear`` indicates that ``window-decrease-factor`` should be |
| 588 | subtracted from the previous window value. A value of ``exponential`` |
| 589 | indicates that ``window-decrease-factor`` should be divided against |
| 590 | the previous window value and the result will become the window size. |
| 591 | Default: ``exponential``. |
| 592 | |
| 593 | **window-decrease-factor** |
| 594 | DependentPipelineManagers only. The value to be subtracted or divided |
| 595 | against the previous window value to determine the new window after |
| 596 | unsuccessful change merges. |
| 597 | Default: ``2``. |
| 598 | |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 599 | Some example pipeline configurations are included in the sample layout |
| 600 | file. The first is called a *check* pipeline:: |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 601 | |
| 602 | - name: check |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 603 | manager: IndependentPipelineManager |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 604 | trigger: |
| 605 | - event: patchset-created |
| 606 | success: |
Joshua Hesketh | 1879cf7 | 2013-08-19 14:13:15 +1000 | [diff] [blame] | 607 | gerrit: |
| 608 | verified: 1 |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 609 | failure: |
Joshua Hesketh | 1879cf7 | 2013-08-19 14:13:15 +1000 | [diff] [blame] | 610 | gerrit: |
| 611 | verified: -1 |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 612 | |
| 613 | This will trigger jobs each time a new patchset (or change) is |
| 614 | uploaded to Gerrit, and report +/-1 values to Gerrit in the |
| 615 | ``verified`` review category. :: |
| 616 | |
| 617 | - name: gate |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 618 | manager: DependentPipelineManager |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 619 | trigger: |
| 620 | - event: comment-added |
| 621 | approval: |
| 622 | - approved: 1 |
| 623 | success: |
Joshua Hesketh | 1879cf7 | 2013-08-19 14:13:15 +1000 | [diff] [blame] | 624 | gerrit: |
| 625 | verified: 2 |
| 626 | submit: true |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 627 | failure: |
Joshua Hesketh | 1879cf7 | 2013-08-19 14:13:15 +1000 | [diff] [blame] | 628 | gerrit: |
| 629 | verified: -2 |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 630 | |
| 631 | This will trigger jobs whenever a reviewer leaves a vote of ``1`` in the |
| 632 | ``approved`` review category in Gerrit (a non-standard category). |
| 633 | Changes will be tested in such a way as to guarantee that they will be |
| 634 | merged exactly as tested, though that will happen in parallel by |
| 635 | creating a virtual queue of dependent changes and performing |
| 636 | speculative execution of jobs. :: |
| 637 | |
| 638 | - name: post |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 639 | manager: IndependentPipelineManager |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 640 | trigger: |
| 641 | - event: ref-updated |
| 642 | ref: ^(?!refs/).*$ |
| 643 | |
| 644 | This will trigger jobs whenever a change is merged to a named branch |
| 645 | (e.g., ``master``). No output will be reported to Gerrit. This is |
| 646 | useful for side effects such as creating per-commit tarballs. :: |
| 647 | |
| 648 | - name: silent |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 649 | manager: IndependentPipelineManager |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 650 | trigger: |
| 651 | - event: patchset-created |
| 652 | |
| 653 | This also triggers jobs when changes are uploaded to Gerrit, but no |
| 654 | results are reported to Gerrit. This is useful for jobs that are in |
Antoine Musso | ce33384 | 2012-10-16 14:42:35 +0200 | [diff] [blame] | 655 | development and not yet ready to be presented to developers. :: |
| 656 | |
| 657 | pipelines: |
| 658 | - name: post-merge |
| 659 | manager: IndependentPipelineManager |
| 660 | trigger: |
| 661 | - event: change-merged |
| 662 | success: |
Joshua Hesketh | 1879cf7 | 2013-08-19 14:13:15 +1000 | [diff] [blame] | 663 | gerrit: |
| 664 | force-message: True |
Antoine Musso | ce33384 | 2012-10-16 14:42:35 +0200 | [diff] [blame] | 665 | failure: |
Joshua Hesketh | 1879cf7 | 2013-08-19 14:13:15 +1000 | [diff] [blame] | 666 | gerrit: |
| 667 | force-message: True |
Antoine Musso | ce33384 | 2012-10-16 14:42:35 +0200 | [diff] [blame] | 668 | |
| 669 | The ``change-merged`` events happen when a change has been merged in the git |
| 670 | repository. The change is thus closed and Gerrit will not accept modifications |
| 671 | to the review scoring such as ``code-review`` or ``verified``. By using the |
| 672 | ``force-message: True`` parameter, Zuul will pass ``--force-message`` to the |
| 673 | ``gerrit review`` command, thus making sure the message is actually |
| 674 | sent back to Gerrit regardless of approval scores. |
| 675 | That kind of pipeline is nice to run regression or performance tests. |
| 676 | |
| 677 | .. note:: |
| 678 | The ``change-merged`` event does not include the commit sha1 which can be |
| 679 | hazardous, it would let you report back to Gerrit though. If you were to |
Michael Prokop | 526926a | 2013-10-24 16:16:57 +0200 | [diff] [blame] | 680 | build a tarball for a specific commit, you should consider instead using |
Łukasz Jernaś | 048acb4 | 2014-03-02 18:49:41 +0100 | [diff] [blame] | 681 | the ``ref-updated`` event which does include the commit sha1 (but lacks the |
Antoine Musso | ce33384 | 2012-10-16 14:42:35 +0200 | [diff] [blame] | 682 | Gerrit change number). |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 683 | |
Joshua Hesketh | 36c3fa5 | 2014-01-22 11:40:52 +1100 | [diff] [blame] | 684 | |
| 685 | .. _jobs: |
| 686 | |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 687 | Jobs |
| 688 | """" |
| 689 | |
| 690 | The jobs section is optional, and can be used to set attributes of |
| 691 | jobs that are independent of their association with a project. For |
| 692 | example, if a job should return a customized message on failure, that |
| 693 | may be specified here. Otherwise, Zuul does not need to be told about |
| 694 | each job as it builds a list from the project specification. |
| 695 | |
| 696 | **name** |
| 697 | The name of the job. This field is treated as a regular expression |
| 698 | and will be applied to each job that matches. |
| 699 | |
James E. Blair | c8a1e05 | 2014-02-25 09:29:26 -0800 | [diff] [blame] | 700 | **queue-name (optional)** |
| 701 | Zuul will automatically combine projects that share a job into |
| 702 | shared change queues for dependent pipeline managers. In order to |
| 703 | report statistics about these queues, it is convenient for them to |
| 704 | have names. Zuul can automatically name change queues, however |
| 705 | these can grow quite long and are prone to changing as projects in |
| 706 | the queue change. If you assign a queue-name to a job, Zuul will |
| 707 | use that as the name for the shared change queue that contains that |
| 708 | job instead of the automatically generated one. It is an error for |
| 709 | a shared change queue to have more than one job with a queue-name if |
| 710 | they are not the same. |
| 711 | |
James E. Blair | e5a847f | 2012-07-10 15:29:14 -0700 | [diff] [blame] | 712 | **failure-message (optional)** |
| 713 | The message that should be reported to Gerrit if the job fails. |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 714 | |
James E. Blair | e5a847f | 2012-07-10 15:29:14 -0700 | [diff] [blame] | 715 | **success-message (optional)** |
| 716 | The message that should be reported to Gerrit if the job fails. |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 717 | |
James E. Blair | 6aea36d | 2012-12-17 13:03:24 -0800 | [diff] [blame] | 718 | **failure-pattern (optional)** |
| 719 | The URL that should be reported to Gerrit if the job fails. |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 720 | Defaults to the build URL or the url_pattern configured in |
James E. Blair | 6aea36d | 2012-12-17 13:03:24 -0800 | [diff] [blame] | 721 | zuul.conf. May be supplied as a string pattern with substitutions |
| 722 | as described in url_pattern in :ref:`zuulconf`. |
| 723 | |
| 724 | **success-pattern (optional)** |
| 725 | The URL that should be reported to Gerrit if the job succeeds. |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 726 | Defaults to the build URL or the url_pattern configured in |
James E. Blair | 6aea36d | 2012-12-17 13:03:24 -0800 | [diff] [blame] | 727 | zuul.conf. May be supplied as a string pattern with substitutions |
| 728 | as described in url_pattern in :ref:`zuulconf`. |
| 729 | |
James E. Blair | 222d498 | 2012-07-16 09:31:19 -0700 | [diff] [blame] | 730 | **hold-following-changes (optional)** |
| 731 | This is a boolean that indicates that changes that follow this |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 732 | change in a dependent change pipeline should wait until this job |
James E. Blair | 222d498 | 2012-07-16 09:31:19 -0700 | [diff] [blame] | 733 | succeeds before launching. If this is applied to a very short job |
| 734 | that can predict whether longer jobs will fail early, this can be |
| 735 | used to reduce the number of jobs that Zuul will launch and |
| 736 | ultimately have to cancel. In that case, a small amount of |
Mathieu Gagné | d6d2a64 | 2013-06-11 20:59:58 -0400 | [diff] [blame] | 737 | parallelization of jobs is traded for more efficient use of testing |
James E. Blair | 222d498 | 2012-07-16 09:31:19 -0700 | [diff] [blame] | 738 | resources. On the other hand, to apply this to a long running job |
| 739 | would largely defeat the parallelization of dependent change testing |
Mathieu Gagné | d6d2a64 | 2013-06-11 20:59:58 -0400 | [diff] [blame] | 740 | that is the main feature of Zuul. Default: ``false``. |
James E. Blair | 222d498 | 2012-07-16 09:31:19 -0700 | [diff] [blame] | 741 | |
James E. Blair | e5a847f | 2012-07-10 15:29:14 -0700 | [diff] [blame] | 742 | **branch (optional)** |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 743 | This job should only be run on matching branches. This field is |
| 744 | treated as a regular expression and multiple branches may be |
| 745 | listed. |
| 746 | |
James E. Blair | 70c7158 | 2013-03-06 08:50:50 -0800 | [diff] [blame] | 747 | **files (optional)** |
| 748 | This job should only be run if at least one of the files involved in |
| 749 | the change (added, deleted, or modified) matches at least one of the |
| 750 | file patterns listed here. This field is treated as a regular |
| 751 | expression and multiple expressions may be listed. |
| 752 | |
Mathieu Gagné | d6d2a64 | 2013-06-11 20:59:58 -0400 | [diff] [blame] | 753 | **voting (optional)** |
| 754 | Boolean value (``true`` or ``false``) that indicates whatever |
| 755 | a job is voting or not. Default: ``true``. |
| 756 | |
James E. Blair | e5a847f | 2012-07-10 15:29:14 -0700 | [diff] [blame] | 757 | **parameter-function (optional)** |
| 758 | Specifies a function that should be applied to the parameters before |
| 759 | the job is launched. The function should be defined in a python file |
| 760 | included with the :ref:`includes` directive. The function |
| 761 | should have the following signature: |
| 762 | |
James E. Blair | d047097 | 2013-07-29 10:05:43 -0700 | [diff] [blame] | 763 | .. function:: parameters(item, job, parameters) |
James E. Blair | e5a847f | 2012-07-10 15:29:14 -0700 | [diff] [blame] | 764 | |
| 765 | Manipulate the parameters passed to a job before a build is |
| 766 | launched. The ``parameters`` dictionary will already contain the |
| 767 | standard Zuul job parameters, and is expected to be modified |
| 768 | in-place. |
| 769 | |
James E. Blair | d78576a | 2013-07-09 10:39:17 -0700 | [diff] [blame] | 770 | :param item: the current queue item |
| 771 | :type item: zuul.model.QueueItem |
James E. Blair | d047097 | 2013-07-29 10:05:43 -0700 | [diff] [blame] | 772 | :param job: the job about to be run |
| 773 | :type job: zuul.model.Job |
James E. Blair | e5a847f | 2012-07-10 15:29:14 -0700 | [diff] [blame] | 774 | :param parameters: parameters to be passed to the job |
| 775 | :type parameters: dict |
| 776 | |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 777 | If the parameter **ZUUL_NODE** is set by this function, then it will |
| 778 | be used to specify on what node (or class of node) the job should be |
| 779 | run. |
| 780 | |
Joshua Hesketh | 36c3fa5 | 2014-01-22 11:40:52 +1100 | [diff] [blame] | 781 | **swift** |
| 782 | If :ref:`swift` is configured then each job can define a destination |
| 783 | container for the builder to place logs and/or assets into. Multiple |
| 784 | containers can be listed for each job by providing a unique ``name``. |
| 785 | |
| 786 | *name* |
| 787 | Set an identifying name for the container. This is used in the |
| 788 | parameter key sent to the builder. For example if it ``logs`` then |
| 789 | one of the parameters sent will be ``SWIFT_logs_CONTAINER`` |
| 790 | (case-sensitive). |
| 791 | |
| 792 | Each of the defaults defined in :ref:`swift` can be overwritten as: |
| 793 | |
| 794 | *container* (optional) |
| 795 | Container name to place the log into |
| 796 | ``For example, logs`` |
| 797 | |
| 798 | *expiry* (optional) |
| 799 | How long the signed destination should be available for |
| 800 | |
James E. Blair | d650023 | 2014-06-23 15:05:48 -0700 | [diff] [blame] | 801 | *max-file-size** (optional) |
Joshua Hesketh | 36c3fa5 | 2014-01-22 11:40:52 +1100 | [diff] [blame] | 802 | The maximum size of an individual file |
| 803 | |
James E. Blair | d650023 | 2014-06-23 15:05:48 -0700 | [diff] [blame] | 804 | *max_file_size** (optional, deprecated) |
| 805 | A deprecated alternate spelling of *max-file-size*. |
| 806 | |
| 807 | *max-file-count* (optional) |
Joshua Hesketh | 36c3fa5 | 2014-01-22 11:40:52 +1100 | [diff] [blame] | 808 | The maximum number of separate files to allow |
| 809 | |
James E. Blair | d650023 | 2014-06-23 15:05:48 -0700 | [diff] [blame] | 810 | *max_file_count* (optional, deprecated) |
| 811 | A deprecated alternate spelling of *max-file-count*. |
| 812 | |
| 813 | *logserver-prefix* |
Joshua Hesketh | 36c3fa5 | 2014-01-22 11:40:52 +1100 | [diff] [blame] | 814 | Provide a URL to the CDN or logserver app so that a worker knows |
| 815 | what URL to return. |
| 816 | ``For example: http://logs.example.org/server.app?obj=`` |
James E. Blair | d650023 | 2014-06-23 15:05:48 -0700 | [diff] [blame] | 817 | The worker should return the logserver-prefix url and the object |
Joshua Hesketh | 36c3fa5 | 2014-01-22 11:40:52 +1100 | [diff] [blame] | 818 | path as the URL in the results data packet. |
| 819 | |
James E. Blair | d650023 | 2014-06-23 15:05:48 -0700 | [diff] [blame] | 820 | *logserver_prefix* (deprecated) |
| 821 | A deprecated alternate spelling of *logserver-prefix*. |
| 822 | |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 823 | Here is an example of setting the failure message for jobs that check |
| 824 | whether a change merges cleanly:: |
| 825 | |
| 826 | - name: ^.*-merge$ |
| 827 | failure-message: This change was unable to be automatically merged |
| 828 | with the current state of the repository. Please rebase your |
| 829 | change and upload a new patchset. |
| 830 | |
| 831 | Projects |
| 832 | """""""" |
| 833 | |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 834 | The projects section indicates what jobs should be run in each pipeline |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 835 | for events associated with each project. It contains a list of |
| 836 | projects. Here is an example:: |
| 837 | |
| 838 | - name: example/project |
| 839 | check: |
| 840 | - project-merge: |
| 841 | - project-unittest |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 842 | - project-pep8 |
| 843 | - project-pyflakes |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 844 | gate: |
| 845 | - project-merge: |
| 846 | - project-unittest |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 847 | - project-pep8 |
| 848 | - project-pyflakes |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 849 | post: |
| 850 | - project-publish |
| 851 | |
| 852 | **name** |
| 853 | The name of the project (as known by Gerrit). |
| 854 | |
James E. Blair | 19deff2 | 2013-08-25 13:17:35 -0700 | [diff] [blame] | 855 | **merge-mode (optional)** |
| 856 | An optional value that indicates what strategy should be used to |
| 857 | merge changes to this project. Supported values are: |
| 858 | |
| 859 | ** merge-resolve ** |
| 860 | Equivalent to 'git merge -s resolve'. This corresponds closely to |
| 861 | what Gerrit performs (using JGit) for a project if the "Merge if |
| 862 | necessary" merge mode is selected and "Automatically resolve |
| 863 | conflicts" is checked. This is the default. |
| 864 | |
| 865 | ** merge ** |
| 866 | Equivalent to 'git merge'. |
| 867 | |
| 868 | ** cherry-pick ** |
| 869 | Equivalent to 'git cherry-pick'. |
| 870 | |
Clark Boylan | 00635dc | 2012-09-19 14:03:08 -0700 | [diff] [blame] | 871 | This is followed by a section for each of the pipelines defined above. |
| 872 | Pipelines may be omitted if no jobs should run for this project in a |
| 873 | given pipeline. Within the pipeline section, the jobs that should be |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 874 | executed are listed. If a job is entered as a dictionary key, then |
| 875 | jobs contained within that key are only executed if the key job |
| 876 | succeeds. In the above example, project-unittest, project-pep8, and |
| 877 | project-pyflakes are only executed if project-merge succeeds. This |
| 878 | can help avoid running unnecessary jobs. |
| 879 | |
James E. Blair | 18c6444 | 2014-03-18 10:14:45 -0700 | [diff] [blame] | 880 | The special job named ``noop`` is internal to Zuul and will always |
| 881 | return ``SUCCESS`` immediately. This can be useful if you require |
| 882 | that all changes be processed by a pipeline but a project has no jobs |
| 883 | that can be run on it. |
| 884 | |
Paul Belanger | ffef9e4 | 2013-02-11 22:15:18 -0500 | [diff] [blame] | 885 | .. seealso:: The OpenStack Zuul configuration for a comprehensive example: https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/zuul/layout.yaml |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 886 | |
Antoine Musso | 80edd5a | 2013-02-13 15:37:53 +0100 | [diff] [blame] | 887 | Project Templates |
| 888 | """"""""""""""""" |
| 889 | |
Michael Prokop | 526926a | 2013-10-24 16:16:57 +0200 | [diff] [blame] | 890 | Whenever you have lot of similar projects (such as plugins for a project) you |
Antoine Musso | 80edd5a | 2013-02-13 15:37:53 +0100 | [diff] [blame] | 891 | will most probably want to use the same pipeline configurations. The |
| 892 | project templates let you define pipelines and job name templates to trigger. |
| 893 | One can then just apply the template on its project which make it easier to |
Michael Prokop | 526926a | 2013-10-24 16:16:57 +0200 | [diff] [blame] | 894 | update several similar projects. As an example:: |
Antoine Musso | 80edd5a | 2013-02-13 15:37:53 +0100 | [diff] [blame] | 895 | |
| 896 | project-templates: |
| 897 | # Name of the template |
| 898 | - name: plugin-triggering |
| 899 | # Definition of pipelines just like for a `project` |
| 900 | check: |
| 901 | - '{jobprefix}-merge': |
| 902 | - '{jobprefix}-pep8' |
| 903 | - '{jobprefix}-pyflakes' |
| 904 | gate: |
| 905 | - '{jobprefix}-merge': |
| 906 | - '{jobprefix}-unittest' |
| 907 | - '{jobprefix}-pep8' |
| 908 | - '{jobprefix}-pyflakes' |
| 909 | |
| 910 | In your projects definition, you will then apply the template using the template |
| 911 | key:: |
| 912 | |
| 913 | projects: |
| 914 | - name: plugin/foobar |
| 915 | template: |
| 916 | - name: plugin-triggering |
| 917 | jobprefix: plugin-foobar |
| 918 | |
James E. Blair | aea6cf6 | 2013-12-16 15:38:12 -0800 | [diff] [blame] | 919 | You can pass several parameters to a template. A ``parameter`` value |
| 920 | will be used for expansion of ``{parameter}`` in the template |
| 921 | strings. The parameter ``name`` will be automatically provided and |
| 922 | will contain the short name of the project, that is the portion of the |
| 923 | project name after the last ``/`` character. |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 924 | |
James E. Blair | 3e98c02 | 2013-12-16 15:25:38 -0800 | [diff] [blame] | 925 | Multiple templates can be combined in a project, and the jobs from all |
| 926 | of those templates will be added to the project. Individual jobs may |
| 927 | also be added:: |
| 928 | |
| 929 | projects: |
| 930 | - name: plugin/foobar |
| 931 | template: |
| 932 | - name: plugin-triggering |
| 933 | jobprefix: plugin-foobar |
| 934 | - name: plugin-extras |
| 935 | jobprefix: plugin-foobar |
| 936 | check: |
| 937 | - foobar-extra-special-job |
| 938 | |
| 939 | The order of the jobs listed in the project (which only affects the |
| 940 | order of jobs listed on the report) will be the jobs from each |
| 941 | template in the order listed, followed by any jobs individually listed |
| 942 | for the project. |
| 943 | |
| 944 | Note that if multiple templates are used for a project and one |
| 945 | template specifies a job that is also specified in another template, |
James E. Blair | 12a92b1 | 2014-03-26 11:54:53 -0700 | [diff] [blame] | 946 | or specified in the project itself, the configuration defined by |
| 947 | either the last template or the project itself will take priority. |
James E. Blair | 3e98c02 | 2013-12-16 15:25:38 -0800 | [diff] [blame] | 948 | |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 949 | logging.conf |
| 950 | ~~~~~~~~~~~~ |
| 951 | This file is optional. If provided, it should be a standard |
| 952 | :mod:`logging.config` module configuration file. If not present, Zuul will |
| 953 | output all log messages of DEBUG level or higher to the console. |
| 954 | |
| 955 | Starting Zuul |
| 956 | ------------- |
| 957 | |
| 958 | To start Zuul, run **zuul-server**:: |
| 959 | |
Antoine Musso | b3aa828 | 2013-04-19 15:16:59 +0200 | [diff] [blame] | 960 | usage: zuul-server [-h] [-c CONFIG] [-l LAYOUT] [-d] [-t] [--version] |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 961 | |
| 962 | Project gating system. |
| 963 | |
| 964 | optional arguments: |
| 965 | -h, --help show this help message and exit |
| 966 | -c CONFIG specify the config file |
Antoine Musso | b3aa828 | 2013-04-19 15:16:59 +0200 | [diff] [blame] | 967 | -l LAYOUT specify the layout file |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 968 | -d do not run as a daemon |
Antoine Musso | b3aa828 | 2013-04-19 15:16:59 +0200 | [diff] [blame] | 969 | -t validate layout file syntax |
| 970 | --version show zuul version |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 971 | |
| 972 | You may want to use the ``-d`` argument while you are initially setting |
| 973 | up Zuul so you can detect any configuration errors quickly. Under |
| 974 | normal operation, omit ``-d`` and let Zuul run as a daemon. |
| 975 | |
| 976 | If you send signal 1 (SIGHUP) to the zuul-server process, Zuul will |
| 977 | stop executing new jobs, wait until all executing jobs are finished, |
| 978 | reload its configuration, and resume. Any values in any of the |
| 979 | configuration files may be changed, except the location of Zuul's PID |
| 980 | file (a change to that will be ignored until Zuul is restarted). |
Clark Boylan | f231fa2 | 2013-02-08 12:28:53 -0800 | [diff] [blame] | 981 | |
| 982 | If you send a SIGUSR1 to the zuul-server process, Zuul will stop |
| 983 | executing new jobs, wait until all executing jobs are finished, |
| 984 | then exit. While waiting to exit Zuul will queue Gerrit events and |
| 985 | save these events prior to exiting. When Zuul starts again it will |
| 986 | read these saved events and act on them. |
Jeremy Stanley | 93e05f4 | 2013-03-08 17:29:17 +0000 | [diff] [blame] | 987 | |
Michael Prokop | 526926a | 2013-10-24 16:16:57 +0200 | [diff] [blame] | 988 | If you need to abort Zuul and intend to manually requeue changes for |
Jeremy Stanley | 93e05f4 | 2013-03-08 17:29:17 +0000 | [diff] [blame] | 989 | jobs which were running in its pipelines, prior to terminating you can |
| 990 | use the zuul-changes.py tool script to simplify the process. For |
| 991 | example, this would give you a list of Gerrit commands to reverify or |
| 992 | recheck changes for the gate and check pipelines respectively:: |
| 993 | |
| 994 | ./tools/zuul-changes.py --review-host=review.openstack.org \ |
| 995 | http://zuul.openstack.org/ gate 'reverify no bug' |
| 996 | ./tools/zuul-changes.py --review-host=review.openstack.org \ |
| 997 | http://zuul.openstack.org/ check 'recheck no bug' |
Clark Boylan | fba9b24 | 2013-08-20 10:11:17 -0700 | [diff] [blame] | 998 | |
| 999 | If you send a SIGUSR2 to the zuul-server process, Zuul will dump a stack |
| 1000 | trace for each running thread into its debug log. This is useful for |
| 1001 | tracking down deadlock or otherwise slow threads. |