James E. Blair | 1de8d40 | 2017-05-07 17:08:04 -0700 | [diff] [blame] | 1 | :title: Project Configuration |
| 2 | |
| 3 | .. _project-config: |
| 4 | |
| 5 | Project Configuration |
| 6 | ===================== |
| 7 | |
| 8 | The following sections describe the main part of Zuul's configuration. |
| 9 | All of what follows is found within files inside of the repositories |
| 10 | that Zuul manages. |
| 11 | |
| 12 | Security Contexts |
| 13 | ----------------- |
| 14 | |
| 15 | When a system administrator configures Zuul to operate on a project, |
| 16 | they specify one of two security contexts for that project. A |
| 17 | *config-project* is one which is primarily tasked with holding |
| 18 | configuration information and job content for Zuul. Jobs which are |
| 19 | defined in a *config-project* are run with elevated privileges, and |
| 20 | all Zuul configuration items are available for use. It is expected |
| 21 | that changes to *config-projects* will undergo careful scrutiny before |
| 22 | being merged. |
| 23 | |
| 24 | An *untrusted-project* is a project whose primary focus is not to |
| 25 | operate Zuul, but rather it is one of the projects being tested or |
| 26 | deployed. The Zuul configuration language available to these projects |
| 27 | is somewhat restricted (as detailed in individual section below), and |
| 28 | jobs defined in these projects run in a restricted execution |
| 29 | environment since they may be operating on changes which have not yet |
| 30 | undergone review. |
| 31 | |
| 32 | Configuration Loading |
| 33 | --------------------- |
| 34 | |
| 35 | When Zuul starts, it examines all of the git repositories which are |
James E. Blair | eff5a9d | 2017-06-20 00:00:37 -0700 | [diff] [blame] | 36 | specified by the system administrator in :ref:`tenant-config` and searches |
James E. Blair | 1de8d40 | 2017-05-07 17:08:04 -0700 | [diff] [blame] | 37 | for files in the root of each repository. In the case of a |
| 38 | *config-project*, Zuul looks for a file named `zuul.yaml`. In the |
| 39 | case of an *untrusted-project*, Zuul looks first for `zuul.yaml` and |
| 40 | if that is not found, `.zuul.yaml` (with a leading dot). In the case |
| 41 | of an *untrusted-project*, the configuration from every branch is |
| 42 | included, however, in the case of a *config-project*, only the |
| 43 | `master` branch is examined. |
| 44 | |
| 45 | When a change is proposed to one of these files in an |
| 46 | *untrusted-project*, the configuration proposed in the change is |
| 47 | merged into the running configuration so that any changes to Zuul's |
| 48 | configuration are self-testing as part of that change. If there is a |
| 49 | configuration error, no jobs will be run and the error will be |
| 50 | reported by any applicable pipelines. In the case of a change to a |
| 51 | *config-project*, the new configuration is parsed and examined for |
| 52 | errors, but the new configuration is not used in testing the change. |
| 53 | This is because configuration in *config-projects* is able to access |
| 54 | elevated privileges and should always be reviewed before being merged. |
| 55 | |
| 56 | As soon as a change containing a Zuul configuration change merges to |
| 57 | any Zuul-managed repository, the new configuration takes effect |
| 58 | immediately. |
| 59 | |
| 60 | Configuration Items |
| 61 | ------------------- |
| 62 | |
| 63 | The `zuul.yaml` and `.zuul.yaml` configuration files are |
| 64 | YAML-formatted and are structured as a series of items, each of which |
| 65 | is described below. |
| 66 | |
| 67 | .. _pipeline: |
| 68 | |
| 69 | Pipeline |
| 70 | ~~~~~~~~ |
| 71 | |
| 72 | A pipeline describes a workflow operation in Zuul. It associates jobs |
| 73 | for a given project with triggering and reporting events. |
| 74 | |
| 75 | Its flexible configuration allows for characterizing any number of |
| 76 | workflows, and by specifying each as a named configuration, makes it |
| 77 | easy to apply similar workflow operations to projects or groups of |
| 78 | projects. |
| 79 | |
| 80 | By way of example, one of the primary uses of Zuul is to perform |
| 81 | project gating. To do so, one can create a *gate* pipeline which |
| 82 | tells Zuul that when a certain event (such as approval by a code |
| 83 | reviewer) occurs, the corresponding change or pull request should be |
| 84 | enqueued into the pipeline. When that happens, the jobs which have |
| 85 | been configured to run for that project in the *gate* pipeline are |
| 86 | run, and when they complete, the pipeline reports the results to the |
| 87 | user. |
| 88 | |
| 89 | Pipeline configuration items may only appear in *config-projects*. |
| 90 | |
| 91 | Generally, a Zuul administrator would define a small number of |
| 92 | pipelines which represent the workflow processes used in their |
| 93 | environment. Each project can then be added to the available |
| 94 | pipelines as appropriate. |
| 95 | |
| 96 | Here is an example *check* pipeline, which runs whenever a new |
| 97 | patchset is created in Gerrit. If the associated jobs all report |
| 98 | success, the pipeline reports back to Gerrit with a *Verified* vote of |
| 99 | +1, or if at least one of them fails, a -1:: |
| 100 | |
| 101 | - pipeline: |
James E. Blair | a8387ec | 2017-07-05 14:00:12 -0700 | [diff] [blame] | 102 | name: check |
James E. Blair | 1de8d40 | 2017-05-07 17:08:04 -0700 | [diff] [blame] | 103 | manager: independent |
| 104 | trigger: |
| 105 | my_gerrit: |
| 106 | - event: patchset-created |
| 107 | success: |
| 108 | my_gerrit: |
| 109 | verified: 1 |
| 110 | failure: |
| 111 | my_gerrit |
| 112 | verified: -1 |
| 113 | |
James E. Blair | eff5a9d | 2017-06-20 00:00:37 -0700 | [diff] [blame] | 114 | .. TODO: See TODO for more annotated examples of common pipeline configurations. |
James E. Blair | 1de8d40 | 2017-05-07 17:08:04 -0700 | [diff] [blame] | 115 | |
| 116 | The attributes available on a pipeline are as follows (all are |
| 117 | optional unless otherwise specified): |
| 118 | |
| 119 | **name** (required) |
| 120 | This is used later in the project definition to indicate what jobs |
| 121 | should be run for events in the pipeline. |
| 122 | |
| 123 | **manager** (required) |
| 124 | There are currently two schemes for managing pipelines: |
| 125 | |
James E. Blair | eff5a9d | 2017-06-20 00:00:37 -0700 | [diff] [blame] | 126 | .. _independent_pipeline_manager: |
| 127 | |
James E. Blair | 1de8d40 | 2017-05-07 17:08:04 -0700 | [diff] [blame] | 128 | *independent* |
| 129 | Every event in this pipeline should be treated as independent of |
| 130 | other events in the pipeline. This is appropriate when the order of |
| 131 | events in the pipeline doesn't matter because the results of the |
| 132 | actions this pipeline performs can not affect other events in the |
| 133 | pipeline. For example, when a change is first uploaded for review, |
| 134 | you may want to run tests on that change to provide early feedback |
| 135 | to reviewers. At the end of the tests, the change is not going to |
| 136 | be merged, so it is safe to run these tests in parallel without |
| 137 | regard to any other changes in the pipeline. They are independent. |
| 138 | |
| 139 | Another type of pipeline that is independent is a post-merge |
| 140 | pipeline. In that case, the changes have already merged, so the |
| 141 | results can not affect any other events in the pipeline. |
| 142 | |
James E. Blair | eff5a9d | 2017-06-20 00:00:37 -0700 | [diff] [blame] | 143 | .. _dependent_pipeline_manager: |
| 144 | |
James E. Blair | 1de8d40 | 2017-05-07 17:08:04 -0700 | [diff] [blame] | 145 | *dependent* |
| 146 | The dependent pipeline manager is designed for gating. It ensures |
| 147 | that every change is tested exactly as it is going to be merged |
| 148 | into the repository. An ideal gating system would test one change |
| 149 | at a time, applied to the tip of the repository, and only if that |
| 150 | change passed tests would it be merged. Then the next change in |
| 151 | line would be tested the same way. In order to achieve parallel |
| 152 | testing of changes, the dependent pipeline manager performs |
| 153 | speculative execution on changes. It orders changes based on |
| 154 | their entry into the pipeline. It begins testing all changes in |
| 155 | parallel, assuming that each change ahead in the pipeline will pass |
| 156 | its tests. If they all succeed, all the changes can be tested and |
| 157 | merged in parallel. If a change near the front of the pipeline |
| 158 | fails its tests, each change behind it ignores whatever tests have |
| 159 | been completed and are tested again without the change in front. |
| 160 | This way gate tests may run in parallel but still be tested |
| 161 | correctly, exactly as they will appear in the repository when |
| 162 | merged. |
| 163 | |
| 164 | For more detail on the theory and operation of Zuul's dependent |
| 165 | pipeline manager, see: :doc:`gating`. |
| 166 | |
James E. Blair | f17aa9c | 2017-07-05 13:21:23 -0700 | [diff] [blame] | 167 | **allow-secrets** |
| 168 | This is a boolean which can be used to prevent jobs which require |
| 169 | secrets from running in this pipeline. Some pipelines run on |
| 170 | proposed changes and therefore execute code which has not yet been |
| 171 | reviewed. In such a case, allowing a job to use a secret could |
| 172 | result in that secret being exposed. The default is False, meaning |
| 173 | that in order to run jobs with secrets, this must be explicitly |
| 174 | enabled on each Pipeline where that is safe. |
| 175 | |
| 176 | For more information, see :ref:`secret`. |
| 177 | |
James E. Blair | 1de8d40 | 2017-05-07 17:08:04 -0700 | [diff] [blame] | 178 | **description** |
| 179 | This field may be used to provide a textual description of the |
| 180 | pipeline. It may appear in the status page or in documentation. |
| 181 | |
| 182 | **success-message** |
| 183 | The introductory text in reports when all the voting jobs are |
| 184 | successful. Defaults to "Build successful." |
| 185 | |
| 186 | **failure-message** |
| 187 | The introductory text in reports when at least one voting job fails. |
| 188 | Defaults to "Build failed." |
| 189 | |
| 190 | **merge-failure-message** |
| 191 | The introductory text in the message reported when a change fails to |
| 192 | merge with the current state of the repository. Defaults to "Merge |
| 193 | failed." |
| 194 | |
| 195 | **footer-message** |
| 196 | Supplies additional information after test results. Useful for |
| 197 | adding information about the CI system such as debugging and contact |
| 198 | details. |
| 199 | |
| 200 | **trigger** |
| 201 | At least one trigger source must be supplied for each pipeline. |
| 202 | Triggers are not exclusive -- matching events may be placed in |
| 203 | multiple pipelines, and they will behave independently in each of |
| 204 | the pipelines they match. |
| 205 | |
| 206 | Triggers are loaded from their connection name. The driver type of |
| 207 | the connection will dictate which options are available. |
James E. Blair | eff5a9d | 2017-06-20 00:00:37 -0700 | [diff] [blame] | 208 | See :ref:`drivers`. |
James E. Blair | 1de8d40 | 2017-05-07 17:08:04 -0700 | [diff] [blame] | 209 | |
| 210 | **require** |
| 211 | If this section is present, it established pre-requisites for any |
| 212 | kind of item entering the Pipeline. Regardless of how the item is |
| 213 | to be enqueued (via any trigger or automatic dependency resolution), |
| 214 | the conditions specified here must be met or the item will not be |
| 215 | enqueued. |
| 216 | |
James E. Blair | eff5a9d | 2017-06-20 00:00:37 -0700 | [diff] [blame] | 217 | .. _pipeline-require-approval: |
James E. Blair | 1de8d40 | 2017-05-07 17:08:04 -0700 | [diff] [blame] | 218 | |
| 219 | **approval** |
| 220 | This requires that a certain kind of approval be present for the |
| 221 | current patchset of the change (the approval could be added by the |
| 222 | event in question). It takes several sub-parameters, all of which |
| 223 | are optional and are combined together so that there must be an |
| 224 | approval matching all specified requirements. |
| 225 | |
| 226 | *username* |
| 227 | If present, an approval from this username is required. It is |
| 228 | treated as a regular expression. |
| 229 | |
| 230 | *email* |
| 231 | If present, an approval with this email address is required. It |
| 232 | is treated as a regular expression. |
| 233 | |
| 234 | *email-filter* (deprecated) |
| 235 | A deprecated alternate spelling of *email*. Only one of *email* or |
| 236 | *email_filter* should be used. |
| 237 | |
| 238 | *older-than* |
| 239 | If present, the approval must be older than this amount of time |
| 240 | to match. Provide a time interval as a number with a suffix of |
| 241 | "w" (weeks), "d" (days), "h" (hours), "m" (minutes), "s" |
| 242 | (seconds). Example ``48h`` or ``2d``. |
| 243 | |
| 244 | *newer-than* |
| 245 | If present, the approval must be newer than this amount of time |
| 246 | to match. Same format as "older-than". |
| 247 | |
| 248 | Any other field is interpreted as a review category and value |
| 249 | pair. For example ``verified: 1`` would require that the approval |
| 250 | be for a +1 vote in the "Verified" column. The value may either |
| 251 | be a single value or a list: ``verified: [1, 2]`` would match |
| 252 | either a +1 or +2 vote. |
| 253 | |
| 254 | **open** |
| 255 | A boolean value (``true`` or ``false``) that indicates whether the change |
| 256 | must be open or closed in order to be enqueued. |
| 257 | |
| 258 | **current-patchset** |
| 259 | A boolean value (``true`` or ``false``) that indicates whether the change |
| 260 | must be the current patchset in order to be enqueued. |
| 261 | |
| 262 | **status** |
| 263 | A string value that corresponds with the status of the change |
| 264 | reported by the trigger. |
| 265 | |
| 266 | **reject** |
| 267 | If this section is present, it establishes pre-requisites that can |
| 268 | block an item from being enqueued. It can be considered a negative |
| 269 | version of **require**. |
| 270 | |
| 271 | **approval** |
| 272 | This takes a list of approvals. If an approval matches the provided |
| 273 | criteria the change can not be entered into the pipeline. It follows |
| 274 | the same syntax as the :ref:`"require approval" pipeline above |
| 275 | <pipeline-require-approval>`. |
| 276 | |
| 277 | Example to reject a change with any negative vote:: |
| 278 | |
| 279 | reject: |
| 280 | approval: |
| 281 | - code-review: [-1, -2] |
| 282 | |
| 283 | **dequeue-on-new-patchset** |
| 284 | Normally, if a new patchset is uploaded to a change that is in a |
| 285 | pipeline, the existing entry in the pipeline will be removed (with |
| 286 | jobs canceled and any dependent changes that can no longer merge as |
| 287 | well. To suppress this behavior (and allow jobs to continue |
| 288 | running), set this to ``false``. Default: ``true``. |
| 289 | |
| 290 | **ignore-dependencies** |
| 291 | In any kind of pipeline (dependent or independent), Zuul will |
| 292 | attempt to enqueue all dependencies ahead of the current change so |
| 293 | that they are tested together (independent pipelines report the |
| 294 | results of each change regardless of the results of changes ahead). |
| 295 | To ignore dependencies completely in an independent pipeline, set |
| 296 | this to ``true``. This option is ignored by dependent pipelines. |
| 297 | The default is: ``false``. |
| 298 | |
James E. Blair | eff5a9d | 2017-06-20 00:00:37 -0700 | [diff] [blame] | 299 | **precedence** |
| 300 | Indicates how the build scheduler should prioritize jobs for |
| 301 | different pipelines. Each pipeline may have one precedence, jobs |
| 302 | for pipelines with a higher precedence will be run before ones with |
| 303 | lower. The value should be one of ``high``, ``normal``, or ``low``. |
| 304 | Default: ``normal``. |
| 305 | |
James E. Blair | 1de8d40 | 2017-05-07 17:08:04 -0700 | [diff] [blame] | 306 | The following options configure *reporters*. Reporters are |
| 307 | complementary to triggers; where a trigger is an event on a connection |
| 308 | which causes Zuul to enqueue an item, a reporter is the action |
| 309 | performed on a connection when an item is dequeued after its jobs |
| 310 | complete. The actual syntax for a reporter is defined by the driver |
| 311 | which implements it. See :ref:`drivers` for more information. |
| 312 | |
| 313 | **success** |
| 314 | Describes where Zuul should report to if all the jobs complete |
| 315 | successfully. This section is optional; if it is omitted, Zuul will |
| 316 | run jobs and do nothing on success -- it will not report at all. If |
| 317 | the section is present, the listed reporters will be asked to report |
| 318 | on the jobs. The reporters are listed by their connection name. The |
| 319 | options available depend on the driver for the supplied connection. |
| 320 | |
| 321 | **failure** |
| 322 | These reporters describe what Zuul should do if at least one job |
| 323 | fails. |
| 324 | |
| 325 | **merge-failure** |
| 326 | These reporters describe what Zuul should do if it is unable to |
| 327 | merge in the patchset. If no merge-failure reporters are listed then |
| 328 | the ``failure`` reporters will be used to notify of unsuccessful |
| 329 | merges. |
| 330 | |
| 331 | **start** |
| 332 | These reporters describe what Zuul should do when a change is added |
| 333 | to the pipeline. This can be used, for example, to reset a |
| 334 | previously reported result. |
| 335 | |
| 336 | **disabled** |
| 337 | These reporters describe what Zuul should do when a pipeline is |
| 338 | disabled. See ``disable-after-consecutive-failures``. |
| 339 | |
James E. Blair | eff5a9d | 2017-06-20 00:00:37 -0700 | [diff] [blame] | 340 | The following options can be used to alter Zuul's behavior to mitigate |
| 341 | situations in which jobs are failing frequently (perhaps due to a |
| 342 | problem with an external dependency, or unusually high |
| 343 | non-deterministic test failures). |
| 344 | |
James E. Blair | 1de8d40 | 2017-05-07 17:08:04 -0700 | [diff] [blame] | 345 | **disable-after-consecutive-failures** |
| 346 | If set, a pipeline can enter a ''disabled'' state if too many changes |
| 347 | in a row fail. When this value is exceeded the pipeline will stop |
| 348 | reporting to any of the ``success``, ``failure`` or ``merge-failure`` |
| 349 | reporters and instead only report to the ``disabled`` reporters. |
| 350 | (No ``start`` reports are made when a pipeline is disabled). |
| 351 | |
James E. Blair | 1de8d40 | 2017-05-07 17:08:04 -0700 | [diff] [blame] | 352 | **window** |
| 353 | Dependent pipeline managers only. Zuul can rate limit dependent |
| 354 | pipelines in a manner similar to TCP flow control. Jobs are only |
| 355 | started for items in the queue if they are within the actionable |
| 356 | window for the pipeline. The initial length of this window is |
| 357 | configurable with this value. The value given should be a positive |
| 358 | integer value. A value of ``0`` disables rate limiting on the |
| 359 | DependentPipelineManager. Default: ``20``. |
| 360 | |
| 361 | **window-floor** |
| 362 | Dependent pipeline managers only. This is the minimum value for the |
| 363 | window described above. Should be a positive non zero integer value. |
| 364 | Default: ``3``. |
| 365 | |
| 366 | **window-increase-type** |
| 367 | Dependent pipeline managers only. This value describes how the window |
| 368 | should grow when changes are successfully merged by zuul. A value of |
| 369 | ``linear`` indicates that ``window-increase-factor`` should be added |
| 370 | to the previous window value. A value of ``exponential`` indicates |
| 371 | that ``window-increase-factor`` should be multiplied against the |
| 372 | previous window value and the result will become the window size. |
| 373 | Default: ``linear``. |
| 374 | |
| 375 | **window-increase-factor** |
| 376 | Dependent pipeline managers only. The value to be added or multiplied |
| 377 | against the previous window value to determine the new window after |
| 378 | successful change merges. |
| 379 | Default: ``1``. |
| 380 | |
| 381 | **window-decrease-type** |
| 382 | Dependent pipeline managers only. This value describes how the window |
| 383 | should shrink when changes are not able to be merged by Zuul. A value |
| 384 | of ``linear`` indicates that ``window-decrease-factor`` should be |
| 385 | subtracted from the previous window value. A value of ``exponential`` |
| 386 | indicates that ``window-decrease-factor`` should be divided against |
| 387 | the previous window value and the result will become the window size. |
| 388 | Default: ``exponential``. |
| 389 | |
| 390 | **window-decrease-factor** |
| 391 | Dependent pipline managers only. The value to be subtracted or divided |
| 392 | against the previous window value to determine the new window after |
| 393 | unsuccessful change merges. |
| 394 | Default: ``2``. |
| 395 | |
| 396 | |
| 397 | .. _job: |
| 398 | |
| 399 | Job |
| 400 | ~~~ |
| 401 | |
| 402 | A job is a unit of work performed by Zuul on an item enqueued into a |
| 403 | pipeline. Items may run any number of jobs (which may depend on each |
| 404 | other). Each job is an invocation of an Ansible playbook with a |
| 405 | specific inventory of hosts. The actual tasks that are run by the job |
| 406 | appear in the playbook for that job while the attributes that appear in the |
| 407 | Zuul configuration specify information about when, where, and how the |
| 408 | job should be run. |
| 409 | |
| 410 | Jobs in Zuul support inheritance. Any job may specify a single parent |
| 411 | job, and any attributes not set on the child job are collected from |
| 412 | the parent job. In this way, a configuration structure may be built |
| 413 | starting with very basic jobs which describe characteristics that all |
| 414 | jobs on the system should have, progressing through stages of |
| 415 | specialization before arriving at a particular job. A job may inherit |
| 416 | from any other job in any project (however, if the other job is marked |
| 417 | as `final`, some attributes may not be overidden). |
| 418 | |
| 419 | Jobs also support a concept called variance. The first time a job |
| 420 | definition appears is called the reference definition of the job. |
| 421 | Subsequent job definitions with the same name are called variants. |
| 422 | These may have different selection criteria which indicate to Zuul |
| 423 | that, for instance, the job should behave differently on a different |
| 424 | git branch. Unlike inheritance, all job variants must be defined in |
| 425 | the same project. |
| 426 | |
| 427 | When Zuul decides to run a job, it performs a process known as |
| 428 | freezing the job. Because any number of job variants may be |
| 429 | applicable, Zuul collects all of the matching variants and applies |
| 430 | them in the order they appeared in the configuration. The resulting |
| 431 | frozen job is built from attributes gathered from all of the |
| 432 | matching variants. In this way, exactly what is run is dependent on |
| 433 | the pipeline, project, branch, and content of the item. |
| 434 | |
| 435 | In addition to the job's main playbook, each job may specify one or |
| 436 | more pre- and post-playbooks. These are run, in order, before and |
| 437 | after (respectively) the main playbook. They may be used to set up |
| 438 | and tear down resources needed by the main playbook. When combined |
| 439 | with inheritance, they provide powerful tools for job construction. A |
| 440 | job only has a single main playbook, and when inheriting from a |
| 441 | parent, the child's main playbook overrides (or replaces) the |
| 442 | parent's. However, the pre- and post-playbooks are appended and |
| 443 | prepended in a nesting fashion. So if a parent job and child job both |
| 444 | specified pre and post playbooks, the sequence of playbooks run would |
| 445 | be: |
| 446 | |
| 447 | * parent pre-run playbook |
| 448 | * child pre-run playbook |
| 449 | * child playbook |
| 450 | * child post-run playbook |
| 451 | * parent post-run playbook |
| 452 | |
| 453 | Further inheritance would nest even deeper. |
| 454 | |
| 455 | Here is an example of two job definitions:: |
| 456 | |
| 457 | - job: |
| 458 | name: base |
| 459 | pre-run: copy-git-repos |
| 460 | post-run: copy-logs |
| 461 | |
| 462 | - job: |
| 463 | name: run-tests |
| 464 | parent: base |
| 465 | nodes: |
| 466 | - name: test-node |
| 467 | image: fedora |
| 468 | |
| 469 | The following attributes are available on a job; all are optional |
| 470 | unless otherwise specified: |
| 471 | |
| 472 | **name** (required) |
| 473 | The name of the job. By default, Zuul looks for a playbook with |
| 474 | this name to use as the main playbook for the job. This name is |
| 475 | also referenced later in a project pipeline configuration. |
| 476 | |
| 477 | **parent** |
| 478 | Specifies a job to inherit from. The parent job can be defined in |
| 479 | this or any other project. Any attributes not specified on a job |
| 480 | will be collected from its parent. |
| 481 | |
| 482 | **description** |
| 483 | A textual description of the job. Not currently used directly by |
| 484 | Zuul, but it is used by the zuul-sphinx extension to Sphinx to |
| 485 | auto-document Zuul jobs (in which case it is interpreted as |
| 486 | ReStructuredText. |
| 487 | |
| 488 | **success-message** |
| 489 | Normally when a job succeeds, the string "SUCCESS" is reported as |
| 490 | the result for the job. If set, this option may be used to supply a |
| 491 | different string. Default: "SUCCESS". |
| 492 | |
| 493 | **failure-message** |
| 494 | Normally when a job fails, the string "FAILURE" is reported as |
| 495 | the result for the job. If set, this option may be used to supply a |
| 496 | different string. Default: "FAILURE". |
| 497 | |
| 498 | **success-url** |
| 499 | When a job succeeds, this URL is reported along with the result. |
| 500 | Default: none. |
| 501 | |
| 502 | **failure-url** |
| 503 | When a job fails, this URL is reported along with the result. |
| 504 | Default: none. |
| 505 | |
| 506 | **hold-following-changes** |
| 507 | In a dependent pipeline, this option may be used to indicate that no |
| 508 | jobs should start on any items which depend on the current item |
| 509 | until this job has completed successfully. This may be used to |
| 510 | conserve build resources, at the expense of inhibiting the |
| 511 | parallelization which speeds the processing of items in a dependent |
| 512 | pipeline. A boolean value, default: false. |
| 513 | |
| 514 | **voting** |
| 515 | Indicates whether the result of this job should be used in |
| 516 | determining the overall result of the item. A boolean value, |
| 517 | default: true. |
| 518 | |
| 519 | **semaphore** |
| 520 | The name of a :ref:`semaphore` which should be acquired and released |
| 521 | when the job begins and ends. If the semaphore is at maximum |
| 522 | capacity, then Zuul will wait until it can be acquired before |
| 523 | starting the job. Default: none. |
| 524 | |
| 525 | **tags** |
| 526 | Metadata about this job. Tags are units of information attached to |
| 527 | the job; they do not affect Zuul's behavior, but they can be used |
| 528 | within the job to characterize the job. For example, a job which |
| 529 | tests a certain subsystem could be tagged with the name of that |
| 530 | subsystem, and if the job's results are reported into a database, |
| 531 | then the results of all jobs affecting that subsystem could be |
| 532 | queried. This attribute is specified as a list of strings, and when |
| 533 | inheriting jobs or applying variants, tags accumulate in a set, so |
| 534 | the result is always a set of all the tags from all the jobs and |
| 535 | variants used in constructing the frozen job, with no duplication. |
| 536 | Default: none. |
| 537 | |
James E. Blair | eff5a9d | 2017-06-20 00:00:37 -0700 | [diff] [blame] | 538 | **branches** |
James E. Blair | 1de8d40 | 2017-05-07 17:08:04 -0700 | [diff] [blame] | 539 | A regular expression (or list of regular expressions) which describe |
| 540 | on what branches a job should run (or in the case of variants: to |
| 541 | alter the behavior of a job for a certain branch). |
| 542 | |
| 543 | If there is no job definition for a given job which matches the |
| 544 | branch of an item, then that job is not run for the item. |
| 545 | Otherwise, all of the job variants which match that branch (and any |
| 546 | other selection criteria) are used when freezing the job. |
| 547 | |
| 548 | This example illustrates a job called *run-tests* which uses a |
| 549 | nodeset based on the current release of an operating system to |
| 550 | perform its tests, except when testing changes to the stable/2.0 |
| 551 | branch, in which case it uses an older release:: |
| 552 | |
| 553 | - job: |
| 554 | name: run-tests |
| 555 | nodes: current-release |
| 556 | |
| 557 | - job: |
| 558 | name: run-tests |
| 559 | branch: stable/2.0 |
| 560 | nodes: old-release |
| 561 | |
| 562 | In some cases, Zuul uses an implied value for the branch specifier |
| 563 | if none is supplied: |
| 564 | |
| 565 | * For a job definition in a *config-project*, no implied branch |
| 566 | specifier is used. If no branch specifier appears, the job |
| 567 | applies to all branches. |
| 568 | |
| 569 | * In the case of an *untrusted-project*, no implied branch specifier |
| 570 | is applied to the reference definition of a job. That is to say, |
| 571 | that if the first appearance of the job definition appears without |
| 572 | a branch specifier, then it will apply to all branches. Note that |
| 573 | when collecting its configuration, Zuul reads the `master` branch |
| 574 | of a given project first, then other branches in alphabetical |
| 575 | order. |
| 576 | |
| 577 | * Any further job variants other than the reference definition in an |
| 578 | *untrusted-project* will, if they do not have a branch specifier, |
| 579 | will have an implied branch specifier for the current branch |
| 580 | applied. |
| 581 | |
| 582 | This allows for the very simple and expected workflow where if a |
| 583 | project defines a job on the master branch with no branch specifier, |
| 584 | and then creates a new branch based on master, any changes to that |
| 585 | job definition within the new branch only affect that branch. |
| 586 | |
| 587 | **files** |
| 588 | This attribute indicates that the job should only run on changes |
| 589 | where the specified files are modified. This is a regular |
| 590 | expression or list of regular expressions. Default: none. |
| 591 | |
| 592 | **irrelevant-files** |
| 593 | This is a negative complement of `files`. It indicates that the job |
| 594 | should run unless *all* of the files changed match this list. In |
| 595 | other words, if the regular expression `docs/.*` is supplied, then |
| 596 | this job will not run if the only files changed are in the docs |
| 597 | directory. A regular expression or list of regular expressions. |
| 598 | Default: none. |
| 599 | |
| 600 | **auth** |
| 601 | Authentication information to be made available to the job. This is |
| 602 | a dictionary with two potential keys: |
| 603 | |
| 604 | **inherit** |
| 605 | A boolean indicating that the authentication information referenced |
| 606 | by this job should be able to be inherited by child jobs. Normally |
| 607 | when a job inherits from another job, the auth section is not |
| 608 | included. This permits jobs to inherit the same basic structure and |
| 609 | playbook, but ensures that secret information is unable to be |
| 610 | exposed by a child job which may alter the job's behavior. If it is |
| 611 | safe for the contents of the authentication section to be used by |
| 612 | child jobs, set this to ``true``. Default: ``false``. |
| 613 | |
| 614 | **secrets** |
| 615 | A list of secrets which may be used by the job. A :ref:`secret` is |
| 616 | a named collection of private information defined separately in the |
| 617 | configuration. The secrets that appear here must be defined in the |
| 618 | same project as this job definition. |
| 619 | |
| 620 | In the future, other types of authentication information may be |
| 621 | added. |
| 622 | |
| 623 | **nodes** |
| 624 | A list of nodes which should be supplied to the job. This parameter |
| 625 | may be supplied either as a string, in which case it references a |
| 626 | :ref:`nodeset` definition which appears elsewhere in the |
| 627 | configuration, or a list, in which case it is interpreted in the |
| 628 | same way as a Nodeset definition (in essence, it is an anonymous |
| 629 | Node definition unique to this job). See the :ref:`nodeset` |
| 630 | reference for the syntax to use in that case. |
| 631 | |
| 632 | If a job has an empty or no node definition, it will still run and |
| 633 | may be able to perform actions on the Zuul executor. |
| 634 | |
| 635 | **override-branch** |
| 636 | When Zuul runs jobs for a proposed change, it normally checks out |
| 637 | the branch associated with that change on every project present in |
| 638 | the job. If jobs are running on a ref (such as a branch tip or |
| 639 | tag), then that ref is normally checked out. This attribute is used |
| 640 | to override that behavior and indicate that this job should, |
| 641 | regardless of the branch for the queue item, use the indicated |
| 642 | branch instead. This can be used, for example, to run a previous |
| 643 | version of the software (from a stable maintenance branch) under |
| 644 | test even if the change being tested applies to a different branch |
| 645 | (this is only likely to be useful if there is some cross-branch |
| 646 | interaction with some component of the system being tested). See |
| 647 | also the project-specific **override-branch** attribute under |
| 648 | **required-projects** to apply this behavior to a subset of a job's |
| 649 | projects. |
| 650 | |
| 651 | **timeout** |
| 652 | The time in minutes that the job should be allowed to run before it |
| 653 | is automatically aborted and failure is reported. If no timeout is |
| 654 | supplied, the job may run indefinitely. Supplying a timeout is |
| 655 | highly recommended. |
| 656 | |
| 657 | **attempts** |
| 658 | When Zuul encounters an error running a job's pre-run playbook, Zuul |
| 659 | will stop and restart the job. Errors during the main or |
| 660 | post-run -playbook phase of a job are not affected by this parameter |
| 661 | (they are reported immediately). This parameter controls the number |
| 662 | of attempts to make before an error is reported. Default: 3. |
| 663 | |
| 664 | **pre-run** |
Tobias Henkel | 2aade26 | 2017-07-12 16:09:06 +0200 | [diff] [blame] | 665 | The name of a playbook or list of playbooks without file extension |
| 666 | to run before the main body of a job. The full path to the playbook |
| 667 | in the repo where the job is defined is expected. |
James E. Blair | 1de8d40 | 2017-05-07 17:08:04 -0700 | [diff] [blame] | 668 | |
| 669 | When a job inherits from a parent, the child's pre-run playbooks are |
| 670 | run after the parent's. See :ref:`job` for more information. |
| 671 | |
| 672 | **post-run** |
Tobias Henkel | 2aade26 | 2017-07-12 16:09:06 +0200 | [diff] [blame] | 673 | The name of a playbook or list of playbooks without file extension |
| 674 | to run after the main body of a job. The full path to the playbook |
| 675 | in the repo where the job is defined is expected. |
James E. Blair | 1de8d40 | 2017-05-07 17:08:04 -0700 | [diff] [blame] | 676 | |
| 677 | When a job inherits from a parent, the child's post-run playbooks |
| 678 | are run before the parent's. See :ref:`job` for more information. |
| 679 | |
| 680 | **run** |
Tobias Henkel | 2aade26 | 2017-07-12 16:09:06 +0200 | [diff] [blame] | 681 | The name of the main playbook for this job. This parameter is |
| 682 | not normally necessary, as it defaults to a playbook with the |
| 683 | same name as the job inside of the `playbooks/` directory (e.g., |
| 684 | the `foo` job would default to `playbooks/foo`. However, if a |
| 685 | playbook with a different name is needed, it can be specified |
| 686 | here. The file extension is not required, but the full path |
| 687 | within the repo is. When a child inherits from a parent, a |
| 688 | playbook with the name of the child job is implicitly searched |
| 689 | first, before falling back on the playbook used by the parent |
| 690 | job (unless the child job specifies a ``run`` attribute, in which |
| 691 | case that value is used). Example:: |
| 692 | |
| 693 | run: playbooks/<name of the job> |
James E. Blair | 1de8d40 | 2017-05-07 17:08:04 -0700 | [diff] [blame] | 694 | |
| 695 | **roles** |
| 696 | A list of Ansible roles to prepare for the job. Because a job runs |
| 697 | an Ansible playbook, any roles which are used by the job must be |
| 698 | prepared and installed by Zuul before the job begins. This value is |
| 699 | a list of dictionaries, each of which indicates one of two types of |
| 700 | roles: a Galaxy role, which is simply a role that is installed from |
| 701 | Ansible Galaxy, or a Zuul role, which is a role provided by a |
| 702 | project managed by Zuul. Zuul roles are able to benefit from |
| 703 | speculative merging and cross-project dependencies when used by jobs |
| 704 | in untrusted projects. |
| 705 | |
| 706 | A project which supplies a role may be structured in one of two |
| 707 | configurations: a bare role (in which the role exists at the root of |
| 708 | the project), or a contained role (in which the role exists within |
| 709 | the `roles/` directory of the project, perhaps along with other |
| 710 | roles). In the case of a contained role, the `roles/` directory of |
| 711 | the project is added to the role search path. In the case of a bare |
| 712 | role, the project itself is added to the role search path. In case |
| 713 | the name of the project is not the name under which the role should |
| 714 | be installed (and therefore referenced from Ansible), the `name` |
| 715 | attribute may be used to specify an alternate. |
| 716 | |
James E. Blair | eff5a9d | 2017-06-20 00:00:37 -0700 | [diff] [blame] | 717 | .. note:: galaxy roles are not yet implemented |
| 718 | |
James E. Blair | 1de8d40 | 2017-05-07 17:08:04 -0700 | [diff] [blame] | 719 | **galaxy** |
| 720 | The name of the role in Ansible Galaxy. If this attribute is |
| 721 | supplied, Zuul will search Ansible Galaxy for a role by this name |
| 722 | and install it. Mutually exclusive with ``zuul``; either |
| 723 | ``galaxy`` or ``zuul`` must be supplied. |
| 724 | |
| 725 | **zuul** |
| 726 | The name of a Zuul project which supplies the role. Mutually |
| 727 | exclusive with ``galaxy``; either ``galaxy`` or ``zuul`` must be |
| 728 | supplied. |
| 729 | |
| 730 | **name** |
| 731 | The installation name of the role. In the case of a bare role, |
| 732 | the role will be made available under this name. Ignored in the |
| 733 | case of a contained role. |
| 734 | |
| 735 | **required-projects** |
| 736 | A list of other projects which are used by this job. Any Zuul |
| 737 | projects specified here will also be checked out by Zuul into the |
| 738 | working directory for the job. Speculative merging and cross-repo |
| 739 | dependencies will be honored. |
| 740 | |
| 741 | The format for this attribute is either a list of strings or |
| 742 | dictionaries. Strings are interpreted as project names, |
| 743 | dictionaries may have the following attributes: |
| 744 | |
| 745 | **name** |
| 746 | The name of the required project. |
| 747 | |
| 748 | **override-branch** |
| 749 | When Zuul runs jobs for a proposed change, it normally checks out |
| 750 | the branch associated with that change on every project present in |
| 751 | the job. If jobs are running on a ref (such as a branch tip or |
| 752 | tag), then that ref is normally checked out. This attribute is |
| 753 | used to override that behavior and indicate that this job should, |
| 754 | regardless of the branch for the queue item, use the indicated |
| 755 | branch instead, for only this project. See also the |
| 756 | **override-branch** attribute of jobs to apply the same behavior |
| 757 | to all projects in a job. |
| 758 | |
| 759 | **vars** |
| 760 | |
| 761 | A dictionary of variables to supply to Ansible. When inheriting from |
| 762 | a job (or creating a variant of a job) vars are merged with previous |
| 763 | definitions. This means a variable definition with the same name will |
| 764 | override a previously defined variable, but new variable names will be |
| 765 | added to the set of defined variables. |
| 766 | |
| 767 | **dependencies** |
| 768 | A list of other jobs upon which this job depends. Zuul will not |
| 769 | start executing this job until all of its dependencies have |
| 770 | completed successfully, and if one or more of them fail, this job |
| 771 | will not be run. |
| 772 | |
| 773 | **allowed-projects** |
| 774 | A list of Zuul projects which may use this job. By default, a job |
| 775 | may be used by any other project known to Zuul, however, some jobs |
| 776 | use resources or perform actions which are not appropriate for other |
| 777 | projects. In these cases, a list of projects which are allowed to |
| 778 | use this job may be supplied. If this list is not empty, then it |
| 779 | must be an exhaustive list of all projects permitted to use the job. |
| 780 | The current project (where the job is defined) is not automatically |
| 781 | included, so if it should be able to run this job, then it must be |
| 782 | explicitly listed. Default: the empty list (all projects may use |
| 783 | the job). |
| 784 | |
| 785 | |
| 786 | .. _project: |
| 787 | |
| 788 | Project |
| 789 | ~~~~~~~ |
| 790 | |
| 791 | A project corresponds to a source code repository with which Zuul is |
| 792 | configured to interact. The main responsibility of the `Project` |
| 793 | configuration item is to specify which jobs should run in which |
| 794 | pipelines for a given project. Within each `Project` definition, a |
| 795 | section for each `Pipeline` may appear. This project-pipeline |
| 796 | definition is what determines how a project participates in a |
| 797 | pipeline. |
| 798 | |
| 799 | Consider the following `Project` definition:: |
| 800 | |
| 801 | - project: |
| 802 | name: yoyodyne |
| 803 | check: |
| 804 | jobs: |
| 805 | - check-syntax |
| 806 | - unit-tests |
| 807 | gate: |
| 808 | queue: integrated |
| 809 | jobs: |
| 810 | - unit-tests |
| 811 | - integration-tests |
| 812 | |
| 813 | The project has two project-pipeline stanzas, one for the `check` |
| 814 | pipeline, and one for `gate`. Each specifies which jobs shuld run |
| 815 | when a change for that project enteres the respective pipeline -- when |
| 816 | a change enters `check`, the `check-syntax` and `unit-test` jobs are |
| 817 | run. |
| 818 | |
| 819 | Pipelines which use the dependent pipeline manager (e.g., the `gate` |
| 820 | example shown earlier) maintain separate queues for groups of |
| 821 | projects. When Zuul serializes a set of changes which represent |
| 822 | future potential project states, it must know about all of the |
| 823 | projects within Zuul which may have an effect on the outcome of the |
| 824 | jobs it runs. If project *A* uses project *B* as a library, then Zuul |
| 825 | must be told about that relationship so that it knows to serialize |
| 826 | changes to A and B together, so that it does not merge a change to B |
| 827 | while it is testing a change to A. |
| 828 | |
| 829 | Zuul could simply assume that all projects are related, or even infer |
| 830 | relationships by which projects a job indicates it uses, however, in a |
| 831 | large system that would become unwieldy very quickly, and |
| 832 | unnecessarily delay changes to unrelated projects. To allow for |
| 833 | flexibility in the construction of groups of related projects, the |
| 834 | change queues used by dependent pipeline managers are specified |
| 835 | manually. To group two or more related projects into a shared queue |
| 836 | for a dependent pipeline, set the ``queue`` parameter to the same |
| 837 | value for those projects. |
| 838 | |
| 839 | The `gate` project-pipeline definition above specifies that this |
| 840 | project participates in the `integrated` shared queue for that |
| 841 | pipeline. |
| 842 | |
| 843 | In addition to a project-pipeline definition for one or more |
| 844 | `Pipelines`, the following attributes may appear in a Project: |
| 845 | |
| 846 | **name** (required) |
| 847 | The name of the project. If Zuul is configured with two or more |
| 848 | unique projects with the same name, the canonical hostname for the |
| 849 | project should be included (e.g., `git.example.com/foo`). |
| 850 | |
| 851 | **templates** |
| 852 | A list of :ref:`project-template` references; the project-pipeline |
| 853 | definitions of each Project Template will be applied to this |
| 854 | project. If more than one template includes jobs for a given |
| 855 | pipeline, they will be combined, as will any jobs specified in |
| 856 | project-pipeline definitions on the project itself. |
| 857 | |
| 858 | .. _project-template: |
| 859 | |
| 860 | Project Template |
| 861 | ~~~~~~~~~~~~~~~~ |
| 862 | |
| 863 | A Project Template defines one or more project-pipeline definitions |
| 864 | which can be re-used by multiple projects. |
| 865 | |
| 866 | A Project Template uses the same syntax as a :ref:`project` |
| 867 | definition, however, in the case of a template, the ``name`` attribute |
| 868 | does not refer to the name of a project, but rather names the template |
| 869 | so that it can be referenced in a `Project` definition. |
| 870 | |
| 871 | .. _secret: |
| 872 | |
| 873 | Secret |
| 874 | ~~~~~~ |
| 875 | |
| 876 | A Secret is a collection of private data for use by one or more jobs. |
| 877 | In order to maintain the security of the data, the values are usually |
| 878 | encrypted, however, data which are not sensitive may be provided |
| 879 | unencrypted as well for convenience. |
| 880 | |
| 881 | A Secret may only be used by jobs defined within the same project. To |
| 882 | use a secret, a :ref:`job` must specify the secret within its `auth` |
| 883 | section. To protect against jobs in other repositories declaring a |
| 884 | job with a secret as a parent and then exposing that secret, jobs |
| 885 | which inherit from a job with secrets will not inherit the secrets |
| 886 | themselves. To alter that behavior, see the `inherit` job attribute. |
| 887 | Further, jobs which do not permit children to inherit secrets (the |
| 888 | default) are also automatically marked `final`, meaning that their |
| 889 | execution related attributes may not be changed in a project-pipeline |
| 890 | stanza. This is to protect against a job with secrets defined in one |
| 891 | project being used by another project in a way which might expose the |
| 892 | secrets. If a job with secrets is unsafe to be used by other |
| 893 | projects, the `allowed-projects` job attribute can be used to restrict |
| 894 | the projects which can invoke that job. Finally, pipelines which are |
| 895 | used to execute proposed but unreviewed changes can set the |
| 896 | `allow-secrets` attribute to indicate that they should not supply |
| 897 | secrets at all in order to protect against someone proposing a change |
| 898 | which exposes a secret. |
| 899 | |
| 900 | The following attributes are required: |
| 901 | |
| 902 | **name** (required) |
| 903 | The name of the secret, used in a :ref:`Job` definition to request |
| 904 | the secret. |
| 905 | |
| 906 | **data** (required) |
| 907 | A dictionary which will be added to the Ansible variables available |
| 908 | to the job. The values can either be plain text strings, or |
| 909 | encrypted values. See :ref:`encryption` for more information. |
| 910 | |
| 911 | .. _nodeset: |
| 912 | |
| 913 | Nodeset |
| 914 | ~~~~~~~ |
| 915 | |
| 916 | A Nodeset is a named collection of nodes for use by a job. Jobs may |
| 917 | specify what nodes they require individually, however, by defining |
| 918 | groups of node types once and referring to them by name, job |
| 919 | configuration may be simplified. |
| 920 | |
| 921 | A Nodeset requires two attributes: |
| 922 | |
| 923 | **name** (required) |
| 924 | The name of the Nodeset, to be referenced by a :ref:`job`. |
| 925 | |
| 926 | **nodes** (required) |
| 927 | A list of node definitions, each of which has the following format: |
| 928 | |
| 929 | **name** (required) |
| 930 | The name of the node. This will appear in the Ansible inventory |
| 931 | for the job. |
| 932 | |
| 933 | **label** (required) |
| 934 | The Nodepool label for the node. Zuul will request a node with |
| 935 | this label. |
| 936 | |
| 937 | .. _semaphore: |
| 938 | |
| 939 | Semaphore |
| 940 | ~~~~~~~~~ |
| 941 | |
| 942 | Semaphores can be used to restrict the number of certain jobs which |
| 943 | are running at the same time. This may be useful for jobs which |
| 944 | access shared or limited resources. A semaphore has a value which |
| 945 | represents the maximum number of jobs which use that semaphore at the |
| 946 | same time. |
| 947 | |
| 948 | Semaphores are never subject to dynamic reconfiguration. If the value |
| 949 | of a semaphore is changed, it will take effect only when the change |
| 950 | where it is updated is merged. An example follows:: |
| 951 | |
| 952 | - semaphore: |
| 953 | name: semaphore-foo |
| 954 | max: 5 |
| 955 | - semaphore: |
| 956 | name: semaphore-bar |
| 957 | max: 3 |
| 958 | |
| 959 | The following attributes are available: |
| 960 | |
| 961 | **name** (required) |
| 962 | The name of the semaphore, referenced by jobs. |
| 963 | |
| 964 | **max** |
| 965 | The maximum number of running jobs which can use this semaphore. |
| 966 | Defaults to 1. |