blob: 7f1c3cb56a1c60863ba4f92a226ad6b8df622202 [file] [log] [blame]
James E. Blaireff5a9d2017-06-20 00:00:37 -07001:title: Job Content
2
3Job Content
4===========
5
David Shrewsburyc50cb572017-08-04 11:55:01 -04006Zuul jobs are implemented as Ansible playbooks. Zuul prepares the
James E. Blaireff5a9d2017-06-20 00:00:37 -07007repositories used for a job, installs any required Ansible roles, and
8then executes the job's playbooks. Any setup or artifact collection
9required is the responsibility of the job itself. While this flexible
10arrangement allows for almost any kind of job to be run by Zuul,
11batteries are included. Zuul has a standard library of jobs upon
12which to build.
13
14Working Directory
15-----------------
16
17Before starting each job, the Zuul executor creates a directory to
18hold all of the content related to the job. This includes some
19directories which are used by Zuul to configure and run Ansible and
20may not be accessible, as well as a directory tree, under ``work/``,
21that is readable and writable by the job. The hierarchy is:
22
23**work/**
24 The working directory of the job.
25
26**work/src/**
27 Contains the prepared git repositories for the job.
28
29**work/logs/**
30 Where the Ansible log for the job is written; your job
31 may place other logs here as well.
32
33Git Repositories
34----------------
35
36The git repositories in ``work/src`` contain the repositories for all
37of the projects specified in the ``required-projects`` section of the
38job, plus the project associated with the queue item if it isn't
39already in that list. In the case of a proposed change, that change
40and all of the changes ahead of it in the pipeline queue will already
41be merged into their respective repositories and target branches. The
42change's project will have the change's branch checked out, as will
43all of the other projects, if that branch exists (otherwise, a
44fallback or default branch will be used). If your job needs to
45operate on multiple branches, simply checkout the appropriate branches
46of these git repos to ensure that the job results reflect the proposed
47future state that Zuul is testing, and all dependencies are present.
48Do not use any git remotes; the local repositories are guaranteed to
49be up to date.
50
James E. Blair4d5dd252017-06-23 21:40:56 +010051The repositories will be placed on the filesystem in directories
52corresponding with the canonical hostname of their source connection.
53For example::
54
55 work/src/git.example.com/project1
56 work/src/github.com/project2
57
58Is the layout that would be present for a job which included project1
59from the connection associated to git.example.com and project2 from
60GitHub. This helps avoid collisions between projects with the same
61name, and some language environments, such as Go, expect repositories
62in this format.
63
James E. Blaireff5a9d2017-06-20 00:00:37 -070064Note that these git repositories are located on the executor; in order
65to be useful to most kinds of jobs, they will need to be present on
66the test nodes. The ``base`` job in the standard library contains a
67pre-playbook which copies the repositories to all of the job's nodes.
68It is recommended to always inherit from this base job to ensure that
69behavior.
70
71.. TODO: link to base job documentation and/or document src (and logs?) directory
72
James E. Blair28c8e3b2017-07-17 16:27:50 -070073Variables
74---------
75
Jamie Lennox7655b552017-03-17 12:33:38 +110076There are several sources of variables which are available to Ansible:
77variables defined in jobs, secrets, and site-wide variables. The
78order of precedence is:
79
80* Site-wide variables
81
82* Secrets
83
84* Job variables
85
86Meaning that a site-wide variable with the same name as any other will
87override its value, and similarly, secrets override job variables of
88the same name. Each of the three sources is described below.
89
90
91Job Variables
92~~~~~~~~~~~~~
93
James E. Blaird9f0efb2017-08-02 16:07:44 -070094Any variables specified in the job definition (using the
95:attr:`job.vars` attribute) are available as Ansible host variables.
96They are added to the ``vars`` section of the inventory file under the
97``all`` hosts group, so they are available to all hosts. Simply refer
98to them by the name specified in the job's ``vars`` section.
James E. Blair28c8e3b2017-07-17 16:27:50 -070099
100Secrets
101~~~~~~~
102
James E. Blaird9f0efb2017-08-02 16:07:44 -0700103:ref:`Secrets <secret>` also appear as variables available to Ansible.
104Unlike job variables, these are not added to the inventory file (so
105that the inventory file may be kept for debugging purposes without
106revealing secrets). But they are still available to Ansible as normal
James E. Blair28c8e3b2017-07-17 16:27:50 -0700107variables. Because secrets are groups of variables, they will appear
108as a dictionary structure in templates, with the dictionary itself
109being the name of the secret, and its members the individual items in
James E. Blaird9f0efb2017-08-02 16:07:44 -0700110the secret. For example, a secret defined as:
111
112.. code-block:: yaml
James E. Blair28c8e3b2017-07-17 16:27:50 -0700113
114 - secret:
115 name: credentials
116 data:
117 username: foo
118 password: bar
119
120Might be used in a template as::
121
122 {{ credentials.username }} {{ credentials.password }}
123
James E. Blair28c8e3b2017-07-17 16:27:50 -0700124
James E. Blaireff5a9d2017-06-20 00:00:37 -0700125Zuul Variables
James E. Blair28c8e3b2017-07-17 16:27:50 -0700126~~~~~~~~~~~~~~
James E. Blaireff5a9d2017-06-20 00:00:37 -0700127
128Zuul supplies not only the variables specified by the job definition
James E. Blaird9f0efb2017-08-02 16:07:44 -0700129to Ansible, but also some variables from Zuul itself.
James E. Blair21037782017-07-19 11:56:55 -0700130
James E. Blairbabefce2017-07-20 17:14:54 -0700131When a pipeline is triggered by an action, it enqueues items which may
James E. Blair21037782017-07-19 11:56:55 -0700132vary based on the pipeline's configuration. For example, when a new
133change is created, that change may be enqueued into the pipeline,
134while a tag may be enqueued into the pipeline when it is pushed.
135
136Information about these items is available to jobs. All of the items
137enqueued in a pipeline are git references, and therefore share some
138attributes in common. But other attributes may vary based on the type
139of item.
140
James E. Blaird9f0efb2017-08-02 16:07:44 -0700141.. var:: zuul
James E. Blair21037782017-07-19 11:56:55 -0700142
James E. Blaird9f0efb2017-08-02 16:07:44 -0700143 All items provide the following information as Ansible variables
144 under the ``zuul`` key:
James E. Blair21037782017-07-19 11:56:55 -0700145
James E. Blaird9f0efb2017-08-02 16:07:44 -0700146 .. var:: build
James E. Blair21037782017-07-19 11:56:55 -0700147
James E. Blaird9f0efb2017-08-02 16:07:44 -0700148 The UUID of the build. A build is a single execution of a job.
149 When an item is enqueued into a pipeline, this usually results
150 in one build of each job configured for that item's project.
151 However, items may be re-enqueued in which case another build
152 may run. In dependent pipelines, the same job may run multiple
153 times for the same item as circumstances change ahead in the
154 queue. Each time a job is run, for whatever reason, it is
155 acompanied with a new unique id.
James E. Blair21037782017-07-19 11:56:55 -0700156
James E. Blaird9f0efb2017-08-02 16:07:44 -0700157 .. var:: buildset
James E. Blair21037782017-07-19 11:56:55 -0700158
James E. Blaird9f0efb2017-08-02 16:07:44 -0700159 The build set UUID. When Zuul runs jobs for an item, the
160 collection of those jobs is known as a buildset. If the
161 configuration of items ahead in a dependent pipeline changes,
162 Zuul creates a new buildset and restarts all of the jobs.
James E. Blair21037782017-07-19 11:56:55 -0700163
James E. Blaird9f0efb2017-08-02 16:07:44 -0700164 .. var:: ref
James E. Blair2ef29e92017-07-21 15:25:05 -0700165
James E. Blaird9f0efb2017-08-02 16:07:44 -0700166 The git ref of the item. This will be the full path (e.g.,
167 `refs/heads/master` or `refs/changes/...`).
James E. Blair21037782017-07-19 11:56:55 -0700168
James E. Blaird9f0efb2017-08-02 16:07:44 -0700169 .. var:: pipeline
James E. Blair21037782017-07-19 11:56:55 -0700170
James E. Blaird9f0efb2017-08-02 16:07:44 -0700171 The name of the pipeline in which the job is being run.
Monty Taylor299f94b2017-07-28 17:16:36 -0500172
James E. Blaird9f0efb2017-08-02 16:07:44 -0700173 .. var:: job
James E. Blair21037782017-07-19 11:56:55 -0700174
James E. Blaird9f0efb2017-08-02 16:07:44 -0700175 The name of the job being run.
James E. Blair21037782017-07-19 11:56:55 -0700176
James E. Blaird9f0efb2017-08-02 16:07:44 -0700177 .. var:: voting
James E. Blair21037782017-07-19 11:56:55 -0700178
James E. Blaird9f0efb2017-08-02 16:07:44 -0700179 A boolean indicating whether the job is voting.
James E. Blair21037782017-07-19 11:56:55 -0700180
James E. Blaird9f0efb2017-08-02 16:07:44 -0700181 .. var:: project
James E. Blair21037782017-07-19 11:56:55 -0700182
James E. Blaird9f0efb2017-08-02 16:07:44 -0700183 The item's project. This is a data structure with the following
184 fields:
James E. Blair21037782017-07-19 11:56:55 -0700185
James E. Blaird9f0efb2017-08-02 16:07:44 -0700186 .. var:: name
Monty Taylor299f94b2017-07-28 17:16:36 -0500187
James E. Blaird9f0efb2017-08-02 16:07:44 -0700188 The name of the project, excluding hostname. E.g., `org/project`.
Monty Taylor299f94b2017-07-28 17:16:36 -0500189
James E. Blaird9f0efb2017-08-02 16:07:44 -0700190 .. var:: short_name
Monty Taylor299f94b2017-07-28 17:16:36 -0500191
James E. Blaird9f0efb2017-08-02 16:07:44 -0700192 The name of the project, excluding directories or
193 organizations. E.g., `project`.
Monty Taylor299f94b2017-07-28 17:16:36 -0500194
James E. Blaird9f0efb2017-08-02 16:07:44 -0700195 .. var:: canonical_hostname
Monty Taylor299f94b2017-07-28 17:16:36 -0500196
James E. Blaird9f0efb2017-08-02 16:07:44 -0700197 The canonical hostname where the project lives. E.g.,
198 `git.example.com`.
Monty Taylor299f94b2017-07-28 17:16:36 -0500199
James E. Blaird9f0efb2017-08-02 16:07:44 -0700200 .. var:: canonical_name
201
202 The full canonical name of the project including hostname.
203 E.g., `git.example.com/org/project`.
204
205 .. var:: tenant
206
207 The name of the current Zuul tenant.
208
209 .. var:: jobtags
210
211 A list of tags associated with the job. Not to be confused with
212 git tags, these are simply free-form text fields that can be
213 used by the job for reporting or classification purposes.
214
215 .. var:: items
216 :type: list
217
218 A list of dictionaries, each representing an item being tested
219 with this change with the format:
220
221 .. var:: project
222
223 The item's project. This is a data structure with the
224 following fields:
225
226 .. var:: name
227
228 The name of the project, excluding hostname. E.g.,
229 `org/project`.
230
231 .. var:: short_name
232
233 The name of the project, excluding directories or
234 organizations. E.g., `project`.
235
236 .. var:: canonical_hostname
237
238 The canonical hostname where the project lives. E.g.,
239 `git.example.com`.
240
241 .. var:: canonical_name
242
243 The full canonical name of the project including hostname.
244 E.g., `git.example.com/org/project`.
245
246 .. var:: branch
247
248 The target branch of the change (without the `refs/heads/` prefix).
249
250 .. var:: change
251
252 The identifier for the change.
253
254 .. var:: patchset
255
256 The patchset identifier for the change. If a change is
257 revised, this will have a different value.
James E. Blair21037782017-07-19 11:56:55 -0700258
259Change Items
260++++++++++++
261
262A change to the repository. Most often, this will be a git reference
263which has not yet been merged into the repository (e.g., a gerrit
264change or a GitHub pull request). The following additional variables
265are available:
266
James E. Blaird9f0efb2017-08-02 16:07:44 -0700267.. var:: zuul
268 :hidden:
James E. Blair21037782017-07-19 11:56:55 -0700269
James E. Blaird9f0efb2017-08-02 16:07:44 -0700270 .. var:: branch
James E. Blair21037782017-07-19 11:56:55 -0700271
James E. Blaird9f0efb2017-08-02 16:07:44 -0700272 The target branch of the change (without the `refs/heads/` prefix).
273
274 .. var:: change
275
276 The identifier for the change.
277
278 .. var:: patchset
279
280 The patchset identifier for the change. If a change is revised,
281 this will have a different value.
James E. Blair21037782017-07-19 11:56:55 -0700282
283Branch Items
284++++++++++++
285
286This represents a branch tip. This item may have been enqueued
287because the branch was updated (via a change having merged, or a
288direct push). Or it may have been enqueued by a timer for the purpose
289of verifying the current condition of the branch. The following
290additional variables are available:
291
James E. Blaird9f0efb2017-08-02 16:07:44 -0700292.. var:: zuul
293 :hidden:
James E. Blair21037782017-07-19 11:56:55 -0700294
James E. Blaird9f0efb2017-08-02 16:07:44 -0700295 .. var:: branch
James E. Blair21037782017-07-19 11:56:55 -0700296
James E. Blaird9f0efb2017-08-02 16:07:44 -0700297 The name of the item's branch (without the `refs/heads/`
298 prefix).
299
300 .. var:: oldrev
301
302 If the item was enqueued as the result of a change merging or
303 being pushed to the branch, the git sha of the old revision will
304 be included here. Otherwise, this variable will be undefined.
305
306 .. var:: newrev
307
308 If the item was enqueued as the result of a change merging or
309 being pushed to the branch, the git sha of the new revision will
310 be included here. Otherwise, this variable will be undefined.
James E. Blair21037782017-07-19 11:56:55 -0700311
312Tag Items
313+++++++++
314
315This represents a git tag. The item may have been enqueued because a
316tag was created or deleted. The following additional variables are
317available:
318
James E. Blaird9f0efb2017-08-02 16:07:44 -0700319.. var:: zuul
320 :hidden:
James E. Blair21037782017-07-19 11:56:55 -0700321
James E. Blaird9f0efb2017-08-02 16:07:44 -0700322 .. var:: tag
James E. Blair21037782017-07-19 11:56:55 -0700323
James E. Blaird9f0efb2017-08-02 16:07:44 -0700324 The name of the item's tag (without the `refs/tags/` prefix).
325
326 .. var:: oldrev
327
328 If the item was enqueued as the result of a tag being deleted,
329 the previous git sha of the tag will be included here. If the
330 tag was created, this variable will be undefined.
331
332 .. var:: newrev
333
334 If the item was enqueued as the result of a tag being created,
335 the new git sha of the tag will be included here. If the tag
336 was deleted, this variable will be undefined.
James E. Blair21037782017-07-19 11:56:55 -0700337
338Ref Items
339+++++++++
340
341This represents a git reference that is neither a change, branch, or
342tag. Note that all items include a `ref` attribute which may be used
343to identify the ref. The following additional variables are
344available:
345
James E. Blaird9f0efb2017-08-02 16:07:44 -0700346.. var:: zuul
347 :hidden:
James E. Blair21037782017-07-19 11:56:55 -0700348
James E. Blaird9f0efb2017-08-02 16:07:44 -0700349 .. var:: oldrev
350
351 If the item was enqueued as the result of a ref being deleted,
352 the previous git sha of the ref will be included here. If the
353 ref was created, this variable will be undefined.
354
355 .. var:: newrev
356
357 If the item was enqueued as the result of a ref being created,
358 the new git sha of the ref will be included here. If the ref
359 was deleted, this variable will be undefined.
James E. Blair21037782017-07-19 11:56:55 -0700360
361Working Directory
362+++++++++++++++++
363
364Additionally, some information about the working directory and the
365executor running the job is available:
James E. Blaireff5a9d2017-06-20 00:00:37 -0700366
James E. Blaird9f0efb2017-08-02 16:07:44 -0700367.. var:: zuul
368 :hidden:
James E. Blaireff5a9d2017-06-20 00:00:37 -0700369
James E. Blaird9f0efb2017-08-02 16:07:44 -0700370 .. var:: executor
James E. Blaireff5a9d2017-06-20 00:00:37 -0700371
James E. Blaird9f0efb2017-08-02 16:07:44 -0700372 A number of values related to the executor running the job are
373 available:
James E. Blaireff5a9d2017-06-20 00:00:37 -0700374
James E. Blaird9f0efb2017-08-02 16:07:44 -0700375 .. var:: hostname
376
377 The hostname of the executor.
378
379 .. var:: src_root
380
381 The path to the source directory.
382
383 .. var:: log_root
384
385 The path to the logs directory.
386
387 .. var:: work_root
388
389 The path to the working directory.
Jamie Lennox7655b552017-03-17 12:33:38 +1100390
391.. _user_sitewide_variables:
392
393Site-wide Variables
394~~~~~~~~~~~~~~~~~~~
395
396The Zuul administrator may define variables which will be available to
397all jobs running in the system. These are statically defined and may
398not be altered by jobs. See the :ref:`Administrator's Guide
399<admin_sitewide_variables>` for information on how a site
400administrator may define these variables.
401
402
James E. Blaireff5a9d2017-06-20 00:00:37 -0700403SSH Keys
404--------
405
406Zuul starts each job with an SSH agent running and the key used to
407access the job's nodes added to that agent. Generally you won't need
408to be aware of this since Ansible will use this when performing any
409tasks on remote nodes. However, under some circumstances you may want
410to interact with the agent. For example, you may wish to add a key
411provided as a secret to the job in order to access a specific host, or
412you may want to, in a pre-playbook, replace the key used to log into
413the assigned nodes in order to further protect it from being abused by
414untrusted job content.
415
416.. TODO: describe standard lib and link to published docs for it.
417
James E. Blair88e79c02017-07-07 13:36:54 -0700418.. _return_values:
419
James E. Blair196f61a2017-06-30 15:42:29 -0700420Return Values
421-------------
422
423The job may return some values to Zuul to affect its behavior. To
424return a value, use the *zuul_return* Ansible module in a job
James E. Blaird9f0efb2017-08-02 16:07:44 -0700425playbook. For example:
426
427.. code-block:: yaml
James E. Blair196f61a2017-06-30 15:42:29 -0700428
429 tasks:
430 - zuul_return:
431 data:
432 foo: bar
433
434Will return the dictionary "{'foo': 'bar'}" to Zuul.
435
436.. TODO: xref to section describing formatting
437
438Several uses of these values are planned, but the only currently
439implemented use is to set the log URL for a build. To do so, set the
James E. Blaird9f0efb2017-08-02 16:07:44 -0700440**zuul.log_url** value. For example:
441
442.. code-block:: yaml
James E. Blair196f61a2017-06-30 15:42:29 -0700443
444 tasks:
445 - zuul_return:
446 data:
447 zuul:
448 log_url: http://logs.example.com/path/to/build/logs