blob: f3e45db02c5cadd17c15a96786c57e5f9a0848fe [file] [log] [blame]
James E. Blaircdd00072012-06-08 19:17:28 -07001:title: Launchers
2
James E. Blair1f4c2bb2013-04-26 08:40:46 -07003.. _Gearman: http://gearman.org/
Antoine Mussod06f2a62012-11-16 17:40:58 +01004
James E. Blair1f4c2bb2013-04-26 08:40:46 -07005.. _`Gearman Plugin`:
6 https://wiki.jenkins-ci.org/display/JENKINS/Gearman+Plugin
7
Joshua Hesketh6a0a6182014-03-13 13:23:30 +11008.. _`Turbo-Hipster`:
9 http://git.openstack.org/cgit/stackforge/turbo-hipster/
10
11.. _`Turbo-Hipster Documentation`:
12 http://turbo-hipster.rtfd.org/
13
James E. Blair1f4c2bb2013-04-26 08:40:46 -070014.. _launchers:
Antoine Mussod06f2a62012-11-16 17:40:58 +010015
James E. Blaircdd00072012-06-08 19:17:28 -070016Launchers
17=========
18
James E. Blair1f4c2bb2013-04-26 08:40:46 -070019Zuul has a modular architecture for launching jobs. Currently, the
20only supported module interfaces with Gearman_. This design allows
21any system to run jobs for Zuul simply by interfacing with a Gearman
22server. The recommended way of integrating a new job-runner with Zuul
23is via this method.
James E. Blaircdd00072012-06-08 19:17:28 -070024
James E. Blair1f4c2bb2013-04-26 08:40:46 -070025If Gearman is unsuitable, Zuul may be extended with a new launcher
26module. Zuul makes very few assumptions about the interface to a
27launcher -- if it can trigger jobs, cancel them, and receive success
28or failure reports, it should be able to be used with Zuul. Patches
29to this effect are welcome.
30
James E. Blair1f4c2bb2013-04-26 08:40:46 -070031Zuul Parameters
32---------------
James E. Blaircdd00072012-06-08 19:17:28 -070033
Joshua Heskethe8987162014-03-13 13:05:33 +110034Zuul will pass some parameters with every job it launches. These are
35for workers to be able to get the repositories into the state they are
36intended to be tested in. Builds can be triggered either by an action
37on a change or by a reference update. Both events share a common set
38of parameters and more specific parameters as follows:
Antoine Mussoe0f5bb42014-03-04 11:30:37 +010039
40Common parameters
41~~~~~~~~~~~~~~~~~
James E. Blaircdd00072012-06-08 19:17:28 -070042
James E. Blair81515ad2012-10-01 18:29:08 -070043**ZUUL_UUID**
Łukasz Jernaś048acb42014-03-02 18:49:41 +010044 Zuul provided key to link builds with Gerrit events.
James E. Blair81515ad2012-10-01 18:29:08 -070045**ZUUL_REF**
Łukasz Jernaś048acb42014-03-02 18:49:41 +010046 Zuul provided ref that includes commit(s) to build.
James E. Blair81515ad2012-10-01 18:29:08 -070047**ZUUL_COMMIT**
Łukasz Jernaś048acb42014-03-02 18:49:41 +010048 The commit SHA1 at the head of ZUUL_REF.
James E. Blair81515ad2012-10-01 18:29:08 -070049**ZUUL_PROJECT**
Łukasz Jernaś048acb42014-03-02 18:49:41 +010050 The project that triggered this build.
James E. Blair81515ad2012-10-01 18:29:08 -070051**ZUUL_PIPELINE**
Łukasz Jernaś048acb42014-03-02 18:49:41 +010052 The Zuul pipeline that is building this job.
Arx Cruzb1b010d2013-10-28 19:49:59 -020053**ZUUL_URL**
Łukasz Jernaś048acb42014-03-02 18:49:41 +010054 The URL for the zuul server as configured in zuul.conf.
Joshua Heskethba8776a2014-01-12 14:35:40 +080055 A test runner may use this URL as the basis for fetching
Arx Cruzb1b010d2013-10-28 19:49:59 -020056 git commits.
James E. Blaircdd00072012-06-08 19:17:28 -070057
Antoine Mussoe0f5bb42014-03-04 11:30:37 +010058Change related parameters
59~~~~~~~~~~~~~~~~~~~~~~~~~
60
James E. Blair1f4c2bb2013-04-26 08:40:46 -070061The following additional parameters will only be provided for builds
62associated with changes (i.e., in response to patchset-created or
63comment-added events):
James E. Blaircdd00072012-06-08 19:17:28 -070064
James E. Blair81515ad2012-10-01 18:29:08 -070065**ZUUL_BRANCH**
Łukasz Jernaś048acb42014-03-02 18:49:41 +010066 The target branch for the change that triggered this build.
James E. Blair81515ad2012-10-01 18:29:08 -070067**ZUUL_CHANGE**
Łukasz Jernaś048acb42014-03-02 18:49:41 +010068 The Gerrit change ID for the change that triggered this build.
James E. Blair81515ad2012-10-01 18:29:08 -070069**ZUUL_CHANGE_IDS**
70 All of the Gerrit change IDs that are included in this build (useful
Łukasz Jernaś048acb42014-03-02 18:49:41 +010071 when the DependentPipelineManager combines changes for testing).
James E. Blair81515ad2012-10-01 18:29:08 -070072**ZUUL_PATCHSET**
Łukasz Jernaś048acb42014-03-02 18:49:41 +010073 The Gerrit patchset number for the change that triggered this build.
James E. Blaircdd00072012-06-08 19:17:28 -070074
Antoine Mussoe0f5bb42014-03-04 11:30:37 +010075Reference updated parameters
76~~~~~~~~~~~~~~~~~~~~~~~~~~~~
77
James E. Blair1f4c2bb2013-04-26 08:40:46 -070078The following additional parameters will only be provided for
James E. Blair81515ad2012-10-01 18:29:08 -070079post-merge (ref-updated) builds:
James E. Blaircdd00072012-06-08 19:17:28 -070080
James E. Blair81515ad2012-10-01 18:29:08 -070081**ZUUL_OLDREV**
82 The SHA1 of the old revision at this ref (recall the ref name is
Łukasz Jernaś048acb42014-03-02 18:49:41 +010083 in ZUUL_REF).
James E. Blair81515ad2012-10-01 18:29:08 -070084**ZUUL_NEWREV**
85 The SHA1 of the new revision at this ref (recall the ref name is
Łukasz Jernaś048acb42014-03-02 18:49:41 +010086 in ZUUL_REF).
James E. Blair81515ad2012-10-01 18:29:08 -070087**ZUUL_SHORT_OLDREV**
Łukasz Jernaś048acb42014-03-02 18:49:41 +010088 The shortened (7 character) SHA1 of the old revision.
James E. Blair81515ad2012-10-01 18:29:08 -070089**ZUUL_SHORT_NEWREV**
Łukasz Jernaś048acb42014-03-02 18:49:41 +010090 The shortened (7 character) SHA1 of the new revision.
James E. Blaircdd00072012-06-08 19:17:28 -070091
Antoine Mussoe0f5bb42014-03-04 11:30:37 +010092Unset revisions default to 00000000000000000000000000000000.
93
94Examples:
95
96When a reference is created::
97
98 ZUUL_OLDREV=00000000000000000000000000000000
99 ZUUL_NEWREV=123456789abcdef123456789abcdef12
100 ZUUL_SHORT_OLDREV=0000000
101 ZUUL_SHORT_NEWREV=1234567
102
103When a reference is deleted::
104
105 ZUUL_OLDREV=123456789abcdef123456789abcdef12
106 ZUUL_NEWREV=00000000000000000000000000000000
107 ZUUL_SHORT_OLDREV=1234567
108 ZUUL_SHORT_NEWREV=0000000
109
110And finally a reference being altered::
111
112 ZUUL_OLDREV=123456789abcdef123456789abcdef12
113 ZUUL_NEWREV=abcdef123456789abcdef123456789ab
114 ZUUL_SHORT_OLDREV=1234567
115 ZUUL_SHORT_NEWREV=abcdef1
116
117Your jobs can check whether the parameters are ``000000`` to act
118differently on each kind of event.
119
Joshua Heskethe8987162014-03-13 13:05:33 +1100120Gearman
121-------
122
123Gearman_ is a general-purpose protocol for distributing jobs to any
124number of workers. Zuul works with Gearman by sending specific
125information with job requests to Gearman, and expects certain
126information to be returned on completion. This protocol is described
127in `Zuul-Gearman Protocol`_.
128
129In order for Zuul to run any jobs, you will need a running Gearman
130server. Zuul includes a Gearman server, and it is recommended that it
131be used as it supports the following features needed by Zuul:
132
133* Canceling jobs in the queue (admin protocol command "cancel job").
134* Strict FIFO queue operation (gearmand's round-robin mode may be
135 sufficient, but is untested).
136
137To enable the built-in server, see the ``gearman_server`` section of
138``zuul.conf``. Be sure that the host allows connections from Zuul and
139any workers (e.g., Jenkins masters) on TCP port 4730, and nowhere else
140(as the Gearman protocol does not include any provision for
141authentication).
142
143Gearman Jenkins Plugin
144~~~~~~~~~~~~~~~~~~~~~~
145
146The `Gearman Jenkins Plugin`_ makes it easy to use Jenkins with Zuul
147by providing an interface between Jenkins and Gearman. In this
148configuration, Zuul asks Gearman to run jobs, and Gearman can then
149distribute those jobs to any number of Jenkins systems (including
150multiple Jenkins masters).
151
152The `Gearman Plugin`_ can be installed in Jenkins in order to
153facilitate Jenkins running jobs for Zuul. Install the plugin and
154configure it with the hostname or IP address of your Gearman server
155and the port on which it is listening (4730 by default). It will
156automatically register all known Jenkins jobs as functions that Zuul
157can invoke via Gearman.
158
159Any number of masters can be configured in this way, and Gearman will
160distribute jobs to all of them as appropriate.
161
162No special Jenkins job configuration is needed to support triggering
163by Zuul.
164
165The Gearman Plugin will ensure the `Zuul Parameters`_ are supplied as
166Jenkins build parameters, so they will be available for use in the job
167configuration as well as to the running job as environment variables.
168
Antoine Mussoe0f5bb42014-03-04 11:30:37 +0100169Jenkins git plugin configuration
Joshua Heskethe8987162014-03-13 13:05:33 +1100170^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Antoine Mussoe0f5bb42014-03-04 11:30:37 +0100171
James E. Blair81515ad2012-10-01 18:29:08 -0700172In order to test the correct build, configure the Jenkins Git SCM
173plugin as follows::
James E. Blaircdd00072012-06-08 19:17:28 -0700174
James E. Blair81515ad2012-10-01 18:29:08 -0700175 Source Code Management:
176 Git
177 Repositories:
178 Repository URL: <your Gerrit or Zuul repository URL>
179 Advanced:
180 Refspec: ${ZUUL_REF}
181 Branches to build:
182 Branch Specifier: ${ZUUL_COMMIT}
James E. Blaire2819012013-06-28 17:17:26 -0400183 Advanced:
184 Clean after checkout: True
James E. Blaircdd00072012-06-08 19:17:28 -0700185
James E. Blair81515ad2012-10-01 18:29:08 -0700186That should be sufficient for a job that only builds a single project.
187If you have multiple interrelated projects (i.e., they share a Zuul
188Change Queue) that are built together, you may be able to configure
189the Git plugin to prepare them, or you may chose to use a shell script
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700190instead. As an example, the OpenStack project uses the following
191script to prepare the workspace for its integration testing:
James E. Blair81515ad2012-10-01 18:29:08 -0700192
K Jonathan Harker97e457e2013-02-26 13:29:38 -0800193 https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate-wrap.sh
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700194
Joshua Hesketh6a0a6182014-03-13 13:23:30 +1100195Turbo Hipster Worker
196~~~~~~~~~~~~~~~~~~~~
197
198As an alternative to Jenkins, `Turbo-Hipster`_ is a small python
199project designed specifically as a zuul job worker which can be
200registered with gearman as a job runner. Please see the
201`Turbo-Hipster Documentation`_ for details on how to set it up.
202
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700203Zuul-Gearman Protocol
Joshua Heskethe8987162014-03-13 13:05:33 +1100204~~~~~~~~~~~~~~~~~~~~~
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700205
206This section is only relevant if you intend to implement a new kind of
207worker that runs jobs for Zuul via Gearman. If you just want to use
208Jenkins, see `Gearman Jenkins Plugin`_ instead.
209
210The Zuul protocol as used with Gearman is as follows:
211
212Starting Builds
Joshua Heskethe8987162014-03-13 13:05:33 +1100213^^^^^^^^^^^^^^^
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700214
215To start a build, Zuul invokes a Gearman function with the following
216format:
217
218 build:FUNCTION_NAME
219
220where **FUNCTION_NAME** is the name of the job that should be run. If
221the job should run on a specific node (or class of node), Zuul will
222instead invoke:
223
224 build:FUNCTION_NAME:NODE_NAME
225
226where **NODE_NAME** is the name or class of node on which the job
227should be run. This can be specified by setting the ZUUL_NODE
Antoine Musso7c5db972013-09-26 11:48:26 +0200228parameter in a parameter-function (see :ref:`includes` section in
229:ref:`zuulconf`).
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700230
231Zuul sends the ZUUL_* parameters described in `Zuul Parameters`_
232encoded in JSON format as the argument included with the
233SUBMIT_JOB_UNIQ request to Gearman. A unique ID (equal to the
234ZUUL_UUID parameter) is also supplied to Gearman, and is accessible as
235an added Gearman parameter with GRAB_JOB_UNIQ.
236
237When a Gearman worker starts running a job for Zuul, it should
238immediately send a WORK_DATA packet with the following information
239encoded in JSON format:
240
James E. Blair3c483cf2013-06-04 16:30:43 -0700241**name**
242 The name of the job.
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700243
244**number**
245 The build number (unique to this job).
246
James E. Blair3c483cf2013-06-04 16:30:43 -0700247**manager**
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700248 A unique identifier associated with the Gearman worker that can
249 abort this build. See `Stopping Builds`_ for more information.
250
James E. Blair3c483cf2013-06-04 16:30:43 -0700251**url** (optional)
252 The URL with the status or results of the build. Will be used in
253 the status page and the final report.
254
Joshua Heskethba8776a2014-01-12 14:35:40 +0800255To help with debugging builds a worker may send back some optional
256metadata:
257
258**worker_name** (optional)
259 The name of the worker.
260
261**worker_hostname** (optional)
262 The hostname of the worker.
263
264**worker_ips** (optional)
265 A list of IPs for the worker.
266
267**worker_fqdn** (optional)
268 The FQDN of the worker.
269
270**worker_program** (optional)
271 The program name of the worker. For example Jenkins or turbo-hipster.
272
273**worker_version** (optional)
274 The version of the software running the job.
275
276**worker_extra** (optional)
277 A dictionary of any extra metadata you may want to pass along.
278
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700279It should then immediately send a WORK_STATUS packet with a value of 0
280percent complete. It may then optionally send subsequent WORK_STATUS
281packets with updated completion values.
282
283When the build is complete, it should send a final WORK_DATA packet
284with the following in JSON format:
285
286**result**
287 Either the string 'SUCCESS' if the job succeeded, or any other value
288 that describes the result if the job failed.
289
290Finally, it should send either a WORK_FAIL or WORK_COMPLETE packet as
291appropriate. A WORK_EXCEPTION packet will be interpreted as a
292WORK_FAIL, but the exception will be logged in Zuul's error log.
293
294Stopping Builds
Joshua Heskethe8987162014-03-13 13:05:33 +1100295^^^^^^^^^^^^^^^
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700296
297If Zuul needs to abort a build already in progress, it will invoke the
298following function through Gearman:
299
James E. Blair3c483cf2013-06-04 16:30:43 -0700300 stop:MANAGER_NAME
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700301
James E. Blair3c483cf2013-06-04 16:30:43 -0700302Where **MANAGER_NAME** is the name of the manager worker supplied in
303the initial WORK_DATA packet when the job started. This is used to
304direct the stop: function invocation to the correct Gearman worker
305that is capable of stopping that particular job. The argument to the
306function should be the following encoded in JSON format:
307
308**name**
309 The job name of the build to stop.
310
311**number**
312 The build number of the build to stop.
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700313
314The original job is expected to complete with a WORK_DATA and
315WORK_FAIL packet as described in `Starting Builds`_.
316
317Build Descriptions
Joshua Heskethe8987162014-03-13 13:05:33 +1100318^^^^^^^^^^^^^^^^^^
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700319
320In order to update the job running system with a description of the
321current state of all related builds, the job runner may optionally
322implement the following Gearman function:
323
James E. Blair3c483cf2013-06-04 16:30:43 -0700324 set_description:MANAGER_NAME
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700325
James E. Blair3c483cf2013-06-04 16:30:43 -0700326Where **MANAGER_NAME** is used as described in `Stopping Builds`_.
327The argument to the function is the following encoded in JSON format:
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700328
James E. Blair3c483cf2013-06-04 16:30:43 -0700329**name**
330 The job name of the build to describe.
331
332**number**
333 The build number of the build to describe.
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700334
335**html_description**
336 The description of the build in HTML format.