blob: db499333e75e238ad32119c9cc43be166cdfcf61 [file] [log] [blame]
James E. Blaircdd00072012-06-08 19:17:28 -07001:title: Launchers
2
James E. Blair1f4c2bb2013-04-26 08:40:46 -07003.. _Gearman: http://gearman.org/
Antoine Mussod06f2a62012-11-16 17:40:58 +01004
James E. Blair1f4c2bb2013-04-26 08:40:46 -07005.. _`Gearman Plugin`:
6 https://wiki.jenkins-ci.org/display/JENKINS/Gearman+Plugin
7
8.. _launchers:
Antoine Mussod06f2a62012-11-16 17:40:58 +01009
James E. Blaircdd00072012-06-08 19:17:28 -070010Launchers
11=========
12
James E. Blair1f4c2bb2013-04-26 08:40:46 -070013Zuul has a modular architecture for launching jobs. Currently, the
14only supported module interfaces with Gearman_. This design allows
15any system to run jobs for Zuul simply by interfacing with a Gearman
16server. The recommended way of integrating a new job-runner with Zuul
17is via this method.
James E. Blaircdd00072012-06-08 19:17:28 -070018
James E. Blair1f4c2bb2013-04-26 08:40:46 -070019If Gearman is unsuitable, Zuul may be extended with a new launcher
20module. Zuul makes very few assumptions about the interface to a
21launcher -- if it can trigger jobs, cancel them, and receive success
22or failure reports, it should be able to be used with Zuul. Patches
23to this effect are welcome.
24
25Gearman
James E. Blaircdd00072012-06-08 19:17:28 -070026-------
27
James E. Blair1f4c2bb2013-04-26 08:40:46 -070028Gearman_ is a general-purpose protocol for distributing jobs to any
29number of workers. Zuul works with Gearman by sending specific
30information with job requests to Gearman, and expects certain
31information to be returned on completion. This protocol is described
32in `Zuul-Gearman Protocol`_.
James E. Blaircdd00072012-06-08 19:17:28 -070033
James E. Blair1f4c2bb2013-04-26 08:40:46 -070034The `Gearman Jenkins Plugin`_ makes it easy to use Jenkins with Zuul
35by providing an interface between Jenkins and Gearman. In this
36configuration, Zuul asks Gearman to run jobs, and Gearman can then
37distribute those jobs to any number of Jenkins systems (including
38multiple Jenkins masters).
James E. Blaircdd00072012-06-08 19:17:28 -070039
James E. Blair1f4c2bb2013-04-26 08:40:46 -070040In order for Zuul to run any jobs, you will need a running Gearman
James E. Blair77cc7b82013-07-15 13:22:37 -070041server. Zuul includes a Gearman server, and it is recommended that it
42be used as it supports the following features needed by Zuul:
43
44* Canceling jobs in the queue (admin protocol command "cancel job").
45* Strict FIFO queue operation (gearmand's round-robin mode may be
46 sufficient, but is untested).
47
48To enable the built-in server, see the ``gearman_server`` section of
49``zuul.conf``. Be sure that the host allows connections from Zuul and
50any workers (e.g., Jenkins masters) on TCP port 4730, and nowhere else
James E. Blair1f4c2bb2013-04-26 08:40:46 -070051(as the Gearman protocol does not include any provision for
Michael Prokop526926a2013-10-24 16:16:57 +020052authentication).
James E. Blaircdd00072012-06-08 19:17:28 -070053
James E. Blair1f4c2bb2013-04-26 08:40:46 -070054Gearman Jenkins Plugin
55----------------------
James E. Blaircdd00072012-06-08 19:17:28 -070056
James E. Blair1f4c2bb2013-04-26 08:40:46 -070057The `Gearman Plugin`_ can be installed in Jenkins in order to
58facilitate Jenkins running jobs for Zuul. Install the plugin and
59configure it with the hostname or IP address of your Gearman server
60and the port on which it is listening (4730 by default). It will
61automatically register all known Jenkins jobs as functions that Zuul
62can invoke via Gearman.
James E. Blaircdd00072012-06-08 19:17:28 -070063
James E. Blair1f4c2bb2013-04-26 08:40:46 -070064Any number of masters can be configured in this way, and Gearman will
65distribute jobs to all of them as appropriate.
James E. Blaircdd00072012-06-08 19:17:28 -070066
James E. Blair1f4c2bb2013-04-26 08:40:46 -070067No special Jenkins job configuration is needed to support triggering
68by Zuul.
James E. Blaircdd00072012-06-08 19:17:28 -070069
James E. Blair1f4c2bb2013-04-26 08:40:46 -070070Zuul Parameters
71---------------
James E. Blaircdd00072012-06-08 19:17:28 -070072
James E. Blair1f4c2bb2013-04-26 08:40:46 -070073Zuul will pass some parameters with every job it launches. The
74Gearman Plugin will ensure these are supplied as Jenkins build
75parameters, so they will be available for use in the job configuration
76as well as to the running job as environment variables. They are as
77follows:
James E. Blaircdd00072012-06-08 19:17:28 -070078
James E. Blair81515ad2012-10-01 18:29:08 -070079**ZUUL_UUID**
James E. Blaircdd00072012-06-08 19:17:28 -070080 Zuul provided key to link builds with Gerrit events
James E. Blair81515ad2012-10-01 18:29:08 -070081**ZUUL_REF**
82 Zuul provided ref that includes commit(s) to build
83**ZUUL_COMMIT**
84 The commit SHA1 at the head of ZUUL_REF
James E. Blair81515ad2012-10-01 18:29:08 -070085**ZUUL_PROJECT**
86 The project that triggered this build
87**ZUUL_PIPELINE**
88 The Zuul pipeline that is building this job
Arx Cruzb1b010d2013-10-28 19:49:59 -020089**ZUUL_URL**
Joshua Heskethba8776a2014-01-12 14:35:40 +080090 The url for the zuul server as configured in zuul.conf.
91 A test runner may use this URL as the basis for fetching
Arx Cruzb1b010d2013-10-28 19:49:59 -020092 git commits.
James E. Blaircdd00072012-06-08 19:17:28 -070093
James E. Blair1f4c2bb2013-04-26 08:40:46 -070094The following additional parameters will only be provided for builds
95associated with changes (i.e., in response to patchset-created or
96comment-added events):
James E. Blaircdd00072012-06-08 19:17:28 -070097
James E. Blair81515ad2012-10-01 18:29:08 -070098**ZUUL_BRANCH**
99 The target branch for the change that triggered this build
100**ZUUL_CHANGE**
101 The Gerrit change ID for the change that triggered this build
102**ZUUL_CHANGE_IDS**
103 All of the Gerrit change IDs that are included in this build (useful
104 when the DependentPipelineManager combines changes for testing)
105**ZUUL_PATCHSET**
106 The Gerrit patchset number for the change that triggered this build
James E. Blaircdd00072012-06-08 19:17:28 -0700107
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700108The following additional parameters will only be provided for
James E. Blair81515ad2012-10-01 18:29:08 -0700109post-merge (ref-updated) builds:
James E. Blaircdd00072012-06-08 19:17:28 -0700110
James E. Blair81515ad2012-10-01 18:29:08 -0700111**ZUUL_OLDREV**
112 The SHA1 of the old revision at this ref (recall the ref name is
113 in ZUUL_REF)
114**ZUUL_NEWREV**
115 The SHA1 of the new revision at this ref (recall the ref name is
116 in ZUUL_REF)
117**ZUUL_SHORT_OLDREV**
118 The shortened (7 character) SHA1 of the old revision
119**ZUUL_SHORT_NEWREV**
120 The shortened (7 character) SHA1 of the new revision
James E. Blaircdd00072012-06-08 19:17:28 -0700121
James E. Blair81515ad2012-10-01 18:29:08 -0700122In order to test the correct build, configure the Jenkins Git SCM
123plugin as follows::
James E. Blaircdd00072012-06-08 19:17:28 -0700124
James E. Blair81515ad2012-10-01 18:29:08 -0700125 Source Code Management:
126 Git
127 Repositories:
128 Repository URL: <your Gerrit or Zuul repository URL>
129 Advanced:
130 Refspec: ${ZUUL_REF}
131 Branches to build:
132 Branch Specifier: ${ZUUL_COMMIT}
James E. Blaire2819012013-06-28 17:17:26 -0400133 Advanced:
134 Clean after checkout: True
James E. Blaircdd00072012-06-08 19:17:28 -0700135
James E. Blair81515ad2012-10-01 18:29:08 -0700136That should be sufficient for a job that only builds a single project.
137If you have multiple interrelated projects (i.e., they share a Zuul
138Change Queue) that are built together, you may be able to configure
139the Git plugin to prepare them, or you may chose to use a shell script
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700140instead. As an example, the OpenStack project uses the following
141script to prepare the workspace for its integration testing:
James E. Blair81515ad2012-10-01 18:29:08 -0700142
K Jonathan Harker97e457e2013-02-26 13:29:38 -0800143 https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate-wrap.sh
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700144
145
146Zuul-Gearman Protocol
147---------------------
148
149This section is only relevant if you intend to implement a new kind of
150worker that runs jobs for Zuul via Gearman. If you just want to use
151Jenkins, see `Gearman Jenkins Plugin`_ instead.
152
153The Zuul protocol as used with Gearman is as follows:
154
155Starting Builds
156~~~~~~~~~~~~~~~
157
158To start a build, Zuul invokes a Gearman function with the following
159format:
160
161 build:FUNCTION_NAME
162
163where **FUNCTION_NAME** is the name of the job that should be run. If
164the job should run on a specific node (or class of node), Zuul will
165instead invoke:
166
167 build:FUNCTION_NAME:NODE_NAME
168
169where **NODE_NAME** is the name or class of node on which the job
170should be run. This can be specified by setting the ZUUL_NODE
Antoine Musso7c5db972013-09-26 11:48:26 +0200171parameter in a parameter-function (see :ref:`includes` section in
172:ref:`zuulconf`).
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700173
174Zuul sends the ZUUL_* parameters described in `Zuul Parameters`_
175encoded in JSON format as the argument included with the
176SUBMIT_JOB_UNIQ request to Gearman. A unique ID (equal to the
177ZUUL_UUID parameter) is also supplied to Gearman, and is accessible as
178an added Gearman parameter with GRAB_JOB_UNIQ.
179
180When a Gearman worker starts running a job for Zuul, it should
181immediately send a WORK_DATA packet with the following information
182encoded in JSON format:
183
James E. Blair3c483cf2013-06-04 16:30:43 -0700184**name**
185 The name of the job.
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700186
187**number**
188 The build number (unique to this job).
189
James E. Blair3c483cf2013-06-04 16:30:43 -0700190**manager**
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700191 A unique identifier associated with the Gearman worker that can
192 abort this build. See `Stopping Builds`_ for more information.
193
James E. Blair3c483cf2013-06-04 16:30:43 -0700194**url** (optional)
195 The URL with the status or results of the build. Will be used in
196 the status page and the final report.
197
Joshua Heskethba8776a2014-01-12 14:35:40 +0800198To help with debugging builds a worker may send back some optional
199metadata:
200
201**worker_name** (optional)
202 The name of the worker.
203
204**worker_hostname** (optional)
205 The hostname of the worker.
206
207**worker_ips** (optional)
208 A list of IPs for the worker.
209
210**worker_fqdn** (optional)
211 The FQDN of the worker.
212
213**worker_program** (optional)
214 The program name of the worker. For example Jenkins or turbo-hipster.
215
216**worker_version** (optional)
217 The version of the software running the job.
218
219**worker_extra** (optional)
220 A dictionary of any extra metadata you may want to pass along.
221
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700222It should then immediately send a WORK_STATUS packet with a value of 0
223percent complete. It may then optionally send subsequent WORK_STATUS
224packets with updated completion values.
225
226When the build is complete, it should send a final WORK_DATA packet
227with the following in JSON format:
228
229**result**
230 Either the string 'SUCCESS' if the job succeeded, or any other value
231 that describes the result if the job failed.
232
233Finally, it should send either a WORK_FAIL or WORK_COMPLETE packet as
234appropriate. A WORK_EXCEPTION packet will be interpreted as a
235WORK_FAIL, but the exception will be logged in Zuul's error log.
236
237Stopping Builds
238~~~~~~~~~~~~~~~
239
240If Zuul needs to abort a build already in progress, it will invoke the
241following function through Gearman:
242
James E. Blair3c483cf2013-06-04 16:30:43 -0700243 stop:MANAGER_NAME
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700244
James E. Blair3c483cf2013-06-04 16:30:43 -0700245Where **MANAGER_NAME** is the name of the manager worker supplied in
246the initial WORK_DATA packet when the job started. This is used to
247direct the stop: function invocation to the correct Gearman worker
248that is capable of stopping that particular job. The argument to the
249function should be the following encoded in JSON format:
250
251**name**
252 The job name of the build to stop.
253
254**number**
255 The build number of the build to stop.
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700256
257The original job is expected to complete with a WORK_DATA and
258WORK_FAIL packet as described in `Starting Builds`_.
259
260Build Descriptions
261~~~~~~~~~~~~~~~~~~
262
263In order to update the job running system with a description of the
264current state of all related builds, the job runner may optionally
265implement the following Gearman function:
266
James E. Blair3c483cf2013-06-04 16:30:43 -0700267 set_description:MANAGER_NAME
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700268
James E. Blair3c483cf2013-06-04 16:30:43 -0700269Where **MANAGER_NAME** is used as described in `Stopping Builds`_.
270The argument to the function is the following encoded in JSON format:
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700271
James E. Blair3c483cf2013-06-04 16:30:43 -0700272**name**
273 The job name of the build to describe.
274
275**number**
276 The build number of the build to describe.
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700277
278**html_description**
279 The description of the build in HTML format.