blob: 832321cb21629f7640fec670c75787b329b376aa [file] [log] [blame]
James E. Blaircdd00072012-06-08 19:17:28 -07001:title: Launchers
2
James E. Blair1f4c2bb2013-04-26 08:40:46 -07003.. _Gearman: http://gearman.org/
Antoine Mussod06f2a62012-11-16 17:40:58 +01004
James E. Blair1f4c2bb2013-04-26 08:40:46 -07005.. _`Gearman Plugin`:
6 https://wiki.jenkins-ci.org/display/JENKINS/Gearman+Plugin
7
8.. _launchers:
Antoine Mussod06f2a62012-11-16 17:40:58 +01009
James E. Blaircdd00072012-06-08 19:17:28 -070010Launchers
11=========
12
James E. Blair1f4c2bb2013-04-26 08:40:46 -070013Zuul has a modular architecture for launching jobs. Currently, the
14only supported module interfaces with Gearman_. This design allows
15any system to run jobs for Zuul simply by interfacing with a Gearman
16server. The recommended way of integrating a new job-runner with Zuul
17is via this method.
James E. Blaircdd00072012-06-08 19:17:28 -070018
James E. Blair1f4c2bb2013-04-26 08:40:46 -070019If Gearman is unsuitable, Zuul may be extended with a new launcher
20module. Zuul makes very few assumptions about the interface to a
21launcher -- if it can trigger jobs, cancel them, and receive success
22or failure reports, it should be able to be used with Zuul. Patches
23to this effect are welcome.
24
25Gearman
James E. Blaircdd00072012-06-08 19:17:28 -070026-------
27
James E. Blair1f4c2bb2013-04-26 08:40:46 -070028Gearman_ is a general-purpose protocol for distributing jobs to any
29number of workers. Zuul works with Gearman by sending specific
30information with job requests to Gearman, and expects certain
31information to be returned on completion. This protocol is described
32in `Zuul-Gearman Protocol`_.
James E. Blaircdd00072012-06-08 19:17:28 -070033
James E. Blair1f4c2bb2013-04-26 08:40:46 -070034The `Gearman Jenkins Plugin`_ makes it easy to use Jenkins with Zuul
35by providing an interface between Jenkins and Gearman. In this
36configuration, Zuul asks Gearman to run jobs, and Gearman can then
37distribute those jobs to any number of Jenkins systems (including
38multiple Jenkins masters).
James E. Blaircdd00072012-06-08 19:17:28 -070039
James E. Blair1f4c2bb2013-04-26 08:40:46 -070040In order for Zuul to run any jobs, you will need a running Gearman
41server. The latest version of gearmand from gearman.org is required
42in order to support canceling jobs while in the queue. The server is
43easy to set up -- just make sure that it allows connections from Zuul
44and any workers (e.g., Jenkins masters) on port 4730, and nowhere else
45(as the Gearman protocol does not include any provision for
46authentication.
James E. Blaircdd00072012-06-08 19:17:28 -070047
James E. Blair1f4c2bb2013-04-26 08:40:46 -070048Gearman Jenkins Plugin
49----------------------
James E. Blaircdd00072012-06-08 19:17:28 -070050
James E. Blair1f4c2bb2013-04-26 08:40:46 -070051The `Gearman Plugin`_ can be installed in Jenkins in order to
52facilitate Jenkins running jobs for Zuul. Install the plugin and
53configure it with the hostname or IP address of your Gearman server
54and the port on which it is listening (4730 by default). It will
55automatically register all known Jenkins jobs as functions that Zuul
56can invoke via Gearman.
James E. Blaircdd00072012-06-08 19:17:28 -070057
James E. Blair1f4c2bb2013-04-26 08:40:46 -070058Any number of masters can be configured in this way, and Gearman will
59distribute jobs to all of them as appropriate.
James E. Blaircdd00072012-06-08 19:17:28 -070060
James E. Blair1f4c2bb2013-04-26 08:40:46 -070061No special Jenkins job configuration is needed to support triggering
62by Zuul.
James E. Blaircdd00072012-06-08 19:17:28 -070063
James E. Blair1f4c2bb2013-04-26 08:40:46 -070064Zuul Parameters
65---------------
James E. Blaircdd00072012-06-08 19:17:28 -070066
James E. Blair1f4c2bb2013-04-26 08:40:46 -070067Zuul will pass some parameters with every job it launches. The
68Gearman Plugin will ensure these are supplied as Jenkins build
69parameters, so they will be available for use in the job configuration
70as well as to the running job as environment variables. They are as
71follows:
James E. Blaircdd00072012-06-08 19:17:28 -070072
James E. Blair81515ad2012-10-01 18:29:08 -070073**ZUUL_UUID**
James E. Blaircdd00072012-06-08 19:17:28 -070074 Zuul provided key to link builds with Gerrit events
James E. Blair81515ad2012-10-01 18:29:08 -070075**ZUUL_REF**
76 Zuul provided ref that includes commit(s) to build
77**ZUUL_COMMIT**
78 The commit SHA1 at the head of ZUUL_REF
James E. Blair81515ad2012-10-01 18:29:08 -070079**ZUUL_PROJECT**
80 The project that triggered this build
81**ZUUL_PIPELINE**
82 The Zuul pipeline that is building this job
James E. Blaircdd00072012-06-08 19:17:28 -070083
James E. Blair1f4c2bb2013-04-26 08:40:46 -070084The following additional parameters will only be provided for builds
85associated with changes (i.e., in response to patchset-created or
86comment-added events):
James E. Blaircdd00072012-06-08 19:17:28 -070087
James E. Blair81515ad2012-10-01 18:29:08 -070088**ZUUL_BRANCH**
89 The target branch for the change that triggered this build
90**ZUUL_CHANGE**
91 The Gerrit change ID for the change that triggered this build
92**ZUUL_CHANGE_IDS**
93 All of the Gerrit change IDs that are included in this build (useful
94 when the DependentPipelineManager combines changes for testing)
95**ZUUL_PATCHSET**
96 The Gerrit patchset number for the change that triggered this build
James E. Blaircdd00072012-06-08 19:17:28 -070097
James E. Blair1f4c2bb2013-04-26 08:40:46 -070098The following additional parameters will only be provided for
James E. Blair81515ad2012-10-01 18:29:08 -070099post-merge (ref-updated) builds:
James E. Blaircdd00072012-06-08 19:17:28 -0700100
James E. Blair81515ad2012-10-01 18:29:08 -0700101**ZUUL_OLDREV**
102 The SHA1 of the old revision at this ref (recall the ref name is
103 in ZUUL_REF)
104**ZUUL_NEWREV**
105 The SHA1 of the new revision at this ref (recall the ref name is
106 in ZUUL_REF)
107**ZUUL_SHORT_OLDREV**
108 The shortened (7 character) SHA1 of the old revision
109**ZUUL_SHORT_NEWREV**
110 The shortened (7 character) SHA1 of the new revision
James E. Blaircdd00072012-06-08 19:17:28 -0700111
James E. Blair81515ad2012-10-01 18:29:08 -0700112In order to test the correct build, configure the Jenkins Git SCM
113plugin as follows::
James E. Blaircdd00072012-06-08 19:17:28 -0700114
James E. Blair81515ad2012-10-01 18:29:08 -0700115 Source Code Management:
116 Git
117 Repositories:
118 Repository URL: <your Gerrit or Zuul repository URL>
119 Advanced:
120 Refspec: ${ZUUL_REF}
121 Branches to build:
122 Branch Specifier: ${ZUUL_COMMIT}
James E. Blaire2819012013-06-28 17:17:26 -0400123 Advanced:
124 Clean after checkout: True
James E. Blaircdd00072012-06-08 19:17:28 -0700125
James E. Blair81515ad2012-10-01 18:29:08 -0700126That should be sufficient for a job that only builds a single project.
127If you have multiple interrelated projects (i.e., they share a Zuul
128Change Queue) that are built together, you may be able to configure
129the Git plugin to prepare them, or you may chose to use a shell script
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700130instead. As an example, the OpenStack project uses the following
131script to prepare the workspace for its integration testing:
James E. Blair81515ad2012-10-01 18:29:08 -0700132
K Jonathan Harker97e457e2013-02-26 13:29:38 -0800133 https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate-wrap.sh
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700134
135
136Zuul-Gearman Protocol
137---------------------
138
139This section is only relevant if you intend to implement a new kind of
140worker that runs jobs for Zuul via Gearman. If you just want to use
141Jenkins, see `Gearman Jenkins Plugin`_ instead.
142
143The Zuul protocol as used with Gearman is as follows:
144
145Starting Builds
146~~~~~~~~~~~~~~~
147
148To start a build, Zuul invokes a Gearman function with the following
149format:
150
151 build:FUNCTION_NAME
152
153where **FUNCTION_NAME** is the name of the job that should be run. If
154the job should run on a specific node (or class of node), Zuul will
155instead invoke:
156
157 build:FUNCTION_NAME:NODE_NAME
158
159where **NODE_NAME** is the name or class of node on which the job
160should be run. This can be specified by setting the ZUUL_NODE
161parameter in a paremeter-function (see :ref:`zuulconf`).
162
163Zuul sends the ZUUL_* parameters described in `Zuul Parameters`_
164encoded in JSON format as the argument included with the
165SUBMIT_JOB_UNIQ request to Gearman. A unique ID (equal to the
166ZUUL_UUID parameter) is also supplied to Gearman, and is accessible as
167an added Gearman parameter with GRAB_JOB_UNIQ.
168
169When a Gearman worker starts running a job for Zuul, it should
170immediately send a WORK_DATA packet with the following information
171encoded in JSON format:
172
James E. Blair3c483cf2013-06-04 16:30:43 -0700173**name**
174 The name of the job.
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700175
176**number**
177 The build number (unique to this job).
178
James E. Blair3c483cf2013-06-04 16:30:43 -0700179**manager**
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700180 A unique identifier associated with the Gearman worker that can
181 abort this build. See `Stopping Builds`_ for more information.
182
James E. Blair3c483cf2013-06-04 16:30:43 -0700183**url** (optional)
184 The URL with the status or results of the build. Will be used in
185 the status page and the final report.
186
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700187It should then immediately send a WORK_STATUS packet with a value of 0
188percent complete. It may then optionally send subsequent WORK_STATUS
189packets with updated completion values.
190
191When the build is complete, it should send a final WORK_DATA packet
192with the following in JSON format:
193
194**result**
195 Either the string 'SUCCESS' if the job succeeded, or any other value
196 that describes the result if the job failed.
197
198Finally, it should send either a WORK_FAIL or WORK_COMPLETE packet as
199appropriate. A WORK_EXCEPTION packet will be interpreted as a
200WORK_FAIL, but the exception will be logged in Zuul's error log.
201
202Stopping Builds
203~~~~~~~~~~~~~~~
204
205If Zuul needs to abort a build already in progress, it will invoke the
206following function through Gearman:
207
James E. Blair3c483cf2013-06-04 16:30:43 -0700208 stop:MANAGER_NAME
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700209
James E. Blair3c483cf2013-06-04 16:30:43 -0700210Where **MANAGER_NAME** is the name of the manager worker supplied in
211the initial WORK_DATA packet when the job started. This is used to
212direct the stop: function invocation to the correct Gearman worker
213that is capable of stopping that particular job. The argument to the
214function should be the following encoded in JSON format:
215
216**name**
217 The job name of the build to stop.
218
219**number**
220 The build number of the build to stop.
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700221
222The original job is expected to complete with a WORK_DATA and
223WORK_FAIL packet as described in `Starting Builds`_.
224
225Build Descriptions
226~~~~~~~~~~~~~~~~~~
227
228In order to update the job running system with a description of the
229current state of all related builds, the job runner may optionally
230implement the following Gearman function:
231
James E. Blair3c483cf2013-06-04 16:30:43 -0700232 set_description:MANAGER_NAME
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700233
James E. Blair3c483cf2013-06-04 16:30:43 -0700234Where **MANAGER_NAME** is used as described in `Stopping Builds`_.
235The argument to the function is the following encoded in JSON format:
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700236
James E. Blair3c483cf2013-06-04 16:30:43 -0700237**name**
238 The job name of the build to describe.
239
240**number**
241 The build number of the build to describe.
James E. Blair1f4c2bb2013-04-26 08:40:46 -0700242
243**html_description**
244 The description of the build in HTML format.