James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 1 | :title: Launchers |
| 2 | |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 3 | .. _Gearman: http://gearman.org/ |
Antoine Musso | d06f2a6 | 2012-11-16 17:40:58 +0100 | [diff] [blame] | 4 | |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 5 | .. _`Gearman Plugin`: |
| 6 | https://wiki.jenkins-ci.org/display/JENKINS/Gearman+Plugin |
| 7 | |
| 8 | .. _launchers: |
Antoine Musso | d06f2a6 | 2012-11-16 17:40:58 +0100 | [diff] [blame] | 9 | |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 10 | Launchers |
| 11 | ========= |
| 12 | |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 13 | Zuul has a modular architecture for launching jobs. Currently, the |
| 14 | only supported module interfaces with Gearman_. This design allows |
| 15 | any system to run jobs for Zuul simply by interfacing with a Gearman |
| 16 | server. The recommended way of integrating a new job-runner with Zuul |
| 17 | is via this method. |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 18 | |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 19 | If Gearman is unsuitable, Zuul may be extended with a new launcher |
| 20 | module. Zuul makes very few assumptions about the interface to a |
| 21 | launcher -- if it can trigger jobs, cancel them, and receive success |
| 22 | or failure reports, it should be able to be used with Zuul. Patches |
| 23 | to this effect are welcome. |
| 24 | |
| 25 | Gearman |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 26 | ------- |
| 27 | |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 28 | Gearman_ is a general-purpose protocol for distributing jobs to any |
| 29 | number of workers. Zuul works with Gearman by sending specific |
| 30 | information with job requests to Gearman, and expects certain |
| 31 | information to be returned on completion. This protocol is described |
| 32 | in `Zuul-Gearman Protocol`_. |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 33 | |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 34 | The `Gearman Jenkins Plugin`_ makes it easy to use Jenkins with Zuul |
| 35 | by providing an interface between Jenkins and Gearman. In this |
| 36 | configuration, Zuul asks Gearman to run jobs, and Gearman can then |
| 37 | distribute those jobs to any number of Jenkins systems (including |
| 38 | multiple Jenkins masters). |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 39 | |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 40 | In order for Zuul to run any jobs, you will need a running Gearman |
| 41 | server. The latest version of gearmand from gearman.org is required |
| 42 | in order to support canceling jobs while in the queue. The server is |
| 43 | easy to set up -- just make sure that it allows connections from Zuul |
| 44 | and any workers (e.g., Jenkins masters) on port 4730, and nowhere else |
| 45 | (as the Gearman protocol does not include any provision for |
| 46 | authentication. |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 47 | |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 48 | Gearman Jenkins Plugin |
| 49 | ---------------------- |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 50 | |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 51 | The `Gearman Plugin`_ can be installed in Jenkins in order to |
| 52 | facilitate Jenkins running jobs for Zuul. Install the plugin and |
| 53 | configure it with the hostname or IP address of your Gearman server |
| 54 | and the port on which it is listening (4730 by default). It will |
| 55 | automatically register all known Jenkins jobs as functions that Zuul |
| 56 | can invoke via Gearman. |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 57 | |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 58 | Any number of masters can be configured in this way, and Gearman will |
| 59 | distribute jobs to all of them as appropriate. |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 60 | |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 61 | No special Jenkins job configuration is needed to support triggering |
| 62 | by Zuul. |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 63 | |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 64 | Zuul Parameters |
| 65 | --------------- |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 66 | |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 67 | Zuul will pass some parameters with every job it launches. The |
| 68 | Gearman Plugin will ensure these are supplied as Jenkins build |
| 69 | parameters, so they will be available for use in the job configuration |
| 70 | as well as to the running job as environment variables. They are as |
| 71 | follows: |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 72 | |
James E. Blair | 81515ad | 2012-10-01 18:29:08 -0700 | [diff] [blame] | 73 | **ZUUL_UUID** |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 74 | Zuul provided key to link builds with Gerrit events |
James E. Blair | 81515ad | 2012-10-01 18:29:08 -0700 | [diff] [blame] | 75 | **ZUUL_REF** |
| 76 | Zuul provided ref that includes commit(s) to build |
| 77 | **ZUUL_COMMIT** |
| 78 | The commit SHA1 at the head of ZUUL_REF |
James E. Blair | 81515ad | 2012-10-01 18:29:08 -0700 | [diff] [blame] | 79 | **ZUUL_PROJECT** |
| 80 | The project that triggered this build |
| 81 | **ZUUL_PIPELINE** |
| 82 | The Zuul pipeline that is building this job |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 83 | |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 84 | The following additional parameters will only be provided for builds |
| 85 | associated with changes (i.e., in response to patchset-created or |
| 86 | comment-added events): |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 87 | |
James E. Blair | 81515ad | 2012-10-01 18:29:08 -0700 | [diff] [blame] | 88 | **ZUUL_BRANCH** |
| 89 | The target branch for the change that triggered this build |
| 90 | **ZUUL_CHANGE** |
| 91 | The Gerrit change ID for the change that triggered this build |
| 92 | **ZUUL_CHANGE_IDS** |
| 93 | All of the Gerrit change IDs that are included in this build (useful |
| 94 | when the DependentPipelineManager combines changes for testing) |
| 95 | **ZUUL_PATCHSET** |
| 96 | The Gerrit patchset number for the change that triggered this build |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 97 | |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 98 | The following additional parameters will only be provided for |
James E. Blair | 81515ad | 2012-10-01 18:29:08 -0700 | [diff] [blame] | 99 | post-merge (ref-updated) builds: |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 100 | |
James E. Blair | 81515ad | 2012-10-01 18:29:08 -0700 | [diff] [blame] | 101 | **ZUUL_OLDREV** |
| 102 | The SHA1 of the old revision at this ref (recall the ref name is |
| 103 | in ZUUL_REF) |
| 104 | **ZUUL_NEWREV** |
| 105 | The SHA1 of the new revision at this ref (recall the ref name is |
| 106 | in ZUUL_REF) |
| 107 | **ZUUL_SHORT_OLDREV** |
| 108 | The shortened (7 character) SHA1 of the old revision |
| 109 | **ZUUL_SHORT_NEWREV** |
| 110 | The shortened (7 character) SHA1 of the new revision |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 111 | |
James E. Blair | 81515ad | 2012-10-01 18:29:08 -0700 | [diff] [blame] | 112 | In order to test the correct build, configure the Jenkins Git SCM |
| 113 | plugin as follows:: |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 114 | |
James E. Blair | 81515ad | 2012-10-01 18:29:08 -0700 | [diff] [blame] | 115 | Source Code Management: |
| 116 | Git |
| 117 | Repositories: |
| 118 | Repository URL: <your Gerrit or Zuul repository URL> |
| 119 | Advanced: |
| 120 | Refspec: ${ZUUL_REF} |
| 121 | Branches to build: |
| 122 | Branch Specifier: ${ZUUL_COMMIT} |
James E. Blair | e281901 | 2013-06-28 17:17:26 -0400 | [diff] [blame^] | 123 | Advanced: |
| 124 | Clean after checkout: True |
James E. Blair | cdd0007 | 2012-06-08 19:17:28 -0700 | [diff] [blame] | 125 | |
James E. Blair | 81515ad | 2012-10-01 18:29:08 -0700 | [diff] [blame] | 126 | That should be sufficient for a job that only builds a single project. |
| 127 | If you have multiple interrelated projects (i.e., they share a Zuul |
| 128 | Change Queue) that are built together, you may be able to configure |
| 129 | the Git plugin to prepare them, or you may chose to use a shell script |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 130 | instead. As an example, the OpenStack project uses the following |
| 131 | script to prepare the workspace for its integration testing: |
James E. Blair | 81515ad | 2012-10-01 18:29:08 -0700 | [diff] [blame] | 132 | |
K Jonathan Harker | 97e457e | 2013-02-26 13:29:38 -0800 | [diff] [blame] | 133 | https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate-wrap.sh |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 134 | |
| 135 | |
| 136 | Zuul-Gearman Protocol |
| 137 | --------------------- |
| 138 | |
| 139 | This section is only relevant if you intend to implement a new kind of |
| 140 | worker that runs jobs for Zuul via Gearman. If you just want to use |
| 141 | Jenkins, see `Gearman Jenkins Plugin`_ instead. |
| 142 | |
| 143 | The Zuul protocol as used with Gearman is as follows: |
| 144 | |
| 145 | Starting Builds |
| 146 | ~~~~~~~~~~~~~~~ |
| 147 | |
| 148 | To start a build, Zuul invokes a Gearman function with the following |
| 149 | format: |
| 150 | |
| 151 | build:FUNCTION_NAME |
| 152 | |
| 153 | where **FUNCTION_NAME** is the name of the job that should be run. If |
| 154 | the job should run on a specific node (or class of node), Zuul will |
| 155 | instead invoke: |
| 156 | |
| 157 | build:FUNCTION_NAME:NODE_NAME |
| 158 | |
| 159 | where **NODE_NAME** is the name or class of node on which the job |
| 160 | should be run. This can be specified by setting the ZUUL_NODE |
| 161 | parameter in a paremeter-function (see :ref:`zuulconf`). |
| 162 | |
| 163 | Zuul sends the ZUUL_* parameters described in `Zuul Parameters`_ |
| 164 | encoded in JSON format as the argument included with the |
| 165 | SUBMIT_JOB_UNIQ request to Gearman. A unique ID (equal to the |
| 166 | ZUUL_UUID parameter) is also supplied to Gearman, and is accessible as |
| 167 | an added Gearman parameter with GRAB_JOB_UNIQ. |
| 168 | |
| 169 | When a Gearman worker starts running a job for Zuul, it should |
| 170 | immediately send a WORK_DATA packet with the following information |
| 171 | encoded in JSON format: |
| 172 | |
James E. Blair | 3c483cf | 2013-06-04 16:30:43 -0700 | [diff] [blame] | 173 | **name** |
| 174 | The name of the job. |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 175 | |
| 176 | **number** |
| 177 | The build number (unique to this job). |
| 178 | |
James E. Blair | 3c483cf | 2013-06-04 16:30:43 -0700 | [diff] [blame] | 179 | **manager** |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 180 | A unique identifier associated with the Gearman worker that can |
| 181 | abort this build. See `Stopping Builds`_ for more information. |
| 182 | |
James E. Blair | 3c483cf | 2013-06-04 16:30:43 -0700 | [diff] [blame] | 183 | **url** (optional) |
| 184 | The URL with the status or results of the build. Will be used in |
| 185 | the status page and the final report. |
| 186 | |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 187 | It should then immediately send a WORK_STATUS packet with a value of 0 |
| 188 | percent complete. It may then optionally send subsequent WORK_STATUS |
| 189 | packets with updated completion values. |
| 190 | |
| 191 | When the build is complete, it should send a final WORK_DATA packet |
| 192 | with the following in JSON format: |
| 193 | |
| 194 | **result** |
| 195 | Either the string 'SUCCESS' if the job succeeded, or any other value |
| 196 | that describes the result if the job failed. |
| 197 | |
| 198 | Finally, it should send either a WORK_FAIL or WORK_COMPLETE packet as |
| 199 | appropriate. A WORK_EXCEPTION packet will be interpreted as a |
| 200 | WORK_FAIL, but the exception will be logged in Zuul's error log. |
| 201 | |
| 202 | Stopping Builds |
| 203 | ~~~~~~~~~~~~~~~ |
| 204 | |
| 205 | If Zuul needs to abort a build already in progress, it will invoke the |
| 206 | following function through Gearman: |
| 207 | |
James E. Blair | 3c483cf | 2013-06-04 16:30:43 -0700 | [diff] [blame] | 208 | stop:MANAGER_NAME |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 209 | |
James E. Blair | 3c483cf | 2013-06-04 16:30:43 -0700 | [diff] [blame] | 210 | Where **MANAGER_NAME** is the name of the manager worker supplied in |
| 211 | the initial WORK_DATA packet when the job started. This is used to |
| 212 | direct the stop: function invocation to the correct Gearman worker |
| 213 | that is capable of stopping that particular job. The argument to the |
| 214 | function should be the following encoded in JSON format: |
| 215 | |
| 216 | **name** |
| 217 | The job name of the build to stop. |
| 218 | |
| 219 | **number** |
| 220 | The build number of the build to stop. |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 221 | |
| 222 | The original job is expected to complete with a WORK_DATA and |
| 223 | WORK_FAIL packet as described in `Starting Builds`_. |
| 224 | |
| 225 | Build Descriptions |
| 226 | ~~~~~~~~~~~~~~~~~~ |
| 227 | |
| 228 | In order to update the job running system with a description of the |
| 229 | current state of all related builds, the job runner may optionally |
| 230 | implement the following Gearman function: |
| 231 | |
James E. Blair | 3c483cf | 2013-06-04 16:30:43 -0700 | [diff] [blame] | 232 | set_description:MANAGER_NAME |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 233 | |
James E. Blair | 3c483cf | 2013-06-04 16:30:43 -0700 | [diff] [blame] | 234 | Where **MANAGER_NAME** is used as described in `Stopping Builds`_. |
| 235 | The argument to the function is the following encoded in JSON format: |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 236 | |
James E. Blair | 3c483cf | 2013-06-04 16:30:43 -0700 | [diff] [blame] | 237 | **name** |
| 238 | The job name of the build to describe. |
| 239 | |
| 240 | **number** |
| 241 | The build number of the build to describe. |
James E. Blair | 1f4c2bb | 2013-04-26 08:40:46 -0700 | [diff] [blame] | 242 | |
| 243 | **html_description** |
| 244 | The description of the build in HTML format. |