blob: b20aba72241c1957ce001e9b22b5e2969317528a [file] [log] [blame]
James E. Blaireff5a9d2017-06-20 00:00:37 -07001:title: Components
2
3.. _components:
4
5Components
6==========
7
8Zuul is a distributed system consisting of several components, each of
James E. Blaire86abe12017-08-04 14:46:00 -07009which is described below.
10
11
12.. graphviz::
13 :align: center
14
15 graph {
16 node [shape=box]
17 Gearman [shape=ellipse]
18 Gerrit [fontcolor=grey]
19 Zookeeper [shape=ellipse]
20 Nodepool
21 GitHub [fontcolor=grey]
22
23 Merger -- Gearman
24 Executor -- Gearman
25 Web -- Gearman
26
27 Gearman -- Scheduler;
28 Scheduler -- Gerrit;
29 Scheduler -- Zookeeper;
30 Zookeeper -- Nodepool;
31 Scheduler -- GitHub;
32 }
33
34
35
36All Zuul processes read the ``/etc/zuul/zuul.conf`` file (an alternate
37location may be supplied on the command line) which uses an INI file
38syntax. Each component may have its own configuration file, though
39you may find it simpler to use the same file for all components.
James E. Blaireff5a9d2017-06-20 00:00:37 -070040
James E. Blair97940192017-08-03 14:47:47 -070041An example ``zuul.conf``:
42
43.. code-block:: ini
44
45 [gearman]
46 server=localhost
47
48 [gearman_server]
49 start=true
50 log_config=/etc/zuul/gearman-logging.yaml
51
52 [zookeeper]
53 hosts=zk1.example.com,zk2.example.com,zk3.example.com
54
55 [webapp]
56 status_url=https://zuul.example.com/status
57
58 [scheduler]
59 log_config=/etc/zuul/scheduler-logging.yaml
60
61A minimal Zuul system may consist of a :ref:`scheduler` and
62:ref:`executor` both running on the same host. Larger installations
63should consider running multiple executors, each on a dedicated host,
64and running mergers on dedicated hosts as well.
James E. Blaireff5a9d2017-06-20 00:00:37 -070065
66Common
67------
68
69The following applies to all Zuul components.
70
71Configuration
72~~~~~~~~~~~~~
73
James E. Blair97940192017-08-03 14:47:47 -070074The following sections of ``zuul.conf`` are used by all Zuul components:
James E. Blaireff5a9d2017-06-20 00:00:37 -070075
James E. Blaireff5a9d2017-06-20 00:00:37 -070076
James E. Blair97940192017-08-03 14:47:47 -070077.. attr:: gearman
James E. Blaireff5a9d2017-06-20 00:00:37 -070078
James E. Blair97940192017-08-03 14:47:47 -070079 Client connection information for Gearman.
David Shrewsbury6b3b49b2017-07-11 13:00:38 -040080
James E. Blair97940192017-08-03 14:47:47 -070081 .. attr:: server
82 :required:
James E. Blaireff5a9d2017-06-20 00:00:37 -070083
James E. Blair97940192017-08-03 14:47:47 -070084 Hostname or IP address of the Gearman server.
David Shrewsbury6b3b49b2017-07-11 13:00:38 -040085
James E. Blair97940192017-08-03 14:47:47 -070086 .. attr:: port
87 :default: 4730
James E. Blaireff5a9d2017-06-20 00:00:37 -070088
James E. Blair97940192017-08-03 14:47:47 -070089 Port on which the Gearman server is listening.
James E. Blaireff5a9d2017-06-20 00:00:37 -070090
James E. Blair97940192017-08-03 14:47:47 -070091 .. attr:: ssl_ca
James E. Blaireff5a9d2017-06-20 00:00:37 -070092
James E. Blair97940192017-08-03 14:47:47 -070093 An openssl file containing a set of concatenated “certification
94 authority” certificates in PEM formet.
James E. Blaireff5a9d2017-06-20 00:00:37 -070095
James E. Blair97940192017-08-03 14:47:47 -070096 .. attr:: ssl_cert
97
98 An openssl file containing the client public certificate in PEM format.
99
100 .. attr:: ssl_key
101
102 An openssl file containing the client private key in PEM format.
James E. Blairfdb111d2017-06-23 20:56:05 +0100103
James E. Blairded241e2017-10-10 13:22:40 -0700104.. attr:: statsd
105
106 Information about the optional statsd server. If the ``statsd``
107 python module is installed and this section is configured,
108 statistics will be reported to statsd. See :ref:`statsd` for more
109 information.
110
111 .. attr:: server
112
113 Hostname or IP address of the statsd server.
114
115 .. attr:: port
116 :default: 8125
117
118 The UDP port on which the statsd server is listening.
119
120 .. attr:: prefix
121
122 If present, this will be prefixed to all of the keys before
123 transmitting to the statsd server.
124
James E. Blair4f3e6222017-07-05 14:52:08 -0700125.. NOTE: this is a white lie at this point, since only the scheduler
126 uses this, however, we expect other components to use it later, so
127 it's reasonable for admins to plan for this now.
128
James E. Blair97940192017-08-03 14:47:47 -0700129.. attr:: zookeeper
David Shrewsbury6b3b49b2017-07-11 13:00:38 -0400130
James E. Blair97940192017-08-03 14:47:47 -0700131 Client connection information for ZooKeeper
James E. Blairfdb111d2017-06-23 20:56:05 +0100132
James E. Blair97940192017-08-03 14:47:47 -0700133 .. attr:: hosts
134 :required:
135
136 A list of zookeeper hosts for Zuul to use when communicating
137 with Nodepool.
138
James E. Blaire2f0a872017-09-28 10:35:12 -0700139 .. attr:: session_timeout
140 :default: 10.0
141
142 The ZooKeeper session timeout, in seconds.
143
James E. Blair97940192017-08-03 14:47:47 -0700144
145.. _scheduler:
James E. Blaireff5a9d2017-06-20 00:00:37 -0700146
147Scheduler
148---------
149
150The scheduler is the primary component of Zuul. The scheduler is not
151a scalable component; one, and only one, scheduler must be running at
152all times for Zuul to be operational. It receives events from any
153connections to remote systems which have been configured, enqueues
154items into pipelines, distributes jobs to executors, and reports
155results.
156
157Configuration
158~~~~~~~~~~~~~
159
James E. Blair97940192017-08-03 14:47:47 -0700160The following sections of ``zuul.conf`` are used by the scheduler:
James E. Blaireff5a9d2017-06-20 00:00:37 -0700161
James E. Blaireff5a9d2017-06-20 00:00:37 -0700162
James E. Blair97940192017-08-03 14:47:47 -0700163.. attr:: gearman_server
James E. Blaireff5a9d2017-06-20 00:00:37 -0700164
James E. Blair97940192017-08-03 14:47:47 -0700165 The builtin gearman server. Zuul can fork a gearman process from
166 itself rather than connecting to an external one.
David Shrewsbury6b3b49b2017-07-11 13:00:38 -0400167
James E. Blair97940192017-08-03 14:47:47 -0700168 .. attr:: start
169 :default: false
James E. Blaireff5a9d2017-06-20 00:00:37 -0700170
James E. Blair97940192017-08-03 14:47:47 -0700171 Whether to start the internal Gearman server.
David Shrewsbury6b3b49b2017-07-11 13:00:38 -0400172
James E. Blair97940192017-08-03 14:47:47 -0700173 .. attr:: listen_address
174 :default: all addresses
James E. Blaireff5a9d2017-06-20 00:00:37 -0700175
James E. Blair97940192017-08-03 14:47:47 -0700176 IP address or domain name on which to listen.
David Shrewsbury6b3b49b2017-07-11 13:00:38 -0400177
Tristan Cacqueraya7586c92017-08-29 10:10:08 +0000178 .. attr:: port
179 :default: 4730
180
181 TCP port on which to listen.
182
James E. Blair97940192017-08-03 14:47:47 -0700183 .. attr:: log_config
James E. Blaireff5a9d2017-06-20 00:00:37 -0700184
James E. Blair97940192017-08-03 14:47:47 -0700185 Path to log config file for internal Gearman server.
James E. Blaireff5a9d2017-06-20 00:00:37 -0700186
James E. Blair97940192017-08-03 14:47:47 -0700187 .. attr:: ssl_ca
James E. Blaireff5a9d2017-06-20 00:00:37 -0700188
James E. Blair97940192017-08-03 14:47:47 -0700189 An openssl file containing a set of concatenated “certification
190 authority” certificates in PEM formet.
James E. Blaireff5a9d2017-06-20 00:00:37 -0700191
James E. Blair97940192017-08-03 14:47:47 -0700192 .. attr:: ssl_cert
James E. Blaireff5a9d2017-06-20 00:00:37 -0700193
James E. Blair97940192017-08-03 14:47:47 -0700194 An openssl file containing the server public certificate in PEM
195 format.
David Shrewsbury6b3b49b2017-07-11 13:00:38 -0400196
James E. Blair97940192017-08-03 14:47:47 -0700197 .. attr:: ssl_key
James E. Blaireff5a9d2017-06-20 00:00:37 -0700198
James E. Blair97940192017-08-03 14:47:47 -0700199 An openssl file containing the server private key in PEM format.
David Shrewsbury6b3b49b2017-07-11 13:00:38 -0400200
James E. Blair97940192017-08-03 14:47:47 -0700201.. attr:: webapp
James E. Blaireff5a9d2017-06-20 00:00:37 -0700202
James E. Blair97940192017-08-03 14:47:47 -0700203 .. attr:: listen_address
204 :default: all addresses
David Shrewsbury6b3b49b2017-07-11 13:00:38 -0400205
James E. Blair97940192017-08-03 14:47:47 -0700206 IP address or domain name on which to listen.
James E. Blaireff5a9d2017-06-20 00:00:37 -0700207
James E. Blair97940192017-08-03 14:47:47 -0700208 .. attr:: port
209 :default: 8001
David Shrewsbury6b3b49b2017-07-11 13:00:38 -0400210
James E. Blair97940192017-08-03 14:47:47 -0700211 Port on which the webapp is listening.
James E. Blair4f3e6222017-07-05 14:52:08 -0700212
James E. Blair97940192017-08-03 14:47:47 -0700213 .. attr:: status_expiry
214 :default: 1
James E. Blaireff5a9d2017-06-20 00:00:37 -0700215
James E. Blair97940192017-08-03 14:47:47 -0700216 Zuul will cache the status.json file for this many seconds.
David Shrewsbury6b3b49b2017-07-11 13:00:38 -0400217
James E. Blair97940192017-08-03 14:47:47 -0700218 .. attr:: status_url
James E. Blaireff5a9d2017-06-20 00:00:37 -0700219
James E. Blair97940192017-08-03 14:47:47 -0700220 URL that will be posted in Zuul comments made to changes when
221 starting jobs for a change.
David Shrewsbury6b3b49b2017-07-11 13:00:38 -0400222
James E. Blair97940192017-08-03 14:47:47 -0700223 .. TODO: is this effectively required?
James E. Blaireff5a9d2017-06-20 00:00:37 -0700224
James E. Blair97940192017-08-03 14:47:47 -0700225.. attr:: scheduler
David Shrewsbury6b3b49b2017-07-11 13:00:38 -0400226
James E. Blair97940192017-08-03 14:47:47 -0700227 .. attr:: tenant_config
228 :required:
James E. Blaireff5a9d2017-06-20 00:00:37 -0700229
James E. Blair97940192017-08-03 14:47:47 -0700230 Path to :ref:`tenant-config` file.
David Shrewsbury6b3b49b2017-07-11 13:00:38 -0400231
James E. Blair97940192017-08-03 14:47:47 -0700232 .. attr:: log_config
233
234 Path to log config file.
235
236 .. attr:: pidfile
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700237 :default: /var/run/zuul-schedurecr/zuul-scheduler.pid
James E. Blair97940192017-08-03 14:47:47 -0700238
239 Path to PID lock file.
240
241 .. attr:: state_dir
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700242 :default: /var/lib/zuul
James E. Blair97940192017-08-03 14:47:47 -0700243
244 Path to directory in which Zuul should save its state.
James E. Blaireff5a9d2017-06-20 00:00:37 -0700245
246Operation
247~~~~~~~~~
248
249To start the scheduler, run ``zuul-scheduler``. To stop it, kill the
250PID which was saved in the pidfile specified in the configuration.
251
252Most of Zuul's configuration is automatically updated as changes to
253the repositories which contain it are merged. However, Zuul must be
254explicitly notified of changes to the tenant config file, since it is
255not read from a git repository. To do so, send the scheduler PID
James E. Blair97940192017-08-03 14:47:47 -0700256(saved in the pidfile specified in the configuration) a `SIGHUP`
257signal.
James E. Blaireff5a9d2017-06-20 00:00:37 -0700258
259Merger
260------
261
262Mergers are an optional Zuul service; they are not required for Zuul
263to operate, but some high volume sites may benefit from running them.
264Zuul performs quite a lot of git operations in the course of its work.
265Each change that is to be tested must be speculatively merged with the
266current state of its target branch to ensure that it can merge, and to
267ensure that the tests that Zuul perform accurately represent the
268outcome of merging the change. Because Zuul's configuration is stored
269in the git repos it interacts with, and is dynamically evaluated, Zuul
270often needs to perform a speculative merge in order to determine
271whether it needs to perform any further actions.
272
273All of these git operations add up, and while Zuul executors can also
274perform them, large numbers may impact their ability to run jobs.
275Therefore, administrators may wish to run standalone mergers in order
276to reduce the load on executors.
277
278Configuration
279~~~~~~~~~~~~~
280
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700281The following section of ``zuul.conf`` is used by the merger:
James E. Blaireff5a9d2017-06-20 00:00:37 -0700282
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700283.. attr:: merger
James E. Blaireff5a9d2017-06-20 00:00:37 -0700284
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700285 .. attr:: git_dir
David Shrewsbury6b3b49b2017-07-11 13:00:38 -0400286
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700287 Directory in which Zuul should clone git repositories.
James E. Blaireff5a9d2017-06-20 00:00:37 -0700288
Paul Belangeredadfed2017-10-05 16:04:27 -0400289 .. attr:: git_http_low_speed_limit
290 :default: 1000
291
292 If the HTTP transfer speed is less then git_http_low_speed_limit for
293 longer then git_http_low_speed_time, the transfer is aborted.
294
295 Value in bytes, setting to 0 will disable.
296
297 .. attr:: git_http_low_speed_time
298 :default: 30
299
300 If the HTTP transfer speed is less then git_http_low_speed_limit for
301 longer then git_http_low_speed_time, the transfer is aborted.
302
303 Value in seconds, setting to 0 will disable.
304
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700305 .. attr:: git_user_email
David Shrewsbury6b3b49b2017-07-11 13:00:38 -0400306
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700307 Value to pass to `git config user.email
308 <https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup>`_.
James E. Blaireff5a9d2017-06-20 00:00:37 -0700309
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700310 .. attr:: git_user_name
David Shrewsbury6b3b49b2017-07-11 13:00:38 -0400311
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700312 Value to pass to `git config user.name
313 <https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup>`_.
James E. Blaireff5a9d2017-06-20 00:00:37 -0700314
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700315 .. attr:: log_config
David Shrewsbury6b3b49b2017-07-11 13:00:38 -0400316
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700317 Path to log config file for the merger process.
James E. Blaireff5a9d2017-06-20 00:00:37 -0700318
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700319 .. attr:: pidfile
320 :default: /var/run/zuul-merger/zuul-merger.pid
David Shrewsbury6b3b49b2017-07-11 13:00:38 -0400321
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700322 Path to PID lock file for the merger process.
James E. Blaireff5a9d2017-06-20 00:00:37 -0700323
324Operation
325~~~~~~~~~
326
327To start the merger, run ``zuul-merger``. To stop it, kill the
328PID which was saved in the pidfile specified in the configuration.
329
David Shrewsburyb040b0a2017-08-03 15:53:59 -0400330.. _executor:
331
James E. Blaireff5a9d2017-06-20 00:00:37 -0700332Executor
333--------
334
335Executors are responsible for running jobs. At the start of each job,
336an executor prepares an environment in which to run Ansible which
337contains all of the git repositories specified by the job with all
338dependent changes merged into their appropriate branches. The branch
339corresponding to the proposed change will be checked out (in all
340projects, if it exists). Any roles specified by the job will also be
341present (also with dependent changes merged, if appropriate) and added
342to the Ansible role path. The executor also prepares an Ansible
343inventory file with all of the nodes requested by the job.
344
345The executor also contains a merger. This is used by the executor to
346prepare the git repositories used by jobs, but is also available to
347perform any tasks normally performed by standalone mergers. Because
348the executor performs both roles, small Zuul installations may not
349need to run standalone mergers.
350
351Trusted and Untrusted Playbooks
352~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
353
354The executor runs playbooks in one of two execution contexts depending
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700355on whether the project containing the playbook is a
356:term:`config-project` or an :term:`untrusted-project`. If the
357playbook is in a config project, the executor runs the playbook in the
358*trusted* execution context, otherwise, it is run in the *untrusted*
359execution context.
James E. Blaireff5a9d2017-06-20 00:00:37 -0700360
Clint Byrum48b5a672017-08-18 14:44:00 -0700361Both execution contexts use `bubblewrap`_ [#nullwrap]_ to create a
362namespace to ensure that playbook executions are isolated and are unable
363to access files outside of a restricted environment. The administrator
364may configure additional local directories on the executor to be made
James E. Blaireff5a9d2017-06-20 00:00:37 -0700365available to the restricted environment.
366
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700367The trusted execution context has access to all Ansible features,
James E. Blaireff5a9d2017-06-20 00:00:37 -0700368including the ability to load custom Ansible modules. Needless to
369say, extra scrutiny should be given to code that runs in a trusted
370context as it could be used to compromise other jobs running on the
371executor, or the executor itself, especially if the administrator has
372granted additional access through bubblewrap, or a method of escaping
373the restricted environment created by bubblewrap is found.
374
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700375Playbooks run in the untrusted execution context are not permitted to
376load additional Ansible modules or access files outside of the
James E. Blaireff5a9d2017-06-20 00:00:37 -0700377restricted environment prepared for them by the executor. In addition
378to the bubblewrap environment applied to both execution contexts, in
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700379the untrusted context some standard Ansible modules are replaced with
380versions which prohibit some actions, including attempts to access
381files outside of the restricted execution context. These redundant
382protections are made as part of a defense-in-depth strategy.
James E. Blaireff5a9d2017-06-20 00:00:37 -0700383
384.. _bubblewrap: https://github.com/projectatomic/bubblewrap
Clint Byrum48b5a672017-08-18 14:44:00 -0700385.. [#nullwrap] Unless one has set execution_wrapper to nullwrap in the
386 executor configuration.
James E. Blaireff5a9d2017-06-20 00:00:37 -0700387
388Configuration
389~~~~~~~~~~~~~
390
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700391The following sections of ``zuul.conf`` are used by the executor:
James E. Blaireff5a9d2017-06-20 00:00:37 -0700392
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700393.. attr:: executor
James E. Blaireff5a9d2017-06-20 00:00:37 -0700394
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700395 .. attr:: finger_port
396 :default: 79
David Shrewsbury6b3b49b2017-07-11 13:00:38 -0400397
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700398 Port to use for finger log streamer.
James E. Blaireff5a9d2017-06-20 00:00:37 -0700399
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700400 .. attr:: git_dir
401 :default: /var/lib/zuul/git
James E. Blair7e6e0a12017-07-25 11:04:42 -0700402
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700403 Directory that Zuul should clone local git repositories to. The
404 executor keeps a local copy of every git repository it works
405 with to speed operations and perform speculative merging.
David Shrewsbury6b3b49b2017-07-11 13:00:38 -0400406
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700407 This should be on the same filesystem as
408 :attr:`executor.job_dir` so that when git repos are cloned into
409 the job workspaces, they can be hard-linked to the local git
410 cache.
James E. Blaireff5a9d2017-06-20 00:00:37 -0700411
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700412 .. attr:: job_dir
413 :default: /tmp
James E. Blair7e6e0a12017-07-25 11:04:42 -0700414
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700415 Directory that Zuul should use to hold temporary job directories.
416 When each job is run, a new entry will be created under this
417 directory to hold the configuration and scratch workspace for
418 that job. It will be deleted at the end of the job (unless the
419 `--keep-jobdir` command line option is specified).
James E. Blair7e6e0a12017-07-25 11:04:42 -0700420
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700421 This should be on the same filesystem as :attr:`executor.git_dir`
422 so that when git repos are cloned into the job workspaces, they
423 can be hard-linked to the local git cache.
James E. Blair7e6e0a12017-07-25 11:04:42 -0700424
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700425 .. attr:: log_config
David Shrewsbury6b3b49b2017-07-11 13:00:38 -0400426
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700427 Path to log config file for the executor process.
James E. Blaireff5a9d2017-06-20 00:00:37 -0700428
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700429 .. attr:: pidfile
430 :default: /var/run/zuul-executor/zuul-executor.pid
David Shrewsbury6b3b49b2017-07-11 13:00:38 -0400431
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700432 Path to PID lock file for the executor process.
James E. Blaireff5a9d2017-06-20 00:00:37 -0700433
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700434 .. attr:: private_key_file
435 :default: ~/.ssh/id_rsa
David Shrewsbury6b3b49b2017-07-11 13:00:38 -0400436
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700437 SSH private key file to be used when logging into worker nodes.
James E. Blaireff5a9d2017-06-20 00:00:37 -0700438
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700439 .. attr:: user
440 :default: zuul
Jamie Lennox7655b552017-03-17 12:33:38 +1100441
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700442 User ID for the zuul-executor process. In normal operation as a
443 daemon, the executor should be started as the ``root`` user, but
444 it will drop privileges to this user during startup.
Jamie Lennox7655b552017-03-17 12:33:38 +1100445
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700446 .. _admin_sitewide_variables:
Jamie Lennox7655b552017-03-17 12:33:38 +1100447
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700448 .. attr:: variables
Jamie Lennox7655b552017-03-17 12:33:38 +1100449
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700450 Path to an Ansible variables file to supply site-wide variables.
451 This should be a YAML-formatted file consisting of a single
452 dictionary. The contents will be made available to all jobs as
453 Ansible variables. These variables take precedence over all
454 other forms (job variables and secrets). Care should be taken
455 when naming these variables to avoid potential collisions with
456 those used by jobs. Prefixing variable names with a
457 site-specific identifier is recommended. The default is not to
458 add any site-wide variables. See the :ref:`User's Guide
459 <user_sitewide_variables>` for more information.
Clint Byrumdc8a0902017-07-20 16:36:27 -0700460
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700461 .. attr:: disk_limit_per_job
462 :default: 250
Clint Byrumdc8a0902017-07-20 16:36:27 -0700463
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700464 This integer is the maximum number of megabytes that any one job
465 is allowed to consume on disk while it is running. If a job's
466 scratch space has more than this much space consumed, it will be
467 aborted.
Monty Taylor01380dd2017-07-28 16:01:20 -0500468
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700469 .. attr:: trusted_ro_paths
Monty Taylor01380dd2017-07-28 16:01:20 -0500470
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700471 List of paths, separated by ``:`` to read-only bind mount into
472 trusted bubblewrap contexts.
Monty Taylor01380dd2017-07-28 16:01:20 -0500473
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700474 .. attr:: trusted_rw_paths
Monty Taylor01380dd2017-07-28 16:01:20 -0500475
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700476 List of paths, separated by ``:`` to read-write bind mount into
477 trusted bubblewrap contexts.
Monty Taylor01380dd2017-07-28 16:01:20 -0500478
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700479 .. attr:: untrusted_ro_paths
Monty Taylor01380dd2017-07-28 16:01:20 -0500480
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700481 List of paths, separated by ``:`` to read-only bind mount into
482 untrusted bubblewrap contexts.
Monty Taylor01380dd2017-07-28 16:01:20 -0500483
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700484 .. attr:: untrusted_rw_paths
Monty Taylor01380dd2017-07-28 16:01:20 -0500485
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700486 List of paths, separated by ``:`` to read-write bind mount into
487 untrusted bubblewrap contexts.
James E. Blaireff5a9d2017-06-20 00:00:37 -0700488
Clint Byrum48b5a672017-08-18 14:44:00 -0700489 .. attr:: execution_wrapper
490 :default: bubblewrap
491
492 Name of the execution wrapper to use when executing
493 `ansible-playbook`. The default, `bubblewrap` is recommended for
494 all installations.
495
496 There is also a `nullwrap` driver for situations where one wants
497 to run Zuul without access to bubblewrap or in such a way that
498 bubblewrap may interfere with the jobs themselves. However,
499 `nullwrap` is considered unsafe, as `bubblewrap` provides
500 significant protections against malicious users and accidental
501 breakage in playbooks. As such, `nullwrap` is not recommended
502 for use in production.
503
504 This option, and thus, `nullwrap`, may be removed in the future.
505 `bubblewrap` has become integral to securely operating Zuul. If you
506 have a valid use case for it, we encourage you to let us know.
507
Clint Byrum3cef7ed2017-09-29 20:18:54 -0700508 .. attr:: load_multiplier
509 :default: 2.5
510
511 When an executor host gets too busy, the system may suffer
512 timeouts and other ill effects. The executor will stop accepting
513 more than 1 job at a time until load has lowered below a safe
514 level. This level is determined by multiplying the number of
515 CPU's by `load_multiplier`.
516
517 So for example, if the system has 2 CPUs, and load_multiplier
518 is 2.5, the safe load for the system is 5.00. Any time the
519 system load average is over 5.00, the executor will quit
520 accepting multiple jobs at one time.
521
522 The executor will observe system load and determine whether
523 to accept more jobs every 30 seconds.
524
Tobias Henkel055cda32017-10-17 13:08:18 +0200525 .. attr:: hostname
526 :default: hostname of the server
527
528 The executor needs to know its hostname under which it is reachable by
529 zuul-web. Otherwise live console log streaming doesn't work. In most cases
530 This is automatically detected correctly. But when running in environments
531 where it cannot determine its hostname correctly this can be overridden
532 here.
533
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700534.. attr:: merger
David Shrewsbury6b3b49b2017-07-11 13:00:38 -0400535
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700536 .. attr:: git_user_email
James E. Blaireff5a9d2017-06-20 00:00:37 -0700537
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700538 Value to pass to `git config user.email
539 <https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup>`_.
David Shrewsbury6b3b49b2017-07-11 13:00:38 -0400540
James E. Blair9d0b4cc2017-08-03 15:08:47 -0700541 .. attr:: git_user_name
542
543 Value to pass to `git config user.name
544 <https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup>`_.
James E. Blaireff5a9d2017-06-20 00:00:37 -0700545
546Operation
547~~~~~~~~~
548
549To start the executor, run ``zuul-executor``.
550
551There are several commands which can be run to control the executor's
552behavior once it is running.
553
554To stop the executor immediately, aborting all jobs (they may be
555relaunched according to their retry policy), run ``zuul-executor
556stop``.
557
558To request that the executor stop executing new jobs and exit when all
559currently running jobs have completed, run ``zuul-executor graceful``.
560
James E. Blairafe27e62017-08-03 15:11:47 -0700561To enable or disable running Ansible in verbose mode (with the
562``-vvv`` argument to ansible-playbook) run ``zuul-executor verbose``
563and ``zuul-executor unverbose``.
David Shrewsburyae734d12017-07-11 12:41:59 -0400564
565Web Server
566----------
567
568The Zuul web server currently acts as a websocket interface to live log
569streaming. Eventually, it will serve as the single process handling all
570HTTP interactions with Zuul.
571
572Configuration
573~~~~~~~~~~~~~
574
James E. Blairafe27e62017-08-03 15:11:47 -0700575In addition to the common configuration sections, the following
576sections of ``zuul.conf`` are used by the web server:
David Shrewsburyae734d12017-07-11 12:41:59 -0400577
James E. Blairafe27e62017-08-03 15:11:47 -0700578.. attr:: web
David Shrewsburyae734d12017-07-11 12:41:59 -0400579
James E. Blairafe27e62017-08-03 15:11:47 -0700580 .. attr:: listen_address
581 :default: 127.0.0.1
David Shrewsbury6b3b49b2017-07-11 13:00:38 -0400582
James E. Blairafe27e62017-08-03 15:11:47 -0700583 IP address or domain name on which to listen.
David Shrewsburyae734d12017-07-11 12:41:59 -0400584
James E. Blairafe27e62017-08-03 15:11:47 -0700585 .. attr:: log_config
David Shrewsbury6b3b49b2017-07-11 13:00:38 -0400586
James E. Blairafe27e62017-08-03 15:11:47 -0700587 Path to log config file for the web server process.
David Shrewsburyae734d12017-07-11 12:41:59 -0400588
James E. Blairafe27e62017-08-03 15:11:47 -0700589 .. attr:: pidfile
590 :default: /var/run/zuul-web/zuul-web.pid
David Shrewsbury6b3b49b2017-07-11 13:00:38 -0400591
James E. Blairafe27e62017-08-03 15:11:47 -0700592 Path to PID lock file for the web server process.
David Shrewsburyae734d12017-07-11 12:41:59 -0400593
James E. Blairafe27e62017-08-03 15:11:47 -0700594 .. attr:: port
595 :default: 9000
David Shrewsbury6b3b49b2017-07-11 13:00:38 -0400596
James E. Blairafe27e62017-08-03 15:11:47 -0700597 Port to use for web server process.
David Shrewsburyae734d12017-07-11 12:41:59 -0400598
James E. Blairafe27e62017-08-03 15:11:47 -0700599 .. attr:: websocket_url
600
601 Base URL on which the websocket service is exposed, if different
602 than the base URL of the web app.
Tobias Henkelb4407fc2017-07-07 13:52:56 +0200603
David Shrewsburyae734d12017-07-11 12:41:59 -0400604Operation
605~~~~~~~~~
606
607To start the web server, run ``zuul-web``. To stop it, kill the
608PID which was saved in the pidfile specified in the configuration.