Reorganize docs into user/admin guide
Refresh the user and admin guide for v3 changes, and reorganize into
a narrative structure which makes more sense for v3.
Change-Id: I4ac3b18d5ed33b0fea4e2ef0318b19bfc3447ccc
diff --git a/doc/source/admin-config.rst b/doc/source/admin-config.rst
deleted file mode 100644
index 03f742b..0000000
--- a/doc/source/admin-config.rst
+++ /dev/null
@@ -1,24 +0,0 @@
-:title: Administrative Configuration
-
-.. _admin-config:
-.. _drivers:
-
-Administrative Configuration
-============================
-
-TODO:
-
-* zuul.conf
-* connections
-
-* main.yaml
-* tenants
-* trusted/untrusted
-
-* drivers
-* gerrit
- * trigger config
- * reporter config
-* github
- * trigger config
- * reporter config
diff --git a/doc/source/client.rst b/doc/source/admin/client.rst
similarity index 100%
rename from doc/source/client.rst
rename to doc/source/admin/client.rst
diff --git a/doc/source/admin/components.rst b/doc/source/admin/components.rst
new file mode 100644
index 0000000..1c639d6
--- /dev/null
+++ b/doc/source/admin/components.rst
@@ -0,0 +1,312 @@
+:title: Components
+
+.. _components:
+
+Components
+==========
+
+Zuul is a distributed system consisting of several components, each of
+which is described below. All Zuul processes read the
+**/etc/zuul/zuul.conf** file (an alternate location may be supplied on
+the command line) which uses an INI file syntax. Each component may
+have its own configuration file, though you may find it simpler to use
+the same file for all components.
+
+A minimal Zuul system may consist of a *scheduler* and *executor* both
+running on the same host. Larger installations should consider
+running multiple executors, each on a dedicated host, and running
+mergers on dedicated hosts as well.
+
+Common
+------
+
+The following applies to all Zuul components.
+
+Configuration
+~~~~~~~~~~~~~
+
+The following sections of **zuul.conf** are used by all Zuul components:
+
+gearman
+"""""""
+
+Client connection information for gearman.
+
+**server**
+ Hostname or IP address of the Gearman server.
+ ``server=gearman.example.com`` (required)
+
+**port**
+ Port on which the Gearman server is listening.
+ ``port=4730`` (optional)
+
+**ssl_ca**
+ Optional: An openssl file containing a set of concatenated
+ “certification authority” certificates in PEM formet.
+
+**ssl_cert**
+ Optional: An openssl file containing the client public certificate in
+ PEM format.
+
+**ssl_key**
+ Optional: An openssl file containing the client private key in PEM format.
+
+
+Scheduler
+---------
+
+The scheduler is the primary component of Zuul. The scheduler is not
+a scalable component; one, and only one, scheduler must be running at
+all times for Zuul to be operational. It receives events from any
+connections to remote systems which have been configured, enqueues
+items into pipelines, distributes jobs to executors, and reports
+results.
+
+Configuration
+~~~~~~~~~~~~~
+
+The following sections of **zuul.conf** are used by the scheduler:
+
+gearman_server
+""""""""""""""
+
+The builtin gearman server. Zuul can fork a gearman process from itself rather
+than connecting to an external one.
+
+**start**
+ Whether to start the internal Gearman server (default: False).
+ ``start=true``
+
+**listen_address**
+ IP address or domain name on which to listen (default: all addresses).
+ ``listen_address=127.0.0.1``
+
+**log_config**
+ Path to log config file for internal Gearman server.
+ ``log_config=/etc/zuul/gearman-logging.yaml``
+
+**ssl_ca**
+ Optional: An openssl file containing a set of concatenated “certification authority” certificates
+ in PEM formet.
+
+**ssl_cert**
+ Optional: An openssl file containing the server public certificate in PEM format.
+
+**ssl_key**
+ Optional: An openssl file containing the server private key in PEM format.
+
+webapp
+""""""
+
+**listen_address**
+ IP address or domain name on which to listen (default: 0.0.0.0).
+ ``listen_address=127.0.0.1``
+
+**port**
+ Port on which the webapp is listening (default: 8001).
+ ``port=8008``
+
+.. TODO: move this to webapp (currently in 'zuul')
+
+**status_expiry**
+ Zuul will cache the status.json file for this many seconds. This is an
+ optional value and ``1`` is used by default.
+ ``status_expiry=1``
+
+scheduler
+"""""""""
+.. TODO: rename this to 'scheduler' (currently 'zuul') and update to match these docs
+
+**tenant_config**
+ Path to tenant config file.
+ ``layout_config=/etc/zuul/tenant.yaml``
+
+**log_config**
+ Path to log config file.
+ ``log_config=/etc/zuul/scheduler-logging.yaml``
+
+**pidfile**
+ Path to PID lock file.
+ ``pidfile=/var/run/zuul/scheduler.pid``
+
+**state_dir**
+ Path to directory that Zuul should save state to.
+ ``state_dir=/var/lib/zuul``
+
+Operation
+~~~~~~~~~
+
+To start the scheduler, run ``zuul-scheduler``. To stop it, kill the
+PID which was saved in the pidfile specified in the configuration.
+
+Most of Zuul's configuration is automatically updated as changes to
+the repositories which contain it are merged. However, Zuul must be
+explicitly notified of changes to the tenant config file, since it is
+not read from a git repository. To do so, send the scheduler PID
+(saved in the pidfile specified in the configuration) a SIGHUP signal.
+
+Merger
+------
+
+Mergers are an optional Zuul service; they are not required for Zuul
+to operate, but some high volume sites may benefit from running them.
+Zuul performs quite a lot of git operations in the course of its work.
+Each change that is to be tested must be speculatively merged with the
+current state of its target branch to ensure that it can merge, and to
+ensure that the tests that Zuul perform accurately represent the
+outcome of merging the change. Because Zuul's configuration is stored
+in the git repos it interacts with, and is dynamically evaluated, Zuul
+often needs to perform a speculative merge in order to determine
+whether it needs to perform any further actions.
+
+All of these git operations add up, and while Zuul executors can also
+perform them, large numbers may impact their ability to run jobs.
+Therefore, administrators may wish to run standalone mergers in order
+to reduce the load on executors.
+
+Configuration
+~~~~~~~~~~~~~
+
+The following section of **zuul.conf** is used by the merger:
+
+merger
+""""""
+
+**git_dir**
+ Directory that Zuul should clone local git repositories to.
+ ``git_dir=/var/lib/zuul/git``
+
+**git_user_email**
+ Optional: Value to pass to `git config user.email`.
+ ``git_user_email=zuul@example.com``
+
+**git_user_name**
+ Optional: Value to pass to `git config user.name`.
+ ``git_user_name=zuul``
+
+**log_config**
+ Path to log config file for the merger process.
+ ``log_config=/etc/zuul/logging.yaml``
+
+**pidfile**
+ Path to PID lock file for the merger process.
+ ``pidfile=/var/run/zuul-merger/merger.pid``
+
+Operation
+~~~~~~~~~
+
+To start the merger, run ``zuul-merger``. To stop it, kill the
+PID which was saved in the pidfile specified in the configuration.
+
+Executor
+--------
+
+Executors are responsible for running jobs. At the start of each job,
+an executor prepares an environment in which to run Ansible which
+contains all of the git repositories specified by the job with all
+dependent changes merged into their appropriate branches. The branch
+corresponding to the proposed change will be checked out (in all
+projects, if it exists). Any roles specified by the job will also be
+present (also with dependent changes merged, if appropriate) and added
+to the Ansible role path. The executor also prepares an Ansible
+inventory file with all of the nodes requested by the job.
+
+The executor also contains a merger. This is used by the executor to
+prepare the git repositories used by jobs, but is also available to
+perform any tasks normally performed by standalone mergers. Because
+the executor performs both roles, small Zuul installations may not
+need to run standalone mergers.
+
+Trusted and Untrusted Playbooks
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The executor runs playbooks in one of two execution contexts depending
+on whether the project containing the playbook is a *config project*
+or an *untrusted project*. If the playbook is in a *config project*,
+the executor runs the playbook in the *trusted* execution context,
+otherwise, it is run in the *untrusted* execution context.
+
+Both execution contexts use `bubblewrap`_ to create a namespace to
+ensure that playbook executions are isolated and are unable to access
+files outside of a restricted environment. The administrator may
+configure additional local directories on the executor to be made
+available to the restricted environment.
+
+The *trusted* execution context has access to all Ansible features,
+including the ability to load custom Ansible modules. Needless to
+say, extra scrutiny should be given to code that runs in a trusted
+context as it could be used to compromise other jobs running on the
+executor, or the executor itself, especially if the administrator has
+granted additional access through bubblewrap, or a method of escaping
+the restricted environment created by bubblewrap is found.
+
+Playbooks run in the *untrusted* execution context are not permitted
+to load additional Ansible modules or access files outside of the
+restricted environment prepared for them by the executor. In addition
+to the bubblewrap environment applied to both execution contexts, in
+the *untrusted* context some standard Ansible modules are replaced
+with versions which prohibit some actions, including attempts to
+access files outside of the restricted execution context. These
+redundant protections are made as part of a defense-in-depth strategy.
+
+.. _bubblewrap: https://github.com/projectatomic/bubblewrap
+
+Configuration
+~~~~~~~~~~~~~
+
+The following sections of **zuul.conf** are used by the executor:
+
+executor
+""""""""
+
+**finger_port**
+ Port to use for finger log streamer.
+ ``finger_port=79``
+
+**git_dir**
+ Directory that Zuul should clone local git repositories to.
+ ``git_dir=/var/lib/zuul/git``
+
+**log_config**
+ Path to log config file for the executor process.
+ ``log_config=/etc/zuul/logging.yaml``
+
+**private_key_file**
+ SSH private key file to be used when logging into worker nodes.
+ ``private_key_file=~/.ssh/id_rsa``
+
+**user**
+ User ID for the zuul-executor process. In normal operation as a daemon,
+ the executor should be started as the ``root`` user, but it will drop
+ privileges to this user during startup.
+ ``user=zuul``
+
+merger
+""""""
+
+**git_user_email**
+ Optional: Value to pass to `git config user.email`.
+ ``git_user_email=zuul@example.com``
+
+**git_user_name**
+ Optional: Value to pass to `git config user.name`.
+ ``git_user_name=zuul``
+
+Operation
+~~~~~~~~~
+
+To start the executor, run ``zuul-executor``.
+
+There are several commands which can be run to control the executor's
+behavior once it is running.
+
+To stop the executor immediately, aborting all jobs (they may be
+relaunched according to their retry policy), run ``zuul-executor
+stop``.
+
+To request that the executor stop executing new jobs and exit when all
+currently running jobs have completed, run ``zuul-executor graceful``.
+
+To enable or disable running Ansible in verbose mode (with the '-vvv'
+argument to ansible-playbook) run ``zuul-executor verbose`` and
+``zuul-executor unverbose``.
diff --git a/doc/source/admin/connections.rst b/doc/source/admin/connections.rst
new file mode 100644
index 0000000..5b40e5b
--- /dev/null
+++ b/doc/source/admin/connections.rst
@@ -0,0 +1,59 @@
+:title: Connection Configuration
+
+.. _connection-config:
+
+Connection Configuration
+========================
+
+Most of Zuul's configuration is contained in the git repositories upon
+which Zuul operates, however, some configuration outside of git
+repositories is still required to bootstrap the system. This includes
+information on connections between Zuul and other systems, as well as
+identifying the projects Zuul uses.
+
+.. _connections:
+
+Connections
+-----------
+
+In order to interact with external systems, Zuul must have a
+*connection* to that system configured. Zuul includes a number of
+drivers, each of which implements the functionality necessary to
+connect to a system. Each connection in Zuul is associated with a
+driver.
+
+To configure a connection in Zuul, select a unique name for the
+connection and add a section to **zuul.conf** with the form
+"[connection NAME]". For example, a connection to a gerrit server may
+appear as::
+
+ [connection mygerritserver]
+ driver=gerrit
+ server=review.example.com
+
+.. _drivers:
+
+Drivers
+-------
+
+Drivers may support any of the following functions:
+
+* Sources -- hosts git repositories for projects. Zuul can clone git
+ repos for projects and fetch refs.
+* Triggers -- emits events to which Zuul may respond. Triggers are
+ configured in pipelines to cause changes or other refs to be
+ enqueued.
+* Reporters -- outputs information when a pipeline is finished
+ processing an item.
+
+Zuul includes the following drivers:
+
+.. toctree::
+ :maxdepth: 2
+
+ drivers/gerrit
+ drivers/github
+ drivers/smtp
+ drivers/sql
+ drivers/timer
+ drivers/zuul
diff --git a/doc/source/admin/drivers/gerrit.rst b/doc/source/admin/drivers/gerrit.rst
new file mode 100644
index 0000000..6cd2f3d
--- /dev/null
+++ b/doc/source/admin/drivers/gerrit.rst
@@ -0,0 +1,168 @@
+:title: Gerrit Driver
+
+Gerrit
+======
+
+`Gerrit`_ is a code review system. The Gerrit driver supports
+sources, triggers, and reporters.
+
+.. _Gerrit: https://www.gerritcodereview.com/
+
+Zuul will need access to a Gerrit user.
+
+Create an SSH keypair for Zuul to use if there isn't one already, and
+create a Gerrit user with that key::
+
+ cat ~/id_rsa.pub | ssh -p29418 review.example.com gerrit create-account --ssh-key - --full-name Zuul zuul
+
+Give that user whatever permissions will be needed on the projects you
+want Zuul to report on. For instance, you may want to grant
+``Verified +/-1`` and ``Submit`` to the user. Additional categories
+or values may be added to Gerrit. Zuul is very flexible and can take
+advantage of those.
+
+Connection Configuration
+------------------------
+
+The supported options in zuul.conf connections are:
+
+**driver=gerrit**
+
+**server**
+ FQDN of Gerrit server.
+ ``server=review.example.com``
+
+**canonical_hostname**
+ The canonical hostname associated with the git repos on the Gerrit
+ server. Defaults to the value of **server**. This is used to
+ identify repos from this connection by name and in preparing repos
+ on the filesystem for use by jobs.
+ ``canonical_hostname=git.example.com``
+
+**port**
+ Optional: Gerrit server port.
+ ``port=29418``
+
+**baseurl**
+ Optional: path to Gerrit web interface. Defaults to ``https://<value
+ of server>/``. ``baseurl=https://review.example.com/review_site/``
+
+**user**
+ User name to use when logging into above server via ssh.
+ ``user=zuul``
+
+**sshkey**
+ Path to SSH key to use when logging into above server.
+ ``sshkey=/home/zuul/.ssh/id_rsa``
+
+**keepalive**
+ Optional: Keepalive timeout, 0 means no keepalive.
+ ``keepalive=60``
+
+Trigger Configuration
+---------------------
+
+Zuul works with standard versions of Gerrit by invoking the ``gerrit
+stream-events`` command over an SSH connection. It also reports back
+to Gerrit using SSH.
+
+If using Gerrit 2.7 or later, make sure the user is a member of a group
+that is granted the ``Stream Events`` permission, otherwise it will not
+be able to invoke the ``gerrit stream-events`` command over SSH.
+
+The supported pipeline trigger options are:
+
+**event**
+The event name from gerrit. Examples: ``patchset-created``,
+``comment-added``, ``ref-updated``. This field is treated as a
+regular expression.
+
+**branch**
+The branch associated with the event. Example: ``master``. This
+field is treated as a regular expression, and multiple branches may
+be listed.
+
+**ref**
+On ref-updated events, the branch parameter is not used, instead the
+ref is provided. Currently Gerrit has the somewhat idiosyncratic
+behavior of specifying bare refs for branch names (e.g., ``master``),
+but full ref names for other kinds of refs (e.g., ``refs/tags/foo``).
+Zuul matches what you put here exactly against what Gerrit
+provides. This field is treated as a regular expression, and
+multiple refs may be listed.
+
+**ignore-deletes**
+When a branch is deleted, a ref-updated event is emitted with a newrev
+of all zeros specified. The ``ignore-deletes`` field is a boolean value
+that describes whether or not these newrevs trigger ref-updated events.
+The default is True, which will not trigger ref-updated events.
+
+**approval**
+This is only used for ``comment-added`` events. It only matches if
+the event has a matching approval associated with it. Example:
+``code-review: 2`` matches a ``+2`` vote on the code review category.
+Multiple approvals may be listed.
+
+**email**
+This is used for any event. It takes a regex applied on the performer
+email, i.e. Gerrit account email address. If you want to specify
+several email filters, you must use a YAML list. Make sure to use non
+greedy matchers and to escapes dots!
+Example: ``email: ^.*?@example\.org$``.
+
+**email_filter** (deprecated)
+A deprecated alternate spelling of *email*. Only one of *email* or
+*email_filter* should be used.
+
+**username**
+This is used for any event. It takes a regex applied on the performer
+username, i.e. Gerrit account name. If you want to specify several
+username filters, you must use a YAML list. Make sure to use non greedy
+matchers and to escapes dots!
+Example: ``username: ^jenkins$``.
+
+**username_filter** (deprecated)
+A deprecated alternate spelling of *username*. Only one of *username* or
+*username_filter* should be used.
+
+**comment**
+This is only used for ``comment-added`` events. It accepts a list of
+regexes that are searched for in the comment string. If any of these
+regexes matches a portion of the comment string the trigger is
+matched. ``comment: retrigger`` will match when comments
+containing 'retrigger' somewhere in the comment text are added to a
+change.
+
+**comment_filter** (deprecated)
+A deprecated alternate spelling of *comment*. Only one of *comment* or
+*comment_filter* should be used.
+
+*require-approval*
+This may be used for any event. It requires that a certain kind
+of approval be present for the current patchset of the change (the
+approval could be added by the event in question). It follows the
+same syntax as the :ref:`"approval" pipeline requirement
+<pipeline-require-approval>`. For each specified criteria there must
+exist a matching approval.
+
+*reject-approval*
+This takes a list of approvals in the same format as
+*require-approval* but will fail to enter the pipeline if there is
+a matching approval.
+
+Reporter Configuration
+----------------------
+
+Zuul works with standard versions of Gerrit by invoking the
+``gerrit`` command over an SSH connection. It reports back to
+Gerrit using SSH.
+
+The dictionary passed to the Gerrit reporter is used for ``gerrit
+review`` arguments, with the boolean value of ``true`` simply
+indicating that the argument should be present without following it
+with a value. For example, ``verified: 1`` becomes ``gerrit review
+--verified 1`` and ``submit: true`` becomes ``gerrit review
+--submit``.
+
+A :ref:`connection<connections>` that uses the gerrit driver must be
+supplied to the trigger.
diff --git a/doc/source/admin/drivers/github.rst b/doc/source/admin/drivers/github.rst
new file mode 100644
index 0000000..cc2a5c9
--- /dev/null
+++ b/doc/source/admin/drivers/github.rst
@@ -0,0 +1,162 @@
+:title: GitHub Driver
+
+GitHub
+======
+
+The GitHub driver supports sources, triggers, and reporters. It can
+interact with the public GitHub service as well as site-local
+installations of GitHub enterprise.
+
+.. TODO: make this section more user friendly
+
+Configure GitHub `webhook events
+<https://developer.github.com/webhooks/creating/>`_.
+
+Set *Payload URL* to
+``http://<zuul-hostname>/connection/<connection-name>/payload``.
+
+Set *Content Type* to ``application/json``.
+
+Select *Events* you are interested in. See below for the supported events.
+
+Connection Configuration
+------------------------
+
+The supported options in zuul.conf connections are:
+
+**driver=github**
+
+**api_token**
+ API token for accessing GitHub.
+ See `Creating an access token for command-line use
+ <https://help.github.com/articles/creating-an-access-token-for-command-line-use/>`_.
+
+**webhook_token**
+ Optional: Token for validating the webhook event payloads.
+ If not specified, payloads are not validated.
+ See `Securing your webhooks
+ <https://developer.github.com/webhooks/securing/>`_.
+
+**sshkey**
+ Path to SSH key to use when cloning github repositories.
+ ``sshkey=/home/zuul/.ssh/id_rsa``
+
+**git_host**
+ Optional: Hostname of the github install (such as a GitHub Enterprise)
+ If not specified, defaults to ``github.com``
+ ``git_host=github.myenterprise.com``
+
+Trigger Configuration
+---------------------
+GitHub webhook events can be configured as triggers.
+
+A connection name with the github driver can take multiple events with the
+following options.
+
+**event**
+The event from github. Supported events are ``pull_request``,
+``pull_request_review``, and ``push``.
+
+A ``pull_request`` event will
+have associated action(s) to trigger from. The supported actions are:
+
+ *opened* - pull request opened
+
+ *changed* - pull request synchronized
+
+ *closed* - pull request closed
+
+ *reopened* - pull request reopened
+
+ *comment* - comment added on pull request
+
+ *labeled* - label added on pull request
+
+ *unlabeled* - label removed from pull request
+
+ *status* - status set on commit
+
+A ``pull_request_review`` event will
+have associated action(s) to trigger from. The supported actions are:
+
+ *submitted* - pull request review added
+
+ *dismissed* - pull request review removed
+
+**branch**
+The branch associated with the event. Example: ``master``. This
+field is treated as a regular expression, and multiple branches may
+be listed. Used for ``pull_request`` and ``pull_request_review`` events.
+
+**comment**
+This is only used for ``pull_request`` ``comment`` actions. It accepts a
+list of regexes that are searched for in the comment string. If any of these
+regexes matches a portion of the comment string the trigger is matched.
+``comment: retrigger`` will match when comments containing 'retrigger'
+somewhere in the comment text are added to a pull request.
+
+**label**
+This is only used for ``labeled`` and ``unlabeled`` ``pull_request`` actions.
+It accepts a list of strings each of which matches the label name in the
+event literally. ``label: recheck`` will match a ``labeled`` action when
+pull request is labeled with a ``recheck`` label. ``label: 'do not test'``
+will match a ``unlabeled`` action when a label with name ``do not test`` is
+removed from the pull request.
+
+**state**
+This is only used for ``pull_request_review`` events. It accepts a list of
+strings each of which is matched to the review state, which can be one of
+``approved``, ``comment``, or ``request_changes``.
+
+**status**
+This is used for ``pull-request`` and ``status`` actions. It accepts a
+list of strings each of which matches the user setting the status, the
+status context, and the status itself in the format of
+``user:context:status``. For example,
+``zuul_github_ci_bot:check_pipeline:success``.
+
+**ref**
+This is only used for ``push`` events. This field is treated as a regular
+expression and multiple refs may be listed. GitHub always sends full ref
+name, eg. ``refs/tags/bar`` and this string is matched against the regexp.
+
+Reporter Configuration
+----------------------
+Zuul reports back to GitHub via GitHub API. Available reports include a PR
+comment containing the build results, a commit status on start, success and
+failure, an issue label addition/removal on the PR, and a merge of the PR
+itself. Status name, description, and context is taken from the pipeline.
+
+A :ref:`connection<connections>` that uses the github driver must be
+supplied to the reporter. It has the following options:
+
+**status**
+String value (``pending``, ``success``, ``failure``) that the reporter should
+set as the commit status on github.
+``status: 'success'``
+
+**status-url**
+String value for a link url to set in the github status. Defaults to the zuul
+server status_url, or the empty string if that is unset.
+
+**comment**
+Boolean value (``true`` or ``false``) that determines if the reporter should
+add a comment to the pipeline status to the github pull request. Defaults
+to ``true``. Only used for Pull Request based events.
+``comment: false``
+
+**merge**
+Boolean value (``true`` or ``false``) that determines if the reporter should
+merge the pull reqeust. Defaults to ``false``. Only used for Pull Request based
+events.
+``merge=true``
+
+**label**
+List of strings each representing an exact label name which should be added
+to the pull request by reporter. Only used for Pull Request based events.
+``label: 'test successful'``
+
+**unlabel**
+List of strings each representing an exact label name which should be removed
+from the pull request by reporter. Only used for Pull Request based events.
+``unlabel: 'test failed'``
diff --git a/doc/source/admin/drivers/smtp.rst b/doc/source/admin/drivers/smtp.rst
new file mode 100644
index 0000000..6f24355
--- /dev/null
+++ b/doc/source/admin/drivers/smtp.rst
@@ -0,0 +1,53 @@
+:title: SMTP Driver
+
+SMTP
+====
+
+The SMTP driver supports reporters only. It is used to send email
+when items report.
+
+Connection Configuration
+------------------------
+
+**driver=smtp**
+
+**server**
+ SMTP server hostname or address to use.
+ ``server=localhost``
+
+**port**
+ Optional: SMTP server port.
+ ``port=25``
+
+**default_from**
+ Who the email should appear to be sent from when emailing the report.
+ This can be overridden by individual pipelines.
+ ``default_from=zuul@example.com``
+
+**default_to**
+ Who the report should be emailed to by default.
+ This can be overridden by individual pipelines.
+ ``default_to=you@example.com``
+
+Reporter Configuration
+----------------------
+
+A simple email reporter is also available.
+
+A :ref:`connection<connections>` that uses the smtp driver must be supplied to the
+reporter. The connection also may specify a default *To* or *From*
+address.
+
+Each pipeline can overwrite the ``subject`` or the ``to`` or ``from`` address by
+providing alternatives as arguments to the reporter. For example, ::
+
+ - pipeline:
+ name: post-merge
+ success:
+ outgoing_smtp:
+ to: you@example.com
+ failure:
+ internal_smtp:
+ to: you@example.com
+ from: alternative@example.com
+ subject: Change {change} failed
diff --git a/doc/source/admin/drivers/sql.rst b/doc/source/admin/drivers/sql.rst
new file mode 100644
index 0000000..b890f08
--- /dev/null
+++ b/doc/source/admin/drivers/sql.rst
@@ -0,0 +1,44 @@
+:title: SQL Driver
+
+SQL
+===
+
+The SQL driver supports reporters only. Only one connection per
+database is permitted. The connection options are:
+
+**driver=sql**
+
+**dburi**
+ Database connection information in the form of a URI understood by
+ sqlalchemy. eg http://docs.sqlalchemy.org/en/rel_1_0/core/engines.html#database-urls
+ ``dburi=mysql://user:pass@localhost/db``
+
+Reporter Configuration
+----------------------
+
+This reporter is used to store results in a database.
+
+A :ref:`connection<connections>` that uses the sql driver must be
+supplied to the reporter.
+
+zuul.conf contains the database connection and credentials. To store different
+reports in different databases you'll need to create a new connection per
+database.
+
+The SQL reporter does nothing on "start" or "merge-failure"; it only
+acts on "success" or "failure" reporting stages.
+
+**score**
+ A score to store for the result of the build. eg: -1 might indicate a failed
+ build.
+
+For example ::
+
+ - pipeline:
+ name: post-merge
+ success:
+ mydb_conn:
+ score: 1
+ failure:
+ mydb_conn:
+ score: -1
diff --git a/doc/source/admin/drivers/timer.rst b/doc/source/admin/drivers/timer.rst
new file mode 100644
index 0000000..574ee1e
--- /dev/null
+++ b/doc/source/admin/drivers/timer.rst
@@ -0,0 +1,24 @@
+:title: Timer Driver
+
+Timer
+=====
+
+The timer driver supports triggers only. It is used for configuring
+pipelines so that jobs run at scheduled times. No connection
+configuration is required.
+
+Trgger Configuration
+--------------------
+
+Timers don't require a special connection or driver. Instead they can
+simply be used by listing **timer** as the trigger.
+
+This trigger will run based on a cron-style time specification.
+It will enqueue an event into its pipeline for every project
+defined in the configuration. Any job associated with the
+pipeline will run in response to that event.
+
+**time**
+The time specification in cron syntax. Only the 5 part syntax is
+supported, not the symbolic names. Example: ``0 0 * * *`` runs
+at midnight.
diff --git a/doc/source/admin/drivers/zuul.rst b/doc/source/admin/drivers/zuul.rst
new file mode 100644
index 0000000..a875457
--- /dev/null
+++ b/doc/source/admin/drivers/zuul.rst
@@ -0,0 +1,40 @@
+:title: Zuul Driver
+
+Zuul
+====
+
+The Zuul driver supports triggers only. It is used for triggering
+pipelines based on internal Zuul events.
+
+Trigger Configuration
+---------------------
+
+Zuul events don't require a special connection or driver. Instead they
+can simply be used by listing **zuul** as the trigger.
+
+**event**
+The event name. Currently supported:
+
+ *project-change-merged* when Zuul merges a change to a project,
+ it generates this event for every open change in the project.
+
+ *parent-change-enqueued* when Zuul enqueues a change into any
+ pipeline, it generates this event for every child of that
+ change.
+
+**pipeline**
+Only available for ``parent-change-enqueued`` events. This is the
+name of the pipeline in which the parent change was enqueued.
+
+*require-approval*
+This may be used for any event. It requires that a certain kind
+of approval be present for the current patchset of the change (the
+approval could be added by the event in question). It follows the
+same syntax as the :ref:`"approval" pipeline requirement
+<pipeline-require-approval>`. For each specified criteria there must
+exist a matching approval.
+
+*reject-approval*
+This takes a list of approvals in the same format as
+*require-approval* but will fail to enter the pipeline if there is
+a matching approval.
diff --git a/doc/source/admin/index.rst b/doc/source/admin/index.rst
new file mode 100644
index 0000000..a2a2ee7
--- /dev/null
+++ b/doc/source/admin/index.rst
@@ -0,0 +1,20 @@
+Administrator's Guide
+=====================
+
+This guide is intended for administrators of Zuul systems. It covers
+installation, operation, and the portion of Zuul configuration that
+occurs outside of the projects upon which Zuul operates. Advanced
+users may be interested in some of the concepts described here, as
+well as understanding what features the underlying configuration
+provides to in-project configuration.
+
+.. toctree::
+ :maxdepth: 2
+
+ quick-start
+ installation
+ components
+ connections
+ tenants
+ monitoring
+ client
diff --git a/doc/source/admin/installation.rst b/doc/source/admin/installation.rst
new file mode 100644
index 0000000..bc61f7e
--- /dev/null
+++ b/doc/source/admin/installation.rst
@@ -0,0 +1,69 @@
+Installation
+============
+
+Install Zuul
+------------
+
+To install a Zuul release from PyPI, run::
+
+ pip install zuul
+
+Or from a git checkout, run::
+
+ pip install .
+
+That will also install Zuul's python dependencies. To minimize
+interaction with other python packages installed on a system, you may
+wish to install Zuul within a Python virtualenv.
+
+Zuul has several system-level dependencies as well. You can find a
+list of operating system packages in `bindep.txt` in Zuul's source
+directory.
+
+External Dependencies
+---------------------
+
+Zuul interacts with several other systems described below.
+
+Gearman
+~~~~~~~
+
+Gearman is a job distribution system that Zuul uses to communicate
+with its distributed components. The Zuul scheduler distributes work
+to Zuul mergers and executors use Gearman. You may supply your own
+gearman server, but the Zuul scheduler includes a built-in server
+which is recommended. Ensure that all Zuul hosts can communicate with
+the gearman server.
+
+Zuul distributes secrets to executors via gearman, so be sure to
+secure it with TLS and certificate authentication. Obtain (or
+generate) a certificate for both the server and the clients (they may
+use the same certificate or have individual certificates). They must
+be signed by a CA, but it can be your own CA.
+
+Nodepool
+~~~~~~~~
+
+In order to run all but the simplest jobs, Zuul uses a companion
+program, Nodepool, to supply the nodes (whether dynamic cloud
+instances or static hardware) used by jobs. Before starting Zuul,
+ensure you have Nodepool installed and any images you require built.
+Zuul only makes one requirement of these nodes: that it be able to log
+in given a username and ssh private key.
+
+.. TODO: SpamapS any zookeeper config recommendations?
+
+Nodepool uses Zookeeper to communicate internally among its
+components, and also to communicate with Zuul. You can run a simple
+single-node Zookeeper instance, or a multi-node cluster. Ensure that
+The host running the Zuul scheduler has access to the cluster.
+
+Ansible
+~~~~~~~
+
+Zuul uses Ansible to run jobs. Each version of Zuul is designed to
+work with a specific, contemporary version of Ansible. Zuul specifies
+that version of Ansible in its python package metadata, and normally
+the correct version will be installed automatically with Zuul.
+Because of the close integration of Zuul and Ansible, attempting to
+use other versions of Ansible with Zuul is not recommended.
diff --git a/doc/source/statsd.rst b/doc/source/admin/monitoring.rst
similarity index 76%
rename from doc/source/statsd.rst
rename to doc/source/admin/monitoring.rst
index fb6989e..2a6c959 100644
--- a/doc/source/statsd.rst
+++ b/doc/source/admin/monitoring.rst
@@ -1,15 +1,17 @@
-:title: Statsd reporting
+:title: Monitoring
+
+Monitoring
+==========
Statsd reporting
-================
+----------------
Zuul comes with support for the statsd protocol, when enabled and configured
(see below), the Zuul scheduler will emit raw metrics to a statsd receiver
-which let you in turn generate nice graphics. An example is OpenStack Zuul
-status page: http://status.openstack.org/zuul/
+which let you in turn generate nice graphics.
Configuration
--------------
+~~~~~~~~~~~~~
Statsd support uses the statsd python module. Note that Zuul will start without
the statsd python module, so an existing Zuul installation may be missing it.
@@ -27,17 +29,16 @@
STATSD_PORT=8125
Metrics
--------
+~~~~~~~
The metrics are emitted by the Zuul scheduler (`zuul/scheduler.py`):
**gerrit.event.<type> (counters)**
- Gerrit emits different kind of message over its `stream-events` interface. As
- a convenience, Zuul emits metrics to statsd which save you from having to use
- a different daemon to measure Gerrit events.
- The Gerrit events have different types defined by Gerrit itself, Zuul will
- relay any type of event reusing the name defined by Gerrit. Some of the
- events emitted are:
+ Gerrit emits different kind of message over its `stream-events`
+ interface. Zuul will report counters for each type of event it
+ receives from Gerrit.
+
+ Some of the events emitted are:
* patchset-created
* draft-published
@@ -52,18 +53,6 @@
Refer to your Gerrit installation documentation for an exhaustive list of
Gerrit event types.
-**zuul.node_type.**
- Holds metrics specifc to build nodes per label. The hierarchy is:
-
- #. **<build node label>** each of the labels associated to a build in
- Jenkins. It contains:
-
- #. **job.<jobname>** subtree detailing per job statistics:
-
- #. **wait_time** counter and timer of the wait time, with the
- difference of the job start time and the execute time, in
- milliseconds.
-
**zuul.pipeline.**
Holds metrics specific to jobs. The hierarchy is:
diff --git a/doc/source/admin/quick-start.rst b/doc/source/admin/quick-start.rst
new file mode 100644
index 0000000..780496b
--- /dev/null
+++ b/doc/source/admin/quick-start.rst
@@ -0,0 +1,120 @@
+Quick Start Guide
+=================
+
+This provides a very simple overview of Zuul. It is recommended to
+read the following sections for more details.
+
+Install Zuul
+------------
+
+You can get zuul from pypi via::
+
+ pip install zuul
+
+Zuul Components
+---------------
+
+Zuul provides the following components:
+
+ - **zuul-scheduler**: The main Zuul process. Handles receiving
+ events, executing jobs, collecting results and posting reports.
+ Coordinates the work of the other components.
+
+ - **zuul-merger**: Scale-out component that performs git merge
+ operations. Zuul performs a large number of git operations in
+ the course of its work. Adding merger processes can help speed
+ Zuul's processing. This component is optional (zero or more of
+ these can be run).
+
+ - **zuul-executor**: Scale-out component for executing jobs. At
+ least one of these is required. Depending on system
+ configuration, you can expect a single executor to handle up to
+ about 100 simultaneous jobs. Can handle the functions of a
+ merger if dedicated mergers are not provided. One or more of
+ these must be run.
+
+ - **gearman**: optional builtin gearman daemon provided by zuul-scheduler
+
+External components:
+
+ - **gearman**: A gearman daemon if the built-in daemon is not used.
+
+ - **zookeeper**: A zookeeper cluster (or single host) for
+ communicating with Nodepool.
+
+ - **nodepool**: Provides nodes for Zuul to use when executing jobs.
+
+
+Zuul Setup
+----------
+
+At minimum you need to provide **zuul.conf** and **main.yaml** placed
+in **/etc/zuul/**. The following example uses the builtin gearman
+service in Zuul, and a connection to Gerrit.
+
+**zuul.conf**::
+
+ [zuul]
+ tenant_config=/etc/zuul/main.yaml
+
+ [gearman_server]
+ start=true
+
+ [gearman]
+ server=127.0.0.1
+
+ [connection gerrit]
+ driver=gerrit
+ server=git.example.com
+ port=29418
+ baseurl=https://git.example.com/gerrit/
+ user=zuul
+ sshkey=/home/zuul/.ssh/id_rsa
+
+See :ref:`components` and :ref:`connections` for more details.
+
+The following tells Zuul to read its configuration from and operate on
+the *example-project* project:
+
+**main.yaml**::
+
+ - tenant:
+ name: example-tenant
+ source:
+ gerrit:
+ untrusted-projects:
+ - example-project
+
+Starting Zuul
+-------------
+
+You can run any zuul process with the **-d** option to make it not
+daemonize. It's a good idea at first to confirm there's no issues with
+your configuration.
+
+To start, simply run::
+
+ zuul-scheduler
+
+Once run you should have two zuul-scheduler processes (if using the
+built-in gearman server, or one process otherwise).
+
+Before Zuul can run any jobs, it needs to load its configuration, most
+of which is in the git repositories that Zuul operates on. Start an
+executor to allow zuul to do that::
+
+ zuul-executor
+
+Zuul should now be able to read its configuration from the configured
+repo and process any jobs defined therein.
+
+Troubleshooting
+---------------
+
+You can use telnet to connect to gearman to check which Zuul
+components are online::
+
+ telnet <gearman_ip> 4730
+
+Useful commands are **workers** and **status** which you can run by just
+typing those commands once connected to gearman.
diff --git a/doc/source/admin/tenants.rst b/doc/source/admin/tenants.rst
new file mode 100644
index 0000000..8872397
--- /dev/null
+++ b/doc/source/admin/tenants.rst
@@ -0,0 +1,95 @@
+:title: Tenant Configuration
+
+.. _tenant-config:
+
+Tenant Configuration
+====================
+
+After *zuul.conf* is configured, Zuul component servers will be able
+to start, but a tenant configuration is required in order for Zuul to
+perform any actions. The tenant configuration file specifies upon
+which projects Zuul should operate. These repositories are
+grouped into tenants. The configuration of each tenant is separate
+from the rest (no pipelines, jobs, etc are shared between them).
+
+A project may appear in more than one tenant; this may be useful if
+you wish to use common job definitions across multiple tenants.
+
+The tenant configuration file is specified by the *tenant_config*
+setting in the *scheduler* section of *zuul.yaml*. It is a YAML file
+which, like other Zuul configuration files, is a list of configuration
+objects, though only one type of object is supported, *tenant*.
+
+Tenant
+------
+
+A tenant is a collection of projects which share a Zuul
+configuration. An example tenant definition is::
+
+ - tenant:
+ name: my-tenant
+ source:
+ gerrit:
+ config-projects:
+ - common-config
+ - shared-jobs:
+ include: jobs
+ untrusted-projects:
+ - project1
+ - project2
+
+The following attributes are supported:
+
+**name** (required)
+ The name of the tenant. This may appear in URLs, paths, and
+ monitoring fields, and so should be restricted to URL friendly
+ characters (ASCII letters, numbers, hyphen and underscore) and you
+ should avoid changing it unless necessary.
+
+**source** (required)
+ A dictionary of sources to consult for projects. A tenant may
+ contain projects from multiple sources; each of those sources must
+ be listed here, along with the projects it supports. The name of a
+ :ref:`connection<connections>` is used as the dictionary key
+ (e.g. `gerrit` in the example above), and the value is a further
+ dictionary containing the keys below.
+
+ **config-projects**
+ A list of projects to be treated as config projects in this
+ tenant. The jobs in a config project are trusted, which means
+ they run with extra privileges, do not have their configuration
+ dynamically loaded for proposed changes, and zuul.yaml files are
+ only searched for in the master branch.
+
+ **untrusted-projects**
+ A list of projects to be treated as untrusted in this tenant. An
+ untrusted project is the typical project operated on by Zuul.
+ Their jobs run in a more restrictive environment, they may not
+ define pipelines, their configuration dynamically changes in
+ response to proposed changes, Zuul will read configuration files
+ in all of their branches.
+
+ Each of the projects listed may be either a simple string value, or
+ it may be a dictionary with the following keys:
+
+ **include**
+ Normally Zuul will load all of the configuration classes
+ appropriate for the type of project (config or untrusted) in
+ question. However, if you only want to load some items, the
+ *include* attribute can be used to specify that *only* the
+ specified classes should be loaded. Supplied as a string, or a
+ list of strings.
+
+ **exclude**
+ A list of configuration classes that should not be loaded.
+
+ The order of the projects listed in a tenant is important. A job
+ which is defined in one project may not be redefined in another
+ project; therefore, once a job appears in one project, a project
+ listed later will be unable to define a job with that name.
+ Further, some aspects of project configuration (such as the merge
+ mode) may only be set on the first appearance of a project
+ definition.
+
+ Zuul loads the configuration from all *config-projects* in the order
+ listed, followed by all *trusted-projects* in order.
diff --git a/doc/source/cloner.rst b/doc/source/cloner.rst
deleted file mode 100644
index 70577cc..0000000
--- a/doc/source/cloner.rst
+++ /dev/null
@@ -1,110 +0,0 @@
-:title: Zuul Cloner
-
-Zuul Cloner
-===========
-
-Zuul includes a simple command line client that may be used to clone
-repositories with Zuul references applied.
-
-Configuration
--------------
-
-Clone map
-'''''''''
-
-By default, Zuul cloner will clone the project under ``basepath`` which
-would create sub directories whenever a project name contains slashes. Since
-you might want to finely tweak the final destination, a clone map lets you
-change the destination on a per project basis. The configuration is done using
-a YAML file passed with ``-m``.
-
-With a project hierarchy such as::
-
- project
- thirdparty/plugins/plugin1
-
-You might want to get ``project`` straight in the base path, the clone map would be::
-
- clonemap:
- - name: 'project'
- dest: '.'
-
-Then to strip out ``thirdparty`` such that the plugins land under the
-``/plugins`` directory of the basepath, you can use regex and capturing
-groups::
-
- clonemap:
- - name: 'project'
- dest: '.'
- - name: 'thirdparty/(plugins/.*)'
- dest: '\1'
-
-The resulting workspace will contains::
-
- project -> ./
- thirdparty/plugins/plugin1 -> ./plugins/plugin1
-
-
-Zuul parameters
-'''''''''''''''
-
-The Zuul cloner reuses Zuul parameters such as ZUUL_BRANCH, ZUUL_REF or
-ZUUL_PROJECT. It will attempt to load them from the environment variables or
-you can pass them as parameters (in which case it will override the
-environment variable if it is set). The matching command line parameters use
-the ``zuul`` prefix hence ZUUL_REF can be passed to the cloner using
-``--zuul-ref``.
-
-Usage
------
-The general options that apply are:
-
-.. program-output:: zuul-cloner --help
-
-
-Ref lookup order
-''''''''''''''''
-
-The Zuul cloner will attempt to lookup references in this order:
-
- 1) Zuul reference for the indicated branch
- 2) Zuul reference for the master branch
- 3) The tip of the indicated branch
- 4) The tip of the master branch
-
-The "indicated branch" is one of the following:
-
- A) The project-specific override branch (from project_branches arg)
- B) The user specified branch (from the branch arg)
- C) ZUUL_BRANCH (from the zuul_branch arg)
-
-Clone order
------------
-
-When cloning repositories, the destination folder should not exist or
-``git clone`` will complain. This happens whenever cloning a sub project
-before its parent project. For example::
-
- zuul-cloner project/plugins/plugin1 project
-
-Will create the directory ``project`` when cloning the plugin. The
-cloner processes the clones in the order supplied, so you should swap the
-projects::
-
- zuul-cloner project project/plugins/plugin1
-
-Cached repositories
--------------------
-
-The ``--cache-dir`` option can be used to reduce network traffic by
-cloning from a local repository which may not be up to date.
-
-If the ``--cache-dir`` option is supplied, zuul-cloner will start by
-cloning any projects it processes from those found in that directory.
-The URL of origin remote of the resulting clone will be reset to use
-the ``git_base_url`` and then the remote will be updated so that the
-repository has all the information in the upstream repository.
-
-The default for ``--cache-dir`` is taken from the environment variable
-``ZUUL_CACHE_DIR``. A value provided explicitly on the command line
-overrides the environment variable setting.
diff --git a/doc/source/configuration.rst b/doc/source/configuration.rst
deleted file mode 100644
index 7ca0744..0000000
--- a/doc/source/configuration.rst
+++ /dev/null
@@ -1,38 +0,0 @@
-:title: Configuration
-
-.. _config:
-
-Configuration
-=============
-
-Zuul uses a very flexible configuration system. Some initial
-configuration must be supplied by the system administrator. From that
-beginning, Zuul builds its configuration by dynamically evaluating the
-contents of every git repository indicated by the configuration.
-
-Zuul configuration is focused primarily around the concept of a
-*Pipeline*. Through a pipeline, a Zuul user expresses a connection
-between events, projects, jobs, and reports. The general flow of
-operations is as follows:
-
-A *Connection* to a remote system generates an event; if it matches a
-*Trigger* on a *Pipeline*, then an *Item* is enqueued into the
-*Pipeline*. An *Item* is some kind of git reference, such as a branch
-tip, a proposed change, or a pull request, and is associated with a
-*Project*. Zuul launches the *Jobs* specified for that *Project* in
-that *Pipeline*, and when they complete, Zuul reports the results as
-specified by the pipeline's *Reporter*.
-
-TODO: flow diagram
-
-The following sections describe how to configure Zuul.
-:ref:`admin-config` covers areas of interest primarily to system
-administrators. :ref:`project-config` covers the portion of Zuul
-configuration of interest to most users -- that which is included
-directly in the git repositories operated on by Zuul.
-
-.. toctree::
- :maxdepth: 2
-
- admin-config
- project-config
diff --git a/doc/source/connections.rst b/doc/source/connections.rst
deleted file mode 100644
index 120d529..0000000
--- a/doc/source/connections.rst
+++ /dev/null
@@ -1,128 +0,0 @@
-:title: Connections
-
-.. _connections:
-
-Connections
-===========
-
-zuul coordinates talking to multiple different systems via the concept
-of connections. A connection is listed in the :ref:`zuulconf` file and is
-then referred to from the :ref:`layoutyaml`. This makes it possible to
-receive events from gerrit via one connection and post results from another
-connection that may report back as a different user.
-
-Gerrit
-------
-
-Create a connection with gerrit.
-
-**driver=gerrit**
-
-**server**
- FQDN of Gerrit server.
- ``server=review.example.com``
-
-**canonical_hostname**
- The canonical hostname associated with the git repos on the Gerrit
- server. Defaults to the value of **server**. This is used to
- identify repos from this connection by name and in preparing repos
- on the filesystem for use by jobs.
- ``canonical_hostname=git.example.com``
-
-**port**
- Optional: Gerrit server port.
- ``port=29418``
-
-**baseurl**
- Optional: path to Gerrit web interface. Defaults to ``https://<value
- of server>/``. ``baseurl=https://review.example.com/review_site/``
-
-**user**
- User name to use when logging into above server via ssh.
- ``user=zuul``
-
-**sshkey**
- Path to SSH key to use when logging into above server.
- ``sshkey=/home/zuul/.ssh/id_rsa``
-
-**keepalive**
- Optional: Keepalive timeout, 0 means no keepalive.
- ``keepalive=60``
-
-Gerrit Configuration
-~~~~~~~~~~~~~~~~~~~~
-
-Zuul will need access to a Gerrit user.
-
-Create an SSH keypair for Zuul to use if there isn't one already, and
-create a Gerrit user with that key::
-
- cat ~/id_rsa.pub | ssh -p29418 gerrit.example.com gerrit create-account --ssh-key - --full-name Jenkins jenkins
-
-Give that user whatever permissions will be needed on the projects you
-want Zuul to gate. For instance, you may want to grant ``Verified
-+/-1`` and ``Submit`` to the user. Additional categories or values may
-be added to Gerrit. Zuul is very flexible and can take advantage of
-those.
-
-GitHub
-------
-
-Create a connection with GitHub.
-
-**driver=github**
-
-**api_token**
- API token for accessing GitHub.
- See `Creating an access token for command-line use
- <https://help.github.com/articles/creating-an-access-token-for-command-line-use/>`_.
-
-**webhook_token**
- Optional: Token for validating the webhook event payloads.
- If not specified, payloads are not validated.
- See `Securing your webhooks
- <https://developer.github.com/webhooks/securing/>`_.
-
-**sshkey**
- Path to SSH key to use when cloning github repositories.
- ``sshkey=/home/zuul/.ssh/id_rsa``
-
-**git_host**
- Optional: Hostname of the github install (such as a GitHub Enterprise)
- If not specified, defaults to ``github.com``
- ``git_host=github.myenterprise.com``
-
-SMTP
-----
-
-**driver=smtp**
-
-**server**
- SMTP server hostname or address to use.
- ``server=localhost``
-
-**port**
- Optional: SMTP server port.
- ``port=25``
-
-**default_from**
- Who the email should appear to be sent from when emailing the report.
- This can be overridden by individual pipelines.
- ``default_from=zuul@example.com``
-
-**default_to**
- Who the report should be emailed to by default.
- This can be overridden by individual pipelines.
- ``default_to=you@example.com``
-
-SQL
-----
-
- Only one connection per a database is permitted.
-
- **driver=sql**
-
- **dburi**
- Database connection information in the form of a URI understood by
- sqlalchemy. eg http://docs.sqlalchemy.org/en/rel_1_0/core/engines.html#database-urls
- ``dburi=mysql://user:pass@localhost/db``
diff --git a/doc/source/encryption.rst b/doc/source/encryption.rst
deleted file mode 100644
index 6698763..0000000
--- a/doc/source/encryption.rst
+++ /dev/null
@@ -1,8 +0,0 @@
-:title: Encryption
-
-.. _encryption:
-
-Encryption
-==========
-
-TODO
diff --git a/doc/source/executors.rst b/doc/source/executors.rst
deleted file mode 100644
index f309a9c..0000000
--- a/doc/source/executors.rst
+++ /dev/null
@@ -1,317 +0,0 @@
-:title: Executors
-
-.. _Gearman: http://gearman.org/
-
-.. _`Gearman Plugin`:
- https://wiki.jenkins-ci.org/display/JENKINS/Gearman+Plugin
-
-.. _`Turbo-Hipster`:
- https://git.openstack.org/cgit/openstack/turbo-hipster/
-
-.. _`Turbo-Hipster Documentation`:
- http://turbo-hipster.rtfd.org/
-
-.. _executors:
-
-Executors
-=========
-
-Zuul has a modular architecture for executing jobs. Currently, the
-only supported module interfaces with Gearman_. This design allows
-any system to run jobs for Zuul simply by interfacing with a Gearman
-server. The recommended way of integrating a new job-runner with Zuul
-is via this method.
-
-If Gearman is unsuitable, Zuul may be extended with a new executor
-module. Zuul makes very few assumptions about the interface to a
-executor -- if it can trigger jobs, cancel them, and receive success
-or failure reports, it should be able to be used with Zuul. Patches
-to this effect are welcome.
-
-Zuul Parameters
----------------
-
-Zuul will pass some parameters with every job it executes. These are
-for workers to be able to get the repositories into the state they are
-intended to be tested in. Builds can be triggered either by an action
-on a change or by a reference update. Both events share a common set
-of parameters and more specific parameters as follows:
-
-Common parameters
-~~~~~~~~~~~~~~~~~
-
-**ZUUL_UUID**
- Zuul provided key to link builds with Gerrit events.
-**ZUUL_REF**
- Zuul provided ref that includes commit(s) to build.
-**ZUUL_COMMIT**
- The commit SHA1 at the head of ZUUL_REF.
-**ZUUL_PROJECT**
- The project that triggered this build.
-**ZUUL_PIPELINE**
- The Zuul pipeline that is building this job.
-**ZUUL_URL**
- The URL for the zuul server as configured in zuul.conf.
- A test runner may use this URL as the basis for fetching
- git commits.
-**BASE_LOG_PATH**
- zuul suggests a path to store and address logs. This is deterministic
- and hence useful for where you want workers to upload to a specific
- destination or need them to have a specific final URL you can link to
- in advanced. For changes it is:
- "last-two-digits-of-change/change-number/patchset-number".
- For reference updates it is: "first-two-digits-of-newrev/newrev"
-**LOG_PATH**
- zuul also suggests a unique path for logs to the worker. This is
- "BASE_LOG_PATH/pipeline-name/job-name/uuid"
-**ZUUL_VOTING**
- Whether Zuul considers this job voting or not. Note that if Zuul is
- reconfigured during the run, the voting status of a job may change
- and this value will be out of date. Values are '1' if voting, '0'
- otherwise.
-
-Change related parameters
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The following additional parameters will only be provided for builds
-associated with changes (i.e., in response to patchset-created or
-comment-added events):
-
-**ZUUL_BRANCH**
- The target branch for the change that triggered this build.
-**ZUUL_CHANGE**
- The Gerrit change ID for the change that triggered this build.
-**ZUUL_CHANGES**
- A caret character separated list of the changes upon which this build
- is dependent upon in the form of a colon character separated list
- consisting of project name, target branch, and revision ref.
-**ZUUL_CHANGE_IDS**
- All of the Gerrit change IDs that are included in this build (useful
- when the DependentPipelineManager combines changes for testing).
-**ZUUL_PATCHSET**
- The Gerrit patchset number for the change that triggered this build.
-
-Reference updated parameters
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The following additional parameters will only be provided for
-post-merge (ref-updated) builds:
-
-**ZUUL_OLDREV**
- The SHA1 of the old revision at this ref (recall the ref name is
- in ZUUL_REF).
-**ZUUL_NEWREV**
- The SHA1 of the new revision at this ref (recall the ref name is
- in ZUUL_REF).
-**ZUUL_SHORT_OLDREV**
- The shortened (7 character) SHA1 of the old revision.
-**ZUUL_SHORT_NEWREV**
- The shortened (7 character) SHA1 of the new revision.
-
-Unset revisions default to 00000000000000000000000000000000.
-
-Examples:
-
-When a reference is created::
-
- ZUUL_OLDREV=00000000000000000000000000000000
- ZUUL_NEWREV=123456789abcdef123456789abcdef12
- ZUUL_SHORT_OLDREV=0000000
- ZUUL_SHORT_NEWREV=1234567
-
-When a reference is deleted::
-
- ZUUL_OLDREV=123456789abcdef123456789abcdef12
- ZUUL_NEWREV=00000000000000000000000000000000
- ZUUL_SHORT_OLDREV=1234567
- ZUUL_SHORT_NEWREV=0000000
-
-And finally a reference being altered::
-
- ZUUL_OLDREV=123456789abcdef123456789abcdef12
- ZUUL_NEWREV=abcdef123456789abcdef123456789ab
- ZUUL_SHORT_OLDREV=1234567
- ZUUL_SHORT_NEWREV=abcdef1
-
-Your jobs can check whether the parameters are ``000000`` to act
-differently on each kind of event.
-
-Gearman
--------
-
-Gearman_ is a general-purpose protocol for distributing jobs to any
-number of workers. Zuul works with Gearman by sending specific
-information with job requests to Gearman, and expects certain
-information to be returned on completion. This protocol is described
-in `Zuul-Gearman Protocol`_.
-
-In order for Zuul to run any jobs, you will need a running Gearman
-server. Zuul includes a Gearman server, and it is recommended that it
-be used as it supports the following features needed by Zuul:
-
-* Canceling jobs in the queue (admin protocol command "cancel job").
-* Strict FIFO queue operation (gearmand's round-robin mode may be
- sufficient, but is untested).
-
-To enable the built-in server, see the ``gearman_server`` section of
-``zuul.conf``. Be sure that the host allows connections from Zuul and
-any workers (e.g., Jenkins masters) on TCP port 4730, and nowhere else
-(as the Gearman protocol does not include any provision for
-authentication).
-
-Gearman Jenkins Plugin
-~~~~~~~~~~~~~~~~~~~~~~
-
-The `Gearman Jenkins Plugin`_ makes it easy to use Jenkins with Zuul
-by providing an interface between Jenkins and Gearman. In this
-configuration, Zuul asks Gearman to run jobs, and Gearman can then
-distribute those jobs to any number of Jenkins systems (including
-multiple Jenkins masters).
-
-The `Gearman Plugin`_ can be installed in Jenkins in order to
-facilitate Jenkins running jobs for Zuul. Install the plugin and
-configure it with the hostname or IP address of your Gearman server
-and the port on which it is listening (4730 by default). It will
-automatically register all known Jenkins jobs as functions that Zuul
-can invoke via Gearman.
-
-Any number of masters can be configured in this way, and Gearman will
-distribute jobs to all of them as appropriate.
-
-No special Jenkins job configuration is needed to support triggering
-by Zuul.
-
-The Gearman Plugin will ensure the `Zuul Parameters`_ are supplied as
-Jenkins build parameters, so they will be available for use in the job
-configuration as well as to the running job as environment variables.
-
-Jenkins git plugin configuration
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-In order to test the correct build, configure the Jenkins Git SCM
-plugin as follows::
-
- Source Code Management:
- Git
- Repositories:
- Repository URL: <your Gerrit or Zuul repository URL>
- Advanced:
- Refspec: ${ZUUL_REF}
- Branches to build:
- Branch Specifier: ${ZUUL_COMMIT}
- Advanced:
- Clean after checkout: True
-
-That should be sufficient for a job that only builds a single project.
-If you have multiple interrelated projects (i.e., they share a Zuul
-Change Queue) that are built together, you may be able to configure
-the Git plugin to prepare them, or you may chose to use a shell script
-instead. As an example, the OpenStack project uses the following
-script to prepare the workspace for its integration testing:
-
- https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate-wrap.sh
-
-Turbo Hipster Worker
-~~~~~~~~~~~~~~~~~~~~
-
-As an alternative to Jenkins, `Turbo-Hipster`_ is a small python
-project designed specifically as a zuul job worker which can be
-registered with gearman as a job runner. Please see the
-`Turbo-Hipster Documentation`_ for details on how to set it up.
-
-Zuul-Gearman Protocol
-~~~~~~~~~~~~~~~~~~~~~
-
-This section is only relevant if you intend to implement a new kind of
-worker that runs jobs for Zuul via Gearman. If you just want to use
-Jenkins, see `Gearman Jenkins Plugin`_ instead.
-
-The Zuul protocol as used with Gearman is as follows:
-
-Starting Builds
-^^^^^^^^^^^^^^^
-
-To start a build, Zuul invokes a Gearman function with the following
-format:
-
- build:FUNCTION_NAME
-
-where **FUNCTION_NAME** is the name of the job that should be run. If
-the job should run on a specific node (or class of node), Zuul will
-instead invoke:
-
- build:FUNCTION_NAME:NODE_NAME
-
-where **NODE_NAME** is the name or class of node on which the job
-should be run.
-
-Zuul sends the ZUUL_* parameters described in `Zuul Parameters`_
-encoded in JSON format as the argument included with the
-SUBMIT_JOB_UNIQ request to Gearman. A unique ID (equal to the
-ZUUL_UUID parameter) is also supplied to Gearman, and is accessible as
-an added Gearman parameter with GRAB_JOB_UNIQ.
-
-When a Gearman worker starts running a job for Zuul, it should
-immediately send a WORK_DATA packet with the following information
-encoded in JSON format:
-
-**name**
- The name of the job.
-
-**number**
- The build number (unique to this job).
-
-**manager**
- A unique identifier associated with the Gearman worker that can
- abort this build. See `Stopping Builds`_ for more information.
-
-**url** (optional)
- The URL with the status or results of the build. Will be used in
- the status page and the final report.
-
-To help with debugging builds a worker may send back some optional
-metadata:
-
-**worker_name** (optional)
- The unique name of the worker.
-
-**worker_hostname** (optional)
- The hostname of the worker.
-
-It should then immediately send a WORK_STATUS packet with a value of 0
-percent complete. It may then optionally send subsequent WORK_STATUS
-packets with updated completion values.
-
-When the build is complete, it should send a final WORK_DATA packet
-with the following in JSON format:
-
-**result**
- Either the string 'SUCCESS' if the job succeeded, or any other value
- that describes the result if the job failed.
-
-Finally, it should send either a WORK_FAIL or WORK_COMPLETE packet as
-appropriate. A WORK_EXCEPTION packet will be interpreted as a
-WORK_FAIL, but the exception will be logged in Zuul's error log.
-
-Stopping Builds
-^^^^^^^^^^^^^^^
-
-If Zuul needs to abort a build already in progress, it will invoke the
-following function through Gearman:
-
- stop:MANAGER_NAME
-
-Where **MANAGER_NAME** is the name of the manager worker supplied in
-the initial WORK_DATA packet when the job started. This is used to
-direct the stop: function invocation to the correct Gearman worker
-that is capable of stopping that particular job. The argument to the
-function should be the following encoded in JSON format:
-
-**name**
- The job name of the build to stop.
-
-**number**
- The build number of the build to stop.
-
-The original job is expected to complete with a WORK_DATA and
-WORK_FAIL packet as described in `Starting Builds`_.
diff --git a/doc/source/index.rst b/doc/source/index.rst
index a3a4df5..24ab31b 100644
--- a/doc/source/index.rst
+++ b/doc/source/index.rst
@@ -1,31 +1,24 @@
Zuul - A Project Gating System
==============================
-Zuul is a program that is used to gate the source code repository of a
-project so that changes are only merged if they pass tests.
+Zuul is a program that drives continuous integration, delivery, and
+deployment systems with a focus on project gating and interrelated
+projects.
-The main component of Zuul is the scheduler. It receives events
-related to proposed changes, triggers tests based on those events, and
-reports back.
+Zuul's documentation is organized in three guides based on audience.
+If Zuul is being used to gate or drive automation around your project,
+read the :doc:`user/index` to find out how to configure Zuul. If you
+are installing or operating a Zuul system, you will also find the
+:doc:`admin/index` useful. If you want help make Zuul itself better,
+take a look at the :doc:`developer/index`.
Contents:
.. toctree::
:maxdepth: 2
- quick-start
- gating
- configuration
- encryption
- connections
- triggers
- reporters
- zuul
- merger
- cloner
- executors
- statsd
- client
+ user/index
+ admin/index
developer/index
Indices and tables
diff --git a/doc/source/merger.rst b/doc/source/merger.rst
deleted file mode 100644
index d8de702..0000000
--- a/doc/source/merger.rst
+++ /dev/null
@@ -1,74 +0,0 @@
-:title: Merger
-
-Merger
-======
-
-The Zuul Merger is a separate component which communicates with the
-main Zuul server via Gearman. Its purpose is to speculatively merge
-the changes for Zuul in preparation for testing. The resulting git
-commits also must be served to the test workers, and the server(s)
-running the Zuul Merger are expected to do this as well. Because both
-of these tasks are resource intensive, any number of Zuul Mergers can
-be run in parallel on distinct hosts.
-
-Configuration
-~~~~~~~~~~~~~
-
-The Zuul Merger can read the same zuul.conf file as the main Zuul
-server and requires the ``gearman``, ``gerrit``, ``merger``, and
-``zuul`` sections (indicated fields only). Be sure the zuul_url is
-set appropriately on each host that runs a zuul-merger.
-
-Zuul References
-~~~~~~~~~~~~~~~
-
-As the DependentPipelineManager may combine several changes together
-for testing when performing speculative execution, determining exactly
-how the workspace should be set up when running a Job can be complex.
-To alleviate this problem, Zuul performs merges itself, merging or
-cherry-picking changes as required and identifies the result with a
-Git reference of the form ``refs/zuul/<branch>/Z<random sha1>``.
-Preparing the workspace is then a simple matter of fetching that ref
-and checking it out. The parameters that provide this information are
-described in :ref:`executors`.
-
-These references need to be made available via a Git repository that
-is available to workers (such as Jenkins). This is accomplished by
-serving Zuul's Git repositories directly.
-
-Serving Zuul Git Repos
-~~~~~~~~~~~~~~~~~~~~~~
-
-Zuul maintains its own copies of any needed Git repositories in the
-directory specified by ``git_dir`` in the ``merger`` section of
-zuul.conf (by default, /var/lib/zuul/git). To directly serve Zuul's
-Git repositories in order to provide Zuul refs for workers, you can
-configure Apache to do so using the following directives::
-
- SetEnv GIT_PROJECT_ROOT /var/lib/zuul/git
- SetEnv GIT_HTTP_EXPORT_ALL
-
- AliasMatch ^/p/(.*/objects/[0-9a-f]{2}/[0-9a-f]{38})$ /var/lib/zuul/git/$1
- AliasMatch ^/p/(.*/objects/pack/pack-[0-9a-f]{40}.(pack|idx))$ /var/lib/zuul/git/$1
- ScriptAlias /p/ /usr/lib/git-core/git-http-backend/
-
-Note that Zuul's Git repositories are not bare, which means they have
-a working tree, and are not suitable for public consumption (for
-instance, a clone will produce a repository in an unpredictable state
-depending on what the state of Zuul's repository is when the clone
-happens). They are, however, suitable for automated systems that
-respond to Zuul triggers.
-
-Clearing old references
-~~~~~~~~~~~~~~~~~~~~~~~
-
-The references created under refs/zuul are not garbage collected. Since
-git fetch send them all to Gerrit to sync the repositories, the time
-spent on merge will slightly grow overtime and start being noticeable.
-
-To clean them you can use the ``tools/zuul-clear-refs.py`` script on
-each repositories. It will delete Zuul references that point to commits
-for which the commit date is older than a given amount of days (default
-360)::
-
- ./tools/zuul-clear-refs.py /path/to/zuul/git/repo
diff --git a/doc/source/quick-start.rst b/doc/source/quick-start.rst
deleted file mode 100644
index 6d16c92..0000000
--- a/doc/source/quick-start.rst
+++ /dev/null
@@ -1,162 +0,0 @@
-Quick Start Guide
-=================
-
-System Requirements
--------------------
-
-For most deployments zuul only needs 1-2GB. OpenStack uses a 30GB setup.
-
-Install Zuul
-------------
-
-You can get zuul from pypi via::
-
- pip install zuul
-
-Zuul Components
----------------
-
-Zuul provides the following components:
-
- - **zuul-server**: scheduler daemon which communicates with Gerrit and
- Gearman. Handles receiving events, executing jobs, collecting results
- and postingreports.
- - **zuul-merger**: speculative-merger which communicates with Gearman.
- Prepares Git repositories for jobs to test against. This additionally
- requires a web server hosting the Git repositories which can be cloned
- by the jobs.
- - **zuul-cloner**: client side script used to setup job workspace. It is
- used to clone the repositories prepared by the zuul-merger described
- previously.
- - **gearmand**: optional builtin gearman daemon provided by zuul-server
-
-External components:
-
- - Jenkins Gearman plugin: Used by Jenkins to connect to Gearman
-
-Zuul Communication
-------------------
-
-All the Zuul components communicate with each other using Gearman. As well as
-the following communication channels:
-
-zuul-server:
-
- - Gerrit
- - Gearman Daemon
-
-zuul-merger:
-
- - Gerrit
- - Gearman Daemon
-
-zuul-cloner:
-
- - http hosted zuul-merger git repos
-
-Jenkins:
-
- - Gearman Daemon via Jenkins Gearman Plugin
-
-Zuul Setup
-----------
-
-At minimum we need to provide **zuul.conf** and **layout.yaml** and placed
-in /etc/zuul/ directory. You will also need a zuul user and ssh key for the
-zuul user in Gerrit. The following example uses the builtin gearmand service
-in zuul.
-
-**zuul.conf**::
-
- [zuul]
- layout_config=/etc/zuul/layout.yaml
-
- [merger]
- git_dir=/git
- zuul_url=http://zuul.example.com/p
-
- [gearman_server]
- start=true
-
- [gearman]
- server=127.0.0.1
-
- [connection gerrit]
- driver=gerrit
- server=git.example.com
- port=29418
- baseurl=https://git.example.com/gerrit/
- user=zuul
- sshkey=/home/zuul/.ssh/id_rsa
-
-See :doc:`zuul` for more details.
-
-The following sets up a basic timer triggered job using zuul.
-
-**layout.yaml**::
-
- pipelines:
- - name: periodic
- source: gerrit
- manager: IndependentPipelineManager
- trigger:
- timer:
- - time: '0 * * * *'
-
- projects:
- - name: aproject
- periodic:
- - aproject-periodic-build
-
-Starting Zuul
--------------
-
-You can run zuul-server with the **-d** option to make it not daemonize. It's
-a good idea at first to confirm there's no issues with your configuration.
-
-Simply run::
-
- zuul-server
-
-Once run you should have 2 zuul-server processes::
-
- zuul 12102 1 0 Jan21 ? 00:15:45 /home/zuul/zuulvenv/bin/python /home/zuul/zuulvenv/bin/zuul-server -d
- zuul 12107 12102 0 Jan21 ? 00:00:01 /home/zuul/zuulvenv/bin/python /home/zuul/zuulvenv/bin/zuul-server -d
-
-Note: In this example zuul was installed in a virtualenv.
-
-The 2nd zuul-server process is gearmand running if you are using the builtin
-gearmand server, otherwise there will only be 1 process.
-
-Zuul won't actually process your Job queue however unless you also have a
-zuul-merger process running.
-
-Simply run::
-
- zuul-merger
-
-Zuul should now be able to process your periodic job as configured above once
-the Jenkins side of things is configured.
-
-Jenkins Setup
--------------
-
-Install the Jenkins Gearman Plugin via Jenkins Plugin management interface.
-Then naviage to **Manage > Configuration > Gearman** and setup the Jenkins
-server hostname/ip and port to connect to gearman.
-
-At this point gearman should be running your Jenkins jobs.
-
-Troubleshooting
----------------
-
-Checking Gearman function registration (jobs). You can use telnet to connect
-to gearman to check that Jenkins is registering your configured jobs in
-gearman::
-
- telnet <gearman_ip> 4730
-
-Useful commands are **workers** and **status** which you can run by just
-typing those commands once connected to gearman. Every job in your Jenkins
-master must appear when you run **workers** for Zuul to be able to run jobs
-against your Jenkins instance.
diff --git a/doc/source/reporters.rst b/doc/source/reporters.rst
deleted file mode 100644
index ae6ab1c..0000000
--- a/doc/source/reporters.rst
+++ /dev/null
@@ -1,143 +0,0 @@
-:title: Reporters
-
-Reporters
-=========
-
-Zuul can communicate results and progress back to configurable
-protocols. For example, after succeeding in a build a pipeline can be
-configured to post a positive review back to Gerrit.
-
-There are three stages when a report can be handled. That is on:
-Start, Success or Failure. Each stage can have multiple reports.
-For example, you can set verified on Gerrit and send an email.
-
-Gerrit
-------
-
-Zuul works with standard versions of Gerrit by invoking the
-``gerrit`` command over an SSH connection. It reports back to
-Gerrit using SSH.
-
-The dictionary passed to the Gerrit reporter is used for ``gerrit
-review`` arguments, with the boolean value of ``true`` simply
-indicating that the argument should be present without following it
-with a value. For example, ``verified: 1`` becomes ``gerrit review
---verified 1`` and ``submit: true`` becomes ``gerrit review
---submit``.
-
-A :ref:`connection` that uses the gerrit driver must be supplied to the
-trigger.
-
-GitHub
-------
-
-Zuul reports back to GitHub via GitHub API. Available reports include a PR
-comment containing the build results, a commit status on start, success and
-failure, an issue label addition/removal on the PR, and a merge of the PR
-itself. Status name, description, and context is taken from the pipeline.
-
-A :ref:`connection` that uses the github driver must be supplied to the
-reporter. It has the following options:
-
- **status**
- String value (``pending``, ``success``, ``failure``) that the reporter should
- set as the commit status on github.
- ``status: 'success'``
-
- **status-url**
- String value for a link url to set in the github status. Defaults to the zuul
- server status_url, or the empty string if that is unset.
-
- **comment**
- Boolean value (``true`` or ``false``) that determines if the reporter should
- add a comment to the pipeline status to the github pull request. Defaults
- to ``true``. Only used for Pull Request based events.
- ``comment: false``
-
- **merge**
- Boolean value (``true`` or ``false``) that determines if the reporter should
- merge the pull reqeust. Defaults to ``false``. Only used for Pull Request based
- events.
- ``merge=true``
-
- **label**
- List of strings each representing an exact label name which should be added
- to the pull request by reporter. Only used for Pull Request based events.
- ``label: 'test successful'``
-
- **unlabel**
- List of strings each representing an exact label name which should be removed
- from the pull request by reporter. Only used for Pull Request based events.
- ``unlabel: 'test failed'``
-
-SMTP
-----
-
-A simple email reporter is also available.
-
-A :ref:`connection` that uses the smtp driver must be supplied to the
-reporter.
-
-SMTP Configuration
-~~~~~~~~~~~~~~~~~~
-
-zuul.conf contains the SMTP server and default to/from as described
-in :ref:`zuulconf`.
-
-Each pipeline can overwrite the ``subject`` or the ``to`` or ``from`` address by
-providing alternatives as arguments to the reporter. For example, ::
-
- pipelines:
- - name: post-merge
- manager: IndependentPipelineManager
- source: my_gerrit
- trigger:
- my_gerrit:
- - event: change-merged
- success:
- outgoing_smtp:
- to: you@example.com
- failure:
- internal_smtp:
- to: you@example.com
- from: alternative@example.com
- subject: Change {change} failed
-
-SQL
----
-
-This reporter is used to store results in a database.
-
-A :ref:`connection` that uses the sql driver must be supplied to the
-reporter.
-
-SQL Configuration
-~~~~~~~~~~~~~~~~~
-
-zuul.conf contains the database connection and credentials. To store different
-reports in different databases you'll need to create a new connection per
-database.
-
-The sql reporter is used to store the results from individual builds rather
-than the change. As such the sql reporter does nothing on "start" or
-"merge-failure".
-
-**score**
- A score to store for the result of the build. eg: -1 might indicate a failed
- build similar to the vote posted back via the gerrit reporter.
-
-For example ::
-
- pipelines:
- - name: post-merge
- manager: IndependentPipelineManager
- source: my_gerrit
- trigger:
- my_gerrit:
- - event: change-merged
- success:
- mydb_conn:
- score: 1
- failure:
- mydb_conn:
- score: -1
diff --git a/doc/source/triggers.rst b/doc/source/triggers.rst
deleted file mode 100644
index 41a56a0..0000000
--- a/doc/source/triggers.rst
+++ /dev/null
@@ -1,247 +0,0 @@
-:title: Triggers
-
-Triggers
-========
-
-The process of merging a change starts with proposing a change to be
-merged. Zuul supports Gerrit and GitHub as triggering systems.
-Zuul's design is modular, so alternate triggering and reporting
-systems can be supported.
-
-Gerrit
-------
-
-Zuul works with standard versions of Gerrit by invoking the ``gerrit
-stream-events`` command over an SSH connection. It also reports back
-to Gerrit using SSH.
-
-If using Gerrit 2.7 or later, make sure the user is a member of a group
-that is granted the ``Stream Events`` permission, otherwise it will not
-be able to invoke the ``gerrit stream-events`` command over SSH.
-
-A connection name with the gerrit driver can take multiple events with
-the following options.
-
- **event**
- The event name from gerrit. Examples: ``patchset-created``,
- ``comment-added``, ``ref-updated``. This field is treated as a
- regular expression.
-
- **branch**
- The branch associated with the event. Example: ``master``. This
- field is treated as a regular expression, and multiple branches may
- be listed.
-
- **ref**
- On ref-updated events, the branch parameter is not used, instead the
- ref is provided. Currently Gerrit has the somewhat idiosyncratic
- behavior of specifying bare refs for branch names (e.g., ``master``),
- but full ref names for other kinds of refs (e.g., ``refs/tags/foo``).
- Zuul matches what you put here exactly against what Gerrit
- provides. This field is treated as a regular expression, and
- multiple refs may be listed.
-
- **ignore-deletes**
- When a branch is deleted, a ref-updated event is emitted with a newrev
- of all zeros specified. The ``ignore-deletes`` field is a boolean value
- that describes whether or not these newrevs trigger ref-updated events.
- The default is True, which will not trigger ref-updated events.
-
- **approval**
- This is only used for ``comment-added`` events. It only matches if
- the event has a matching approval associated with it. Example:
- ``code-review: 2`` matches a ``+2`` vote on the code review category.
- Multiple approvals may be listed.
-
- **email**
- This is used for any event. It takes a regex applied on the performer
- email, i.e. Gerrit account email address. If you want to specify
- several email filters, you must use a YAML list. Make sure to use non
- greedy matchers and to escapes dots!
- Example: ``email: ^.*?@example\.org$``.
-
- **email_filter** (deprecated)
- A deprecated alternate spelling of *email*. Only one of *email* or
- *email_filter* should be used.
-
- **username**
- This is used for any event. It takes a regex applied on the performer
- username, i.e. Gerrit account name. If you want to specify several
- username filters, you must use a YAML list. Make sure to use non greedy
- matchers and to escapes dots!
- Example: ``username: ^jenkins$``.
-
- **username_filter** (deprecated)
- A deprecated alternate spelling of *username*. Only one of *username* or
- *username_filter* should be used.
-
- **comment**
- This is only used for ``comment-added`` events. It accepts a list of
- regexes that are searched for in the comment string. If any of these
- regexes matches a portion of the comment string the trigger is
- matched. ``comment: retrigger`` will match when comments
- containing 'retrigger' somewhere in the comment text are added to a
- change.
-
- **comment_filter** (deprecated)
- A deprecated alternate spelling of *comment*. Only one of *comment* or
- *comment_filter* should be used.
-
- *require-approval*
- This may be used for any event. It requires that a certain kind
- of approval be present for the current patchset of the change (the
- approval could be added by the event in question). It follows the
- same syntax as the :ref:`"approval" pipeline requirement
- <pipeline-require-approval>`. For each specified criteria there must
- exist a matching approval.
-
- *reject-approval*
- This takes a list of approvals in the same format as
- *require-approval* but will fail to enter the pipeline if there is
- a matching approval.
-
-GitHub
-------
-
-Github webhook events can be configured as triggers.
-
-A connection name with the github driver can take multiple events with the
-following options.
-
- **event**
- The event from github. Supported events are ``pull_request``,
- ``pull_request_review``, and ``push``.
-
- A ``pull_request`` event will
- have associated action(s) to trigger from. The supported actions are:
-
- *opened* - pull request opened
-
- *changed* - pull request synchronized
-
- *closed* - pull request closed
-
- *reopened* - pull request reopened
-
- *comment* - comment added on pull request
-
- *labeled* - label added on pull request
-
- *unlabeled* - label removed from pull request
-
- *review* - review added on pull request
-
- *push* - head reference updated (pushed to branch)
-
- *status* - status set on commit
-
- A ``pull_request_review`` event will
- have associated action(s) to trigger from. The supported actions are:
-
- *submitted* - pull request review added
-
- *dismissed* - pull request review removed
-
- **branch**
- The branch associated with the event. Example: ``master``. This
- field is treated as a regular expression, and multiple branches may
- be listed. Used for ``pull_request`` and ``pull_request_review`` events.
-
- **comment**
- This is only used for ``pull_request`` ``comment`` actions. It accepts a
- list of regexes that are searched for in the comment string. If any of these
- regexes matches a portion of the comment string the trigger is matched.
- ``comment: retrigger`` will match when comments containing 'retrigger'
- somewhere in the comment text are added to a pull request.
-
- **label**
- This is only used for ``labeled`` and ``unlabeled`` ``pull_request`` actions.
- It accepts a list of strings each of which matches the label name in the
- event literally. ``label: recheck`` will match a ``labeled`` action when
- pull request is labeled with a ``recheck`` label. ``label: 'do not test'``
- will match a ``unlabeled`` action when a label with name ``do not test`` is
- removed from the pull request.
-
- **state**
- This is only used for ``pull_request_review`` events. It accepts a list of
- strings each of which is matched to the review state, which can be one of
- ``approved``, ``comment``, or ``request_changes``.
-
- **status**
- This is only used for ``status`` actions. It accepts a list of strings each of
- which matches the user setting the status, the status context, and the status
- itself in the format of ``user:context:status``. For example,
- ``zuul_github_ci_bot:check_pipeline:success``.
-
- **ref**
- This is only used for ``push`` events. This field is treated as a regular
- expression and multiple refs may be listed. Github always sends full ref
- name, eg. ``refs/tags/bar`` and this string is matched against the regexp.
-
-GitHub Configuration
-~~~~~~~~~~~~~~~~~~~~
-
-Configure GitHub `webhook events
-<https://developer.github.com/webhooks/creating/>`_.
-
-Set *Payload URL* to
-``http://<zuul-hostname>/connection/<connection-name>/payload``.
-
-Set *Content Type* to ``application/json``.
-
-Select *Events* you are interested in. See above for the supported events.
-
-Timer
------
-
-A simple timer trigger is available as well. It supports triggering
-jobs in a pipeline based on cron-style time instructions.
-
-Timers don't require a special connection or driver. Instead they can
-be used by listing **timer** as the trigger.
-
-This trigger will run based on a cron-style time specification.
-It will enqueue an event into its pipeline for every project
-defined in the configuration. Any job associated with the
-pipeline will run in response to that event.
-
- **time**
- The time specification in cron syntax. Only the 5 part syntax is
- supported, not the symbolic names. Example: ``0 0 * * *`` runs
- at midnight.
-
-Zuul
-----
-
-The Zuul trigger generates events based on internal actions in Zuul.
-Multiple events may be listed.
-
-Zuul events don't require a special connection or driver. Instead they
-can be used by listing **zuul** as the trigger.
-
- **event**
- The event name. Currently supported:
-
- *project-change-merged* when Zuul merges a change to a project,
- it generates this event for every open change in the project.
-
- *parent-change-enqueued* when Zuul enqueues a change into any
- pipeline, it generates this event for every child of that
- change.
-
- **pipeline**
- Only available for ``parent-change-enqueued`` events. This is the
- name of the pipeline in which the parent change was enqueued.
-
- *require-approval*
- This may be used for any event. It requires that a certain kind
- of approval be present for the current patchset of the change (the
- approval could be added by the event in question). It follows the
- same syntax as the :ref:`"approval" pipeline requirement
- <pipeline-require-approval>`. For each specified criteria there must
- exist a matching approval.
-
- *reject-approval*
- This takes a list of approvals in the same format as
- *require-approval* but will fail to enter the pipeline if there is
- a matching approval.
diff --git a/doc/source/user/concepts.rst b/doc/source/user/concepts.rst
new file mode 100644
index 0000000..6197396
--- /dev/null
+++ b/doc/source/user/concepts.rst
@@ -0,0 +1,85 @@
+:title: Zuul Concepts
+
+Zuul Concepts
+=============
+
+Zuul's configuration is organized around the concept of a *pipeline*.
+In Zuul, a pipeline encompasses a workflow process which can be
+applied to one or more projects. For instance, a "check" pipeline
+might describe the actions which should cause newly proposed changes
+to projects to be tested. A "gate" pipeline might implement
+:ref:`project_gating` to automate merging changes to projects only if
+their tests pass. A "post" pipeline might update published
+documentation for a project when changes land.
+
+The names "check", "gate", and "post" are arbitrary -- these are not
+concepts built into Zuul, but rather they are just a few common
+examples of workflow processes that can be described to Zuul and
+implemented as pipelines.
+
+Once a pipeline has been defined, any number of projects may be
+associated with it, each one specifying what jobs should be run for
+that project in a given pipeline.
+
+Pipelines have associated *triggers* which are descriptions of events
+which should cause something to be enqueued into a pipeline. For
+example, when a patchset is uploaded to Gerrit, a Gerrit
+*patchset-created* event is emitted. A pipeline configured to trigger
+on *patchset-created* events would then enqueue the associated change
+when Zuul receives that event. If there are jobs associated with that
+project and pipeline, they will be run. In addition to Gerrit, other
+triggers are available, including GitHub, timer, and zuul. See
+:ref:`drivers` for a full list of available triggers.
+
+Once all of the jobs for an item in a pipeline have been run, the
+pipeline's *reporters* are responsible for reporting the results of
+all of the jobs. Continuing the example from earlier, if a pipeline
+were configured with a Gerrit reporter, it would leave a review
+comment on the change and set any approval votes that are configured.
+There are several reporting phases available; each of which may be
+configured with any number of reporters. See :ref:`drivers` for a
+full list of available reporters.
+
+The items enqueued into a pipeline are each associated with a git ref.
+That ref may point to a proposed change, or it may be the tip of a
+branch or a tag. The triggering event determines the ref, and whether
+it represents a proposed or merged commit. Zuul prepares the ref for
+an item before running jobs. In the case of a proposed change, that
+means speculatively merging the change into its target branch. This
+means that any jobs that operate on the change will run with the git
+repo in the state it will be in after the change merges (which may be
+substantially different than the git repo state of the change itself
+since the repo may have merged other changes since the change was
+originally authored). Items in a pipeline may depend on other items,
+and if they do, all of their dependent changes will be included in the
+git repo state that Zuul prepares. For more detail on this process,
+see :ref:`project_gating` and :ref:`dependencies`.
+
+The configuration for nearly everything described above is held in
+files inside of the git repos upon which Zuul operates. Zuul's
+configuration is global, but distributed. Jobs which are defined in
+one project might be used in another project while pipelines are
+available to all projects. When Zuul starts, it reads its
+configuration from all of the projects it knows about, and when
+changes to its configuration are proposed, those changes may take
+effect temporarily as part of the proposed change, or immediately
+after the change merges, depending on the type of project in which the
+change appears.
+
+Jobs specify the type and quantity of nodes which they require.
+Before executing each job, Zuul will contact it's companion program,
+Nodepool, to supply them. Nodepool may be configured to supply static
+nodes or contact cloud providers to create or delete nodes as
+necessary. The types of nodes available to Zuul are determined by the
+Nodepool administrator.
+
+The executable contents of jobs themselves are Ansible playbooks.
+Ansible's support for orchestrating tasks on remote nodes is
+particularly suited to Zuul's support for multi-node testing. Ansible
+is also easy to use for simple tasks (such as executing a shell
+script) or sophisticated deployment scenarios. When Zuul runs
+Ansible, it attempts to do so in a manner most similar to the way that
+Ansible might be used to orchestrate remote systems. Ansible itself
+is run on the executor and acts remotely upon the test nodes supplied
+to a job. This facilitates continuous delivery by making it possible
+to use the same Ansible playbooks in testing and production.
diff --git a/doc/source/project-config.rst b/doc/source/user/config.rst
similarity index 98%
rename from doc/source/project-config.rst
rename to doc/source/user/config.rst
index 7d72b30..e7226e9 100644
--- a/doc/source/project-config.rst
+++ b/doc/source/user/config.rst
@@ -33,7 +33,7 @@
---------------------
When Zuul starts, it examines all of the git repositories which are
-specified by the system administrator in :ref:`admin-config` and searches
+specified by the system administrator in :ref:`tenant-config` and searches
for files in the root of each repository. In the case of a
*config-project*, Zuul looks for a file named `zuul.yaml`. In the
case of an *untrusted-project*, Zuul looks first for `zuul.yaml` and
@@ -111,7 +111,7 @@
my_gerrit
verified: -1
-See TODO for more annotated examples of common pipeline configurations.
+.. TODO: See TODO for more annotated examples of common pipeline configurations.
The attributes available on a pipeline are as follows (all are
optional unless otherwise specified):
@@ -123,6 +123,8 @@
**manager** (required)
There are currently two schemes for managing pipelines:
+ .. _independent_pipeline_manager:
+
*independent*
Every event in this pipeline should be treated as independent of
other events in the pipeline. This is appropriate when the order of
@@ -138,6 +140,8 @@
pipeline. In that case, the changes have already merged, so the
results can not affect any other events in the pipeline.
+ .. _dependent_pipeline_manager:
+
*dependent*
The dependent pipeline manager is designed for gating. It ensures
that every change is tested exactly as it is going to be merged
@@ -190,7 +194,7 @@
Triggers are loaded from their connection name. The driver type of
the connection will dictate which options are available.
- See :doc:`triggers`.
+ See :ref:`drivers`.
**require**
If this section is present, it established pre-requisites for any
@@ -199,7 +203,7 @@
the conditions specified here must be met or the item will not be
enqueued.
-.. TODO this section is in flux in v3 _pipeline-require-approval:
+.. _pipeline-require-approval:
**approval**
This requires that a certain kind of approval be present for the
@@ -281,6 +285,13 @@
this to ``true``. This option is ignored by dependent pipelines.
The default is: ``false``.
+**precedence**
+ Indicates how the build scheduler should prioritize jobs for
+ different pipelines. Each pipeline may have one precedence, jobs
+ for pipelines with a higher precedence will be run before ones with
+ lower. The value should be one of ``high``, ``normal``, or ``low``.
+ Default: ``normal``.
+
The following options configure *reporters*. Reporters are
complementary to triggers; where a trigger is an event on a connection
which causes Zuul to enqueue an item, a reporter is the action
@@ -315,6 +326,11 @@
These reporters describe what Zuul should do when a pipeline is
disabled. See ``disable-after-consecutive-failures``.
+The following options can be used to alter Zuul's behavior to mitigate
+situations in which jobs are failing frequently (perhaps due to a
+problem with an external dependency, or unusually high
+non-deterministic test failures).
+
**disable-after-consecutive-failures**
If set, a pipeline can enter a ''disabled'' state if too many changes
in a row fail. When this value is exceeded the pipeline will stop
@@ -322,13 +338,6 @@
reporters and instead only report to the ``disabled`` reporters.
(No ``start`` reports are made when a pipeline is disabled).
-**precedence**
- Indicates how the build scheduler should prioritize jobs for
- different pipelines. Each pipeline may have one precedence, jobs
- for pipelines with a higher precedence will be run before ones with
- lower. The value should be one of ``high``, ``normal``, or ``low``.
- Default: ``normal``.
-
**window**
Dependent pipeline managers only. Zuul can rate limit dependent
pipelines in a manner similar to TCP flow control. Jobs are only
@@ -515,7 +524,7 @@
variants used in constructing the frozen job, with no duplication.
Default: none.
-** branches **
+**branches**
A regular expression (or list of regular expressions) which describe
on what branches a job should run (or in the case of variants: to
alter the behavior of a job for a certain branch).
@@ -691,6 +700,8 @@
be installed (and therefore referenced from Ansible), the `name`
attribute may be used to specify an alternate.
+ .. note:: galaxy roles are not yet implemented
+
**galaxy**
The name of the role in Ansible Galaxy. If this attribute is
supplied, Zuul will search Ansible Galaxy for a role by this name
diff --git a/doc/source/user/encryption.rst b/doc/source/user/encryption.rst
new file mode 100644
index 0000000..fdf2c5a
--- /dev/null
+++ b/doc/source/user/encryption.rst
@@ -0,0 +1,46 @@
+:title: Encryption
+
+.. _encryption:
+
+Encryption
+==========
+
+Zuul supports storing encrypted data directly in the git repositories
+of projects it operates on. If you have a job which requires private
+information in order to run (e.g., credentials to interact with a
+third-party service) those credentials can be stored along with the
+job definition.
+
+Each project in Zuul has its own automatically generated RSA keypair
+which can be used by anyone to encrypt a secret and only Zuul is able
+to decrypt it. Zuul serves each project's public key using its
+build-in webserver. They can be fetched at the path
+``/keys/<source>/<project>.pub`` where ``<project>`` is the name of a
+project and ``<source>`` is the name of that project's connection in
+the main Zuul configuration file.
+
+Zuul currently supports one encryption scheme, PKCS#1 with OAEP, which
+can not store secrets longer than the key length, 4096 bits. The
+padding used by this scheme ensures that someone examining the
+encrypted data can not determine the length of the plaintext version
+of the data, except to know that it is not longer than 4096 bits.
+
+In the config files themselves, Zuul uses an extensible method of
+specifying the encryption scheme used for a secret so that other
+schemes may be added later. To specify a secret, use the
+``!encrypted/pkcs1-oaep`` YAML tag along with the base64 encoded
+value. For example::
+
+ - secret:
+ name: test_secret
+ data:
+ password: !encrypted/pkcs1-oaep |
+ BFhtdnm8uXx7kn79RFL/zJywmzLkT1GY78P3bOtp4WghUFWobkifSu7ZpaV4NeO0s71YUsi1wGZZ
+ ...
+
+Zuul provides a standalone script to make encrypting values easy; it
+can be found at `tools/encrypt_secret.py` in the Zuul source
+directory.
+
+.. program-output:: python3 ../../tools/encrypt_secret.py --help
+
diff --git a/doc/source/gating.rst b/doc/source/user/gating.rst
similarity index 62%
rename from doc/source/gating.rst
rename to doc/source/user/gating.rst
index c10ba83..3398892 100644
--- a/doc/source/gating.rst
+++ b/doc/source/user/gating.rst
@@ -1,16 +1,18 @@
:title: Project Gating
+.. _project_gating:
+
Project Gating
==============
Traditionally, many software development projects merge changes from
developers into the repository, and then identify regressions
resulting from those changes (perhaps by running a test suite with a
-continuous integration system such as Jenkins), followed by more
-patches to fix those bugs. When the mainline of development is
-broken, it can be very frustrating for developers and can cause lost
-productivity, particularly so when the number of contributors or
-contributions is large.
+continuous integration system), followed by more patches to fix those
+bugs. When the mainline of development is broken, it can be very
+frustrating for developers and can cause lost productivity,
+particularly so when the number of contributors or contributions is
+large.
The process of gating attempts to prevent changes that introduce
regressions from being merged. This keeps the mainline of development
@@ -25,9 +27,6 @@
this process, with a particular emphasis on ensuring large numbers of
changes are tested correctly.
-Zuul was designed to handle the workflow of the OpenStack project, but
-can be used with any project.
-
Testing in parallel
-------------------
@@ -42,18 +41,17 @@
developers to create changes at a rate faster than they can be tested
and merged.
-Zuul's DependentPipelineManager allows for parallel execution of test
-jobs for gating while ensuring changes are tested correctly, exactly
-as if they had been tested one at a time. It does this by performing
-speculative execution of test jobs; it assumes that all jobs will
-succeed and tests them in parallel accordingly. If they do succeed,
-they can all be merged. However, if one fails, then changes that were
-expecting it to succeed are re-tested without the failed change. In
-the best case, as many changes as execution contexts are available may
-be tested in parallel and merged at once. In the worst case, changes
-are tested one at a time (as each subsequent change fails, changes
-behind it start again). In practice, the OpenStack project observes
-something closer to the best case.
+Zuul's :ref:`dependent pipeline manager<dependent_pipeline_manager>`
+allows for parallel execution of test jobs for gating while ensuring
+changes are tested correctly, exactly as if they had been tested one
+at a time. It does this by performing speculative execution of test
+jobs; it assumes that all jobs will succeed and tests them in parallel
+accordingly. If they do succeed, they can all be merged. However, if
+one fails, then changes that were expecting it to succeed are
+re-tested without the failed change. In the best case, as many
+changes as execution contexts are available may be tested in parallel
+and merged at once. In the worst case, changes are tested one at a
+time (as each subsequent change fails, changes behind it start again).
For example, if a core developer approves five changes in rapid
succession::
@@ -220,80 +218,34 @@
other projects currently enqueued in the gate (since they will
eventually be merged and might introduce breaking features).
-Such relationships can be defined in Zuul configuration by registering
-a job in a DependentPipeline of several projects. Whenever a change
-enters such a pipeline, it will create references for the other
-projects as well. As an example, given a main project ``acme`` and a
-plugin ``plugin`` you can define a job ``acme-tests`` which should be
-run for both projects:
+Such relationships can be defined in Zuul configuration by placing
+projects in a shared queue within a dependent pipeline. Whenever
+changes for any project enter a pipeline with such a shared queue,
+they are tested together, such that the commits for the changes ahead
+in the queue are automatically present in the jobs for the changes
+behind them. See :ref:`project` for more details.
-.. code-block:: yaml
+A given dependent pipeline may have as many shared change queues as
+necessary, so groups of related projects may share a change queue
+without interfering with unrelated projects. Independent pipelines do
+not use shared change queues, however, they may still be used to test
+changes across projects using cross-project dependencies.
- pipelines:
- - name: gate
- manager: DependentPipelineManager
+.. _dependencies:
- projects::
- - name: acme
- gate:
- - acme-tests
- - name: plugin
- gate:
- - acme-tests # Register job again
+Cross-Project Dependencies
+--------------------------
-Whenever a change enters the ``gate`` pipeline queue, Zuul creates a reference
-for it. For each subsequent change, an additional reference is created for the
-changes ahead in the queue. As a result, you will always be able to fetch the
-future state of your project dependencies for each change in the queue.
+Zuul permits users to specify dependencies across projects. Using a
+special footer in Git commit messages, users may specify that a change
+depends on another change in any repository known to Zuul.
-Based on the pipeline and project definitions above, three changes are
-inserted in the ``gate`` pipeline with the associated references:
+Zuul's cross-project dependencies behave like a directed acyclic graph
+(DAG), like git itself, to indicate a one-way dependency relationship
+between changes in different git repositories. Change A may depend on
+B, but B may not depend on A.
- ======== ======= ====== =========
- Change Project Branch Zuul Ref.
- ======== ======= ====== =========
- Change 1 acme master master/Z1
- Change 2 plugin stable stable/Z2
- Change 3 plugin master master/Z3
- ======== ======= ====== =========
-
-Since the changes enter a DependentPipelineManager pipeline, Zuul creates
-additional references:
-
- ====== ======= ========= =============================
- Change Project Zuul Ref. Description
- ====== ======= ========= =============================
- 1 acme master/Z1 acme master + change 1
- ------ ------- --------- -----------------------------
- 2 acme master/Z2 acme master + change 1
- 2 plugin stable/Z2 plugin stable + change 2
- ------ ------- --------- -----------------------------
- 3 acme master/Z3 acme master + change 1
- 3 plugin stable/Z3 plugin stable + change 2
- 3 plugin master/Z3 plugin master + change 3
- ====== ======= ========= =============================
-
-In order to test change 3, you would clone both repositories and simply
-fetch the Z3 reference for each combination of project/branch you are
-interested in testing. For example, you could fetch ``acme`` with
-master/Z3 and ``plugin`` with master/Z3 and thus have ``acme`` with
-change 1 applied as the expected state for when Change 3 would merge.
-When your job fetches several repositories without changes ahead in the
-queue, they may not have a Z reference in which case you can just check
-out the branch.
-
-
-Cross Repository Dependencies
------------------------------
-
-Zuul permits users to specify dependencies across repositories. Using
-a special header in Git commit messages, Users may specify that a
-change depends on another change in any repository known to Zuul.
-
-Zuul's cross-repository dependencies (CRD) behave like a directed
-acyclic graph (DAG), like git itself, to indicate a one-way dependency
-relationship between changes in different git repositories. Change A
-may depend on B, but B may not depend on A.
+.. TODO: update for v3 crd syntax
To use them, include ``Depends-On: <gerrit-change-id>`` in the footer of
a commit message. Use the full Change-ID ('I' + 40 characters).
@@ -302,10 +254,10 @@
Dependent Pipeline
~~~~~~~~~~~~~~~~~~
-When Zuul sees CRD changes, it serializes them in the usual manner when
-enqueuing them into a pipeline. This means that if change A depends on
-B, then when they are added to a dependent pipeline, B will appear first
-and A will follow:
+When Zuul sees changes with cross-project dependencies, it serializes
+them in the usual manner when enqueuing them into a pipeline. This
+means that if change A depends on B, then when they are added to a
+dependent pipeline, B will appear first and A will follow:
.. blockdiag::
:align: center
@@ -333,25 +285,26 @@
.. note::
- If changes with CRD do not share a change queue then Zuul is unable
- to enqueue them together, and the first will be required to merge
- before the second is enqueued.
+ If changes with cross-project dependencies do not share a change
+ queue then Zuul is unable to enqueue them together, and the first
+ will be required to merge before the second is enqueued.
Independent Pipeline
~~~~~~~~~~~~~~~~~~~~
When changes are enqueued into an independent pipeline, all of the
-related dependencies (both normal git-dependencies that come from parent
-commits as well as CRD changes) appear in a dependency graph, as in a
-dependent pipeline. This means that even in an independent pipeline,
-your change will be tested with its dependencies. So changes that were
-previously unable to be fully tested until a related change landed in a
-different repository may now be tested together from the start.
+related dependencies (both normal git-dependencies that come from
+parent commits as well as cross-project dependencies) appear in a
+dependency graph, as in a dependent pipeline. This means that even in
+an independent pipeline, your change will be tested with its
+dependencies. Changes that were previously unable to be fully tested
+until a related change landed in a different repository may now be
+tested together from the start.
-All of the changes are still independent (so you will note that the
-whole pipeline does not share a graph as in a dependent pipeline), but
-for each change tested, all of its dependencies are visually connected
-to it, and they are used to construct the git references that Zuul uses
+All of the changes are still independent (you will note that the whole
+pipeline does not share a graph as in a dependent pipeline), but for
+each change tested, all of its dependencies are visually connected to
+it, and they are used to construct the git repositories that Zuul uses
when testing.
When looking at this graph on the status page, you will note that the
@@ -383,6 +336,8 @@
additionally will appear as its own red or green dot for its test.
+.. TODO: relevant for v3?
+
Multiple Changes
~~~~~~~~~~~~~~~~
@@ -462,10 +417,12 @@
B, C <- A
}
+.. TODO: update for v3
+
Cycles
~~~~~~
-If a cycle is created by use of CRD, Zuul will abort its work very
-early. There will be no message in Gerrit and no changes that are part
-of the cycle will be enqueued into any pipeline. This is to protect
-Zuul from infinite loops.
+If a cycle is created by use of cross-project dependencies, Zuul will
+abort its work very early. There will be no message in Gerrit and no
+changes that are part of the cycle will be enqueued into any pipeline.
+This is to protect Zuul from infinite loops.
diff --git a/doc/source/user/index.rst b/doc/source/user/index.rst
new file mode 100644
index 0000000..3eca04b
--- /dev/null
+++ b/doc/source/user/index.rst
@@ -0,0 +1,18 @@
+User's Guide
+============
+
+This guide is for all users of Zuul. If you work on a project where
+Zuul is used to drive automation (whether that's testing proposed
+changes, building artifacts, or deploying builds), this guide will
+help you understand the concepts that underly Zuul, and how to
+configure it to meet your needs.
+
+
+.. toctree::
+ :maxdepth: 2
+
+ concepts
+ gating
+ config
+ jobs
+ encryption
diff --git a/doc/source/user/jobs.rst b/doc/source/user/jobs.rst
new file mode 100644
index 0000000..ad26327
--- /dev/null
+++ b/doc/source/user/jobs.rst
@@ -0,0 +1,90 @@
+:title: Job Content
+
+Job Content
+===========
+
+Zuul jobs are implemneted as Ansible playbooks. Zuul prepares the
+repositories used for a job, installs any required Ansible roles, and
+then executes the job's playbooks. Any setup or artifact collection
+required is the responsibility of the job itself. While this flexible
+arrangement allows for almost any kind of job to be run by Zuul,
+batteries are included. Zuul has a standard library of jobs upon
+which to build.
+
+Working Directory
+-----------------
+
+Before starting each job, the Zuul executor creates a directory to
+hold all of the content related to the job. This includes some
+directories which are used by Zuul to configure and run Ansible and
+may not be accessible, as well as a directory tree, under ``work/``,
+that is readable and writable by the job. The hierarchy is:
+
+**work/**
+ The working directory of the job.
+
+**work/src/**
+ Contains the prepared git repositories for the job.
+
+**work/logs/**
+ Where the Ansible log for the job is written; your job
+ may place other logs here as well.
+
+Git Repositories
+----------------
+
+The git repositories in ``work/src`` contain the repositories for all
+of the projects specified in the ``required-projects`` section of the
+job, plus the project associated with the queue item if it isn't
+already in that list. In the case of a proposed change, that change
+and all of the changes ahead of it in the pipeline queue will already
+be merged into their respective repositories and target branches. The
+change's project will have the change's branch checked out, as will
+all of the other projects, if that branch exists (otherwise, a
+fallback or default branch will be used). If your job needs to
+operate on multiple branches, simply checkout the appropriate branches
+of these git repos to ensure that the job results reflect the proposed
+future state that Zuul is testing, and all dependencies are present.
+Do not use any git remotes; the local repositories are guaranteed to
+be up to date.
+
+Note that these git repositories are located on the executor; in order
+to be useful to most kinds of jobs, they will need to be present on
+the test nodes. The ``base`` job in the standard library contains a
+pre-playbook which copies the repositories to all of the job's nodes.
+It is recommended to always inherit from this base job to ensure that
+behavior.
+
+.. TODO: link to base job documentation and/or document src (and logs?) directory
+
+Zuul Variables
+--------------
+
+Zuul supplies not only the variables specified by the job definition
+to Ansible, but also some variables from the executor itself. They
+are:
+
+**zuul.executor.hostname**
+ The hostname of the executor.
+
+**zuul.executor.src_root**
+ The path to the source directory.
+
+**zuul.executor.log_root**
+ The path to the logs directory.
+
+SSH Keys
+--------
+
+Zuul starts each job with an SSH agent running and the key used to
+access the job's nodes added to that agent. Generally you won't need
+to be aware of this since Ansible will use this when performing any
+tasks on remote nodes. However, under some circumstances you may want
+to interact with the agent. For example, you may wish to add a key
+provided as a secret to the job in order to access a specific host, or
+you may want to, in a pre-playbook, replace the key used to log into
+the assigned nodes in order to further protect it from being abused by
+untrusted job content.
+
+.. TODO: describe standard lib and link to published docs for it.
+
diff --git a/doc/source/zuul.rst b/doc/source/zuul.rst
deleted file mode 100644
index f07a859..0000000
--- a/doc/source/zuul.rst
+++ /dev/null
@@ -1,967 +0,0 @@
-:title: Zuul
-
-Zuul
-====
-
-Configuration
--------------
-
-Zuul has three configuration files:
-
-**zuul.conf**
- Connection information for Gerrit and Gearman, locations of the
- other config files. (required)
-**layout.yaml**
- Project and pipeline configuration -- what Zuul does. (required)
-**logging.conf**
- Python logging config. (optional)
-
-Examples of each of the three files can be found in the etc/ directory
-of the source distribution.
-
-.. _zuulconf:
-
-zuul.conf
-~~~~~~~~~
-
-Zuul will look for ``/etc/zuul/zuul.conf`` or ``~/zuul.conf`` to
-bootstrap its configuration. Alternately, you may specify ``-c
-/path/to/zuul.conf`` on the command line.
-
-Gerrit and Gearman connection information are each described in a
-section of zuul.conf. The location of the other two configuration
-files (as well as the location of the PID file when running Zuul as a
-server) are specified in a third section.
-
-The three sections of this config and their options are documented below.
-You can also find an example zuul.conf file in the git
-`repository
-<https://git.openstack.org/cgit/openstack-infra/zuul/tree/etc/zuul.conf-sample>`_
-
-gearman
-"""""""
-
-Client connection information for gearman. If using Zuul's builtin gearmand
-server just set **server** to 127.0.0.1.
-
-**server**
- Hostname or IP address of the Gearman server.
- ``server=gearman.example.com`` (required)
-
-**port**
- Port on which the Gearman server is listening.
- ``port=4730`` (optional)
-
-**ssl_ca**
- Optional: An openssl file containing a set of concatenated “certification authority” certificates
- in PEM formet.
-
-**ssl_cert**
- Optional: An openssl file containing the client public certificate in PEM format.
-
-**ssl_key**
- Optional: An openssl file containing the client private key in PEM format.
-
-gearman_server
-""""""""""""""
-
-The builtin gearman server. Zuul can fork a gearman process from itself rather
-than connecting to an external one.
-
-**start**
- Whether to start the internal Gearman server (default: False).
- ``start=true``
-
-**listen_address**
- IP address or domain name on which to listen (default: all addresses).
- ``listen_address=127.0.0.1``
-
-**log_config**
- Path to log config file for internal Gearman server.
- ``log_config=/etc/zuul/gearman-logging.yaml``
-
-**ssl_ca**
- Optional: An openssl file containing a set of concatenated “certification authority” certificates
- in PEM formet.
-
-**ssl_cert**
- Optional: An openssl file containing the server public certificate in PEM format.
-
-**ssl_key**
- Optional: An openssl file containing the server private key in PEM format.
-
-webapp
-""""""
-
-**listen_address**
- IP address or domain name on which to listen (default: 0.0.0.0).
- ``listen_address=127.0.0.1``
-
-**port**
- Port on which the webapp is listening (default: 8001).
- ``port=8008``
-
-zuul
-""""
-
-Zuul's main configuration section. At minimum zuul must be able to find
-layout.yaml to be useful.
-
-.. note:: Must be provided when running zuul-server
-
-.. _layout_config:
-
-**layout_config**
- Path to layout config file. Used by zuul-server only.
- ``layout_config=/etc/zuul/layout.yaml``
-
-**log_config**
- Path to log config file. Used by zuul-server only.
- ``log_config=/etc/zuul/logging.yaml``
-
-**pidfile**
- Path to PID lock file. Used by zuul-server only.
- ``pidfile=/var/run/zuul/zuul.pid``
-
-**state_dir**
- Path to directory that Zuul should save state to. Used by all Zuul
- commands.
- ``state_dir=/var/lib/zuul``
-
-**jobroot_dir**
- Path to directory that Zuul should store temporary job files.
- ``jobroot_dir=/tmp``
-
-**report_times**
- Boolean value (``true`` or ``false``) that determines if Zuul should
- include elapsed times for each job in the textual report. Used by
- zuul-server only.
- ``report_times=true``
-
-**status_url**
- URL that will be posted in Zuul comments made to Gerrit changes when
- starting jobs for a change. Used by zuul-server only.
- ``status_url=https://zuul.example.com/status``
-
-**status_expiry**
- Zuul will cache the status.json file for this many seconds. This is an
- optional value and ``1`` is used by default.
- ``status_expiry=1``
-
-
-merger
-""""""
-
-The zuul-merger process configuration. Detailed documentation on this process
-can be found on the :doc:`merger` page.
-
-.. note:: Must be provided when running zuul-merger. Both services may share the
- same configuration (and even host) or otherwise have an individual
- zuul.conf.
-
-**git_dir**
- Directory that Zuul should clone local git repositories to.
- ``git_dir=/var/lib/zuul/git``
-
-**git_user_email**
- Optional: Value to pass to `git config user.email`.
- ``git_user_email=zuul@example.com``
-
-**git_user_name**
- Optional: Value to pass to `git config user.name`.
- ``git_user_name=zuul``
-
-**zuul_url**
- URL of this merger's git repos, accessible to test workers. Usually
- "http://zuul.example.com/p" or "http://zuul-merger01.example.com/p"
- depending on whether the merger is co-located with the Zuul server.
-
-**log_config**
- Path to log config file for the merger process.
- ``log_config=/etc/zuul/logging.yaml``
-
-**pidfile**
- Path to PID lock file for the merger process.
- ``pidfile=/var/run/zuul-merger/merger.pid``
-
-executor
-""""""""
-
-The zuul-executor process configuration.
-
-**finger_port**
- Port to use for finger log streamer.
- ``finger_port=79``
-
-**git_dir**
- Directory that Zuul should clone local git repositories to.
- ``git_dir=/var/lib/zuul/git``
-
-**log_config**
- Path to log config file for the executor process.
- ``log_config=/etc/zuul/logging.yaml``
-
-**private_key_file**
- SSH private key file to be used when logging into worker nodes.
- ``private_key_file=~/.ssh/id_rsa``
-
-**user**
- User ID for the zuul-executor process. In normal operation as a daemon,
- the executor should be started as the ``root`` user, but it will drop
- privileges to this user during startup.
- ``user=zuul``
-
-.. _connection:
-
-connection ArbitraryName
-""""""""""""""""""""""""
-
-A connection can be listed with any arbitrary name. The required
-parameters are specified in the :ref:`connections` documentation
-depending on what driver you are using.
-
-.. _layoutyaml:
-
-layout.yaml
-~~~~~~~~~~~
-
-This is the main configuration file for Zuul, where all of the pipelines
-and projects are defined, what tests should be run, and what actions
-Zuul should perform. There are three sections: pipelines, jobs, and
-projects.
-
-Pipelines
-"""""""""
-
-Zuul can have any number of independent pipelines. Whenever a matching
-Gerrit event is found for a pipeline, that event is added to the
-pipeline, and the jobs specified for that pipeline are run. When all
-jobs specified for the pipeline that were triggered by an event are
-completed, Zuul reports back to Gerrit the results.
-
-There are no pre-defined pipelines in Zuul, rather you can define
-whatever pipelines you need in the layout file. This is a very flexible
-system that can accommodate many kinds of workflows.
-
-Here is a quick example of a pipeline definition followed by an
-explanation of each of the parameters::
-
- - name: check
- manager: IndependentPipelineManager
- source: my_gerrit
- trigger:
- my_gerrit:
- - event: patchset-created
- success:
- my_gerrit:
- verified: 1
- failure:
- my_gerrit
- verified: -1
-
-**name**
- This is used later in the project definition to indicate what jobs
- should be run for events in the pipeline.
-
-**description**
- This is an optional field that may be used to provide a textual
- description of the pipeline.
-
-**source**
- A required field that specifies a connection that provides access to
- the change objects that this pipeline operates on. The name of the
- connection as per the zuul.conf should be specified. The driver used
- for the connection named will be the source. Currently only ``gerrit``
- drivers are supported.
-
-**success-message**
- An optional field that supplies the introductory text in message
- reported back to Gerrit when all the voting builds are successful.
- Defaults to "Build successful."
-
-**failure-message**
- An optional field that supplies the introductory text in message
- reported back to Gerrit when at least one voting build fails.
- Defaults to "Build failed."
-
-**merge-failure-message**
- An optional field that supplies the introductory text in message
- reported back to Gerrit when a change fails to merge with the
- current state of the repository.
- Defaults to "Merge failed."
-
-**footer-message**
- An optional field to supply additional information after test results.
- Useful for adding information about the CI system such as debugging
- and contact details.
-
-**manager**
- There are currently two schemes for managing pipelines:
-
- *IndependentPipelineManager*
- Every event in this pipeline should be treated as independent of
- other events in the pipeline. This is appropriate when the order of
- events in the pipeline doesn't matter because the results of the
- actions this pipeline performs can not affect other events in the
- pipeline. For example, when a change is first uploaded for review,
- you may want to run tests on that change to provide early feedback
- to reviewers. At the end of the tests, the change is not going to
- be merged, so it is safe to run these tests in parallel without
- regard to any other changes in the pipeline. They are independent.
-
- Another type of pipeline that is independent is a post-merge
- pipeline. In that case, the changes have already merged, so the
- results can not affect any other events in the pipeline.
-
- *DependentPipelineManager*
- The dependent pipeline manager is designed for gating. It ensures
- that every change is tested exactly as it is going to be merged
- into the repository. An ideal gating system would test one change
- at a time, applied to the tip of the repository, and only if that
- change passed tests would it be merged. Then the next change in
- line would be tested the same way. In order to achieve parallel
- testing of changes, the dependent pipeline manager performs
- speculative execution on changes. It orders changes based on
- their entry into the pipeline. It begins testing all changes in
- parallel, assuming that each change ahead in the pipeline will pass
- its tests. If they all succeed, all the changes can be tested and
- merged in parallel. If a change near the front of the pipeline
- fails its tests, each change behind it ignores whatever tests have
- been completed and are tested again without the change in front.
- This way gate tests may run in parallel but still be tested
- correctly, exactly as they will appear in the repository when
- merged.
-
- One important characteristic of the DependentPipelineManager is that
- it analyzes the jobs that are triggered by different projects, and
- if those projects have jobs in common, it treats those projects as
- related, and they share a single virtual queue of changes. Thus,
- if there is a job that performs integration testing on two
- projects, those two projects will automatically share a virtual
- change queue. If a third project does not invoke that job, it
- will be part of a separate virtual change queue, and changes to
- it will not depend on changes to the first two jobs.
-
- For more detail on the theory and operation of Zuul's
- DependentPipelineManager, see: :doc:`gating`.
-
-**trigger**
- At least one trigger source must be supplied for each pipeline.
- Triggers are not exclusive -- matching events may be placed in
- multiple pipelines, and they will behave independently in each of
- the pipelines they match.
-
- Triggers are loaded from their connection name. The driver type of
- the connection will dictate which options are available.
- See :doc:`triggers`.
-
-**require**
- If this section is present, it established pre-requisites for any
- kind of item entering the Pipeline. Regardless of how the item is
- to be enqueued (via any trigger or automatic dependency resolution),
- the conditions specified here must be met or the item will not be
- enqueued.
-
-.. _pipeline-require-approval:
-
- **approval**
- This requires that a certain kind of approval be present for the
- current patchset of the change (the approval could be added by the
- event in question). It takes several sub-parameters, all of which
- are optional and are combined together so that there must be an
- approval matching all specified requirements.
-
- *username*
- If present, an approval from this username is required. It is
- treated as a regular expression.
-
- *email*
- If present, an approval with this email address is required. It
- is treated as a regular expression.
-
- *email-filter* (deprecated)
- A deprecated alternate spelling of *email*. Only one of *email* or
- *email_filter* should be used.
-
- *older-than*
- If present, the approval must be older than this amount of time
- to match. Provide a time interval as a number with a suffix of
- "w" (weeks), "d" (days), "h" (hours), "m" (minutes), "s"
- (seconds). Example ``48h`` or ``2d``.
-
- *newer-than*
- If present, the approval must be newer than this amount of time
- to match. Same format as "older-than".
-
- Any other field is interpreted as a review category and value
- pair. For example ``verified: 1`` would require that the approval
- be for a +1 vote in the "Verified" column. The value may either
- be a single value or a list: ``verified: [1, 2]`` would match
- either a +1 or +2 vote.
-
- **open**
- A boolean value (``true`` or ``false``) that indicates whether the change
- must be open or closed in order to be enqueued.
-
- **current-patchset**
- A boolean value (``true`` or ``false``) that indicates whether the change
- must be the current patchset in order to be enqueued.
-
- **status**
- A string value that corresponds with the status of the change
- reported by the trigger. For example, when using the Gerrit
- trigger, status values such as ``NEW`` or ``MERGED`` may be useful.
-
-**reject**
- If this section is present, it establishes pre-requisites that can
- block an item from being enqueued. It can be considered a negative
- version of **require**.
-
- **approval**
- This takes a list of approvals. If an approval matches the provided
- criteria the change can not be entered into the pipeline. It follows
- the same syntax as the :ref:`"require approval" pipeline above
- <pipeline-require-approval>`.
-
- Example to reject a change with any negative vote::
-
- reject:
- approval:
- - code-review: [-1, -2]
-
-**dequeue-on-new-patchset**
- Normally, if a new patchset is uploaded to a change that is in a
- pipeline, the existing entry in the pipeline will be removed (with
- jobs canceled and any dependent changes that can no longer merge as
- well. To suppress this behavior (and allow jobs to continue
- running), set this to ``false``. Default: ``true``.
-
-**ignore-dependencies**
- In any kind of pipeline (dependent or independent), Zuul will
- attempt to enqueue all dependencies ahead of the current change so
- that they are tested together (independent pipelines report the
- results of each change regardless of the results of changes ahead).
- To ignore dependencies completely in an independent pipeline, set
- this to ``true``. This option is ignored by dependent pipelines.
- The default is: ``false``.
-
-**success**
- Describes where Zuul should report to if all the jobs complete
- successfully.
- This section is optional; if it is omitted, Zuul will run jobs and
- do nothing on success; it will not even report a message to Gerrit.
- If the section is present, the listed reporter plugins will be
- asked to report on the jobs.
- The reporters are listed by their connection name. The options
- available depend on the driver for the supplied connection.
- See :doc:`reporters` for more details.
-
-**failure**
- Uses the same syntax as **success**, but describes what Zuul should
- do if at least one job fails.
-
-**merge-failure**
- Uses the same syntax as **success**, but describes what Zuul should
- do if it is unable to merge in the patchset. If no merge-failure
- reporters are listed then the ``failure`` reporters will be used to
- notify of unsuccessful merges.
-
-**start**
- Uses the same syntax as **success**, but describes what Zuul should
- do when a change is added to the pipeline manager. This can be used,
- for example, to reset the value of the Verified review category.
-
-**disabled**
- Uses the same syntax as **success**, but describes what Zuul should
- do when a pipeline is disabled.
- See ``disable-after-consecutive-failures``.
-
-**disable-after-consecutive-failures**
- If set, a pipeline can enter a ''disabled'' state if too many changes
- in a row fail. When this value is exceeded the pipeline will stop
- reporting to any of the ``success``, ``failure`` or ``merge-failure``
- reporters and instead only report to the ``disabled`` reporters.
- (No ``start`` reports are made when a pipeline is disabled).
-
-**precedence**
- Indicates how the build scheduler should prioritize jobs for
- different pipelines. Each pipeline may have one precedence, jobs
- for pipelines with a higher precedence will be run before ones with
- lower. The value should be one of ``high``, ``normal``, or ``low``.
- Default: ``normal``.
-
-**window**
- DependentPipelineManagers only. Zuul can rate limit
- DependentPipelineManagers in a manner similar to TCP flow control.
- Jobs are only started for changes in the queue if they sit in the
- actionable window for the pipeline. The initial length of this window
- is configurable with this value. The value given should be a positive
- integer value. A value of ``0`` disables rate limiting on the
- DependentPipelineManager.
- Default: ``20``.
-
-**window-floor**
- DependentPipelineManagers only. This is the minimum value for the
- window described above. Should be a positive non zero integer value.
- Default: ``3``.
-
-**window-increase-type**
- DependentPipelineManagers only. This value describes how the window
- should grow when changes are successfully merged by zuul. A value of
- ``linear`` indicates that ``window-increase-factor`` should be added
- to the previous window value. A value of ``exponential`` indicates
- that ``window-increase-factor`` should be multiplied against the
- previous window value and the result will become the window size.
- Default: ``linear``.
-
-**window-increase-factor**
- DependentPipelineManagers only. The value to be added or multiplied
- against the previous window value to determine the new window after
- successful change merges.
- Default: ``1``.
-
-**window-decrease-type**
- DependentPipelineManagers only. This value describes how the window
- should shrink when changes are not able to be merged by Zuul. A value
- of ``linear`` indicates that ``window-decrease-factor`` should be
- subtracted from the previous window value. A value of ``exponential``
- indicates that ``window-decrease-factor`` should be divided against
- the previous window value and the result will become the window size.
- Default: ``exponential``.
-
-**window-decrease-factor**
- DependentPipelineManagers only. The value to be subtracted or divided
- against the previous window value to determine the new window after
- unsuccessful change merges.
- Default: ``2``.
-
-Some example pipeline configurations are included in the sample layout
-file. The first is called a *check* pipeline::
-
- - name: check
- manager: IndependentPipelineManager
- trigger:
- my_gerrit:
- - event: patchset-created
- success:
- my_gerrit:
- verified: 1
- failure:
- my_gerrit:
- verified: -1
-
-This will trigger jobs each time a new patchset (or change) is
-uploaded to Gerrit, and report +/-1 values to Gerrit in the
-``verified`` review category. ::
-
- - name: gate
- manager: DependentPipelineManager
- trigger:
- my_gerrit:
- - event: comment-added
- approval:
- - approved: 1
- success:
- my_gerrit:
- verified: 2
- submit: true
- failure:
- my_gerrit:
- verified: -2
-
-This will trigger jobs whenever a reviewer leaves a vote of ``1`` in the
-``approved`` review category in Gerrit (a non-standard category).
-Changes will be tested in such a way as to guarantee that they will be
-merged exactly as tested, though that will happen in parallel by
-creating a virtual queue of dependent changes and performing
-speculative execution of jobs. ::
-
- - name: post
- manager: IndependentPipelineManager
- trigger:
- my_gerrit:
- - event: ref-updated
- ref: ^(?!refs/).*$
-
-This will trigger jobs whenever a change is merged to a named branch
-(e.g., ``master``). No output will be reported to Gerrit. This is
-useful for side effects such as creating per-commit tarballs. ::
-
- - name: silent
- manager: IndependentPipelineManager
- trigger:
- my_gerrit:
- - event: patchset-created
-
-This also triggers jobs when changes are uploaded to Gerrit, but no
-results are reported to Gerrit. This is useful for jobs that are in
-development and not yet ready to be presented to developers. ::
-
- pipelines:
- - name: post-merge
- manager: IndependentPipelineManager
- trigger:
- my_gerrit:
- - event: change-merged
- success:
- my_gerrit:
- force-message: True
- failure:
- my_gerrit:
- force-message: True
-
-The ``change-merged`` events happen when a change has been merged in the git
-repository. The change is thus closed and Gerrit will not accept modifications
-to the review scoring such as ``code-review`` or ``verified``. By using the
-``force-message: True`` parameter, Zuul will pass ``--force-message`` to the
-``gerrit review`` command, thus making sure the message is actually
-sent back to Gerrit regardless of approval scores.
-That kind of pipeline is nice to run regression or performance tests.
-
-.. note::
- The ``change-merged`` event does not include the commit sha1 which can be
- hazardous, it would let you report back to Gerrit though. If you were to
- build a tarball for a specific commit, you should consider instead using
- the ``ref-updated`` event which does include the commit sha1 (but lacks the
- Gerrit change number).
-
-
-.. _jobs:
-
-Jobs
-""""
-
-The jobs section is optional, and can be used to set attributes of
-jobs that are independent of their association with a project. For
-example, if a job should return a customized message on failure, that
-may be specified here. Otherwise, Zuul does not need to be told about
-each job as it builds a list from the project specification.
-
-**name**
- The name of the job. This field is treated as a regular expression
- and will be applied to each job that matches.
-
-**queue-name (optional)**
- Zuul will automatically combine projects that share a job into
- shared change queues for dependent pipeline managers. In order to
- report statistics about these queues, it is convenient for them to
- have names. Zuul can automatically name change queues, however
- these can grow quite long and are prone to changing as projects in
- the queue change. If you assign a queue-name to a job, Zuul will
- use that as the name for the shared change queue that contains that
- job instead of the automatically generated one. It is an error for
- a shared change queue to have more than one job with a queue-name if
- they are not the same.
-
-**failure-message (optional)**
- The message that should be reported to Gerrit if the job fails.
-
-**success-message (optional)**
- The message that should be reported to Gerrit if the job fails.
-
-**failure-pattern (optional)**
- The URL that should be reported to Gerrit if the job fails.
- Defaults to the build URL or the url_pattern configured in
- zuul.conf. May be supplied as a string pattern with substitutions
- as described in url_pattern in :ref:`zuulconf`.
-
-**success-pattern (optional)**
- The URL that should be reported to Gerrit if the job succeeds.
- Defaults to the build URL or the url_pattern configured in
- zuul.conf. May be supplied as a string pattern with substitutions
- as described in url_pattern in :ref:`zuulconf`.
-
-**hold-following-changes (optional)**
- This is a boolean that indicates that changes that follow this
- change in a dependent change pipeline should wait until this job
- succeeds before executing. If this is applied to a very short job
- that can predict whether longer jobs will fail early, this can be
- used to reduce the number of jobs that Zuul will execute and
- ultimately have to cancel. In that case, a small amount of
- parallelization of jobs is traded for more efficient use of testing
- resources. On the other hand, to apply this to a long running job
- would largely defeat the parallelization of dependent change testing
- that is the main feature of Zuul. Default: ``false``.
-
-**semaphore (optional)**
- This is a string that names a semaphore that should be observed by this
- job. The semaphore defines how many jobs which reference that semaphore
- can be enqueued at a time. This applies across all pipelines in the same
- tenant. The max value of the semaphore can be specified in the config
- repositories and defaults to 1.
-
-**branch (optional)**
- This job should only be run on matching branches. This field is
- treated as a regular expression and multiple branches may be
- listed.
-
-**files (optional)**
- This job should only be run if at least one of the files involved in
- the change (added, deleted, or modified) matches at least one of the
- file patterns listed here. This field is treated as a regular
- expression and multiple expressions may be listed.
-
-**skip-if (optional)**
-
- This job should not be run if all the patterns specified by the
- optional fields listed below match on their targets. When multiple
- sets of parameters are provided, this job will be skipped if any set
- matches. For example: ::
-
- jobs:
- - name: check-tempest-dsvm-neutron
- skip-if:
- - project: ^openstack/neutron$
- branch: ^stable/juno$
- all-files-match-any:
- - ^neutron/tests/.*$
- - ^tools/.*$
- - all-files-match-any:
- - ^doc/.*$
- - ^.*\.rst$
-
- With this configuration, the job would be skipped for a neutron
- patchset for the stable/juno branch provided that every file in the
- change matched at least one of the specified file regexes. The job
- will also be skipped for any patchset that modified only the doc
- tree or rst files.
-
- *project* (optional)
- The regular expression to match against the project of the change.
-
- *branch* (optional)
- The regular expression to match against the branch or ref of the
- change.
-
- *all-files-match-any* (optional)
- A list of regular expressions intended to match the files involved
- in the change. This parameter will be considered matching a
- change only if all files in a change match at least one of these
- expressions.
-
- The pattern for '/COMMIT_MSG' is always matched on and does not
- have to be included. Exception is merge commits (without modified
- files), in this case '/COMMIT_MSG' is not matched, and job is not
- skipped. In case of merge commits it's assumed that list of modified
- files isn't predictible and CI should be run.
-
-**voting (optional)**
- Boolean value (``true`` or ``false``) that indicates whatever
- a job is voting or not. Default: ``true``.
-
-**attempts (optional)**
- Number of attempts zuul will execute a job. Once reached, zuul will report
- RETRY_LIMIT as the job result.
- Defaults to 3.
-
-**tags (optional)**
- A list of arbitrary strings which will be associated with the job.
-
-Here is an example of setting the failure message for jobs that check
-whether a change merges cleanly::
-
- - name: ^.*-merge$
- failure-message: This change or one of its cross-repo dependencies
- was unable to be automatically merged with the current state of
- its repository. Please rebase the change and upload a new
- patchset.
-
-Projects
-""""""""
-
-The projects section indicates what jobs should be run in each pipeline
-for events associated with each project. It contains a list of
-projects. Here is an example::
-
- - name: example/project
- check:
- - project-merge:
- - project-unittest
- - project-pep8
- - project-pyflakes
- gate:
- - project-merge:
- - project-unittest
- - project-pep8
- - project-pyflakes
- post:
- - project-publish
-
-**name**
- The name of the project (as known by Gerrit).
-
-**merge-mode (optional)**
- An optional value that indicates what strategy should be used to
- merge changes to this project. Supported values are:
-
- ** merge-resolve **
- Equivalent to 'git merge -s resolve'. This corresponds closely to
- what Gerrit performs (using JGit) for a project if the "Merge if
- necessary" merge mode is selected and "Automatically resolve
- conflicts" is checked. This is the default.
-
- ** merge **
- Equivalent to 'git merge'.
-
- ** cherry-pick **
- Equivalent to 'git cherry-pick'.
-
-This is followed by a section for each of the pipelines defined above.
-Pipelines may be omitted if no jobs should run for this project in a
-given pipeline. Within the pipeline section, the jobs that should be
-executed are listed. If a job is entered as a dictionary key, then
-jobs contained within that key are only executed if the key job
-succeeds. In the above example, project-unittest, project-pep8, and
-project-pyflakes are only executed if project-merge succeeds.
-Furthermore, project-finaltest is executed only if project-unittest,
-project-pep8 and project-pyflakes all succeed. This can help avoid
-running unnecessary jobs while maximizing parallelism. It is also
-useful when distributing results between jobs.
-
-The special job named ``noop`` is internal to Zuul and will always
-return ``SUCCESS`` immediately. This can be useful if you require
-that all changes be processed by a pipeline but a project has no jobs
-that can be run on it.
-
-.. seealso:: The OpenStack Zuul configuration for a comprehensive example: https://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml
-
-Project Templates
-"""""""""""""""""
-
-Whenever you have lot of similar projects (such as plugins for a project) you
-will most probably want to use the same pipeline configurations. The
-project templates let you define pipelines and job name templates to trigger.
-One can then just apply the template on its project which make it easier to
-update several similar projects. As an example::
-
- project-templates:
- # Name of the template
- - name: plugin-triggering
- # Definition of pipelines just like for a `project`
- check:
- - '{jobprefix}-merge':
- - '{jobprefix}-pep8'
- - '{jobprefix}-pyflakes'
- gate:
- - '{jobprefix}-merge':
- - '{jobprefix}-unittest'
- - '{jobprefix}-pep8'
- - '{jobprefix}-pyflakes'
-
-In your projects definition, you will then apply the template using the template
-key::
-
- projects:
- - name: plugin/foobar
- template:
- - name: plugin-triggering
- jobprefix: plugin-foobar
-
-You can pass several parameters to a template. A ``parameter`` value
-will be used for expansion of ``{parameter}`` in the template
-strings. The parameter ``name`` will be automatically provided and
-will contain the short name of the project, that is the portion of the
-project name after the last ``/`` character.
-
-Multiple templates can be combined in a project, and the jobs from all
-of those templates will be added to the project. Individual jobs may
-also be added::
-
- projects:
- - name: plugin/foobar
- template:
- - name: plugin-triggering
- jobprefix: plugin-foobar
- - name: plugin-extras
- jobprefix: plugin-foobar
- check:
- - foobar-extra-special-job
-
-Individual jobs may optionally be added to pipelines (e.g. check,
-gate, et cetera) for a project, in addition to those provided by
-templates.
-
-The order of the jobs listed in the project (which only affects the
-order of jobs listed on the report) will be the jobs from each
-template in the order listed, followed by any jobs individually listed
-for the project.
-
-Note that if multiple templates are used for a project and one
-template specifies a job that is also specified in another template,
-or specified in the project itself, the configuration defined by
-either the last template or the project itself will take priority.
-
-
-Semaphores
-""""""""""
-
-When using semaphores the maximum value of each one can be specified in their
-respective config repositories. Unspecified semaphores default to 1::
-
- - semaphore:
- name: semaphore-foo
- max: 5
- - semaphore:
- name: semaphore-bar
- max: 3
-
-
-logging.conf
-~~~~~~~~~~~~
-This file is optional. If provided, it should be a standard
-:mod:`logging.config` module configuration file. If not present, Zuul will
-output all log messages of DEBUG level or higher to the console.
-
-Starting Zuul
--------------
-
-To start Zuul, run **zuul-server**::
-
- usage: zuul-server [-h] [-c CONFIG] [-l LAYOUT] [-d] [-t] [--version]
-
- Project gating system.
-
- optional arguments:
- -h, --help show this help message and exit
- -c CONFIG specify the config file
- -l LAYOUT specify the layout file
- -d do not run as a daemon
- -t validate layout file syntax
- --version show zuul version
-
-You may want to use the ``-d`` argument while you are initially setting
-up Zuul so you can detect any configuration errors quickly. Under
-normal operation, omit ``-d`` and let Zuul run as a daemon.
-
-If you send signal 1 (SIGHUP) to the zuul-server process, Zuul will
-stop executing new jobs, wait until all executing jobs are finished,
-reload its layout.yaml, and resume. Changes to any connections or
-the PID file will be ignored until Zuul is restarted.
-
-If you send a SIGUSR1 to the zuul-server process, Zuul will stop
-executing new jobs, wait until all executing jobs are finished,
-then exit. While waiting to exit Zuul will queue Gerrit events and
-save these events prior to exiting. When Zuul starts again it will
-read these saved events and act on them.
-
-If you need to abort Zuul and intend to manually requeue changes for
-jobs which were running in its pipelines, prior to terminating you can
-use the zuul-changes.py tool script to simplify the process. For
-example, this would give you a list of zuul-enqueue commands to requeue
-changes for the gate and check pipelines respectively::
-
- ./tools/zuul-changes.py http://zuul.openstack.org/ gate
- ./tools/zuul-changes.py http://zuul.openstack.org/ check
-
-If you send a SIGUSR2 to the zuul-server process, or the forked process
-that runs the Gearman daemon, Zuul will dump a stack trace for each
-running thread into its debug log. It is written under the log bucket
-``zuul.stack_dump``. This is useful for tracking down deadlock or
-otherwise slow threads.
-
-When `yappi <https://code.google.com/p/yappi/>`_ (Yet Another Python
-Profiler) is available, additional functions' and threads' stats are
-emitted as well. The first SIGUSR2 will enable yappi, on the second
-SIGUSR2 it dumps the information collected, resets all yappi state and
-stops profiling. This is to minimize the impact of yappi on a running
-system.