Merge "Re-enable test_zuul_trigger_project_change_merged" into feature/zuulv3
diff --git a/.zuul.yaml b/.zuul.yaml
index ede4391..041681a 100644
--- a/.zuul.yaml
+++ b/.zuul.yaml
@@ -17,29 +17,16 @@
name: zuul-stream-functional
parent: multinode
nodeset: zuul-functional
- pre-run: playbooks/zuul-stream/pre
- run: playbooks/zuul-stream/functional
+ pre-run: playbooks/zuul-stream/pre.yaml
+ run: playbooks/zuul-stream/functional.yaml
post-run:
- - playbooks/zuul-stream/post
- - playbooks/zuul-stream/post-ara
+ - playbooks/zuul-stream/post.yaml
+ - playbooks/zuul-stream/post-ara.yaml
required-projects:
- openstack/ara
files:
- - "zuul/ansible/callback/.*"
- - "playbooks/zuul-stream/.*"
-
-- job:
- name: zuul-migrate
- parent: unittests
- run: playbooks/zuul-migrate/run
- post-run: playbooks/zuul-migrate/post
- # We're adding zuul to the required-projects so that we can also trigger
- # this from project-config changes
- required-projects:
- - openstack-infra/openstack-zuul-jobs
- - openstack-infra/project-config
- - name: openstack-infra/zuul
- override-branch: feature/zuulv3
+ - zuul/ansible/callback/.*
+ - playbooks/zuul-stream/.*
- project:
name: openstack-infra/zuul
@@ -60,10 +47,6 @@
- zuul/cmd/migrate.py
- playbooks/zuul-migrate/.*
- zuul-stream-functional
- - zuul-migrate:
- files:
- - zuul/cmd/migrate.py
- - playbooks/zuul-migrate/.*
gate:
jobs:
- build-openstack-sphinx-docs:
diff --git a/doc/source/admin/client.rst b/doc/source/admin/client.rst
index 961b205..8ee6b6f 100644
--- a/doc/source/admin/client.rst
+++ b/doc/source/admin/client.rst
@@ -40,6 +40,62 @@
Note that the format of change id is <number>,<patchset>.
+Enqueue-ref
+^^^^^^^^^^^
+
+.. program-output:: zuul enqueue-ref --help
+
+This command is provided to manually simulate a trigger from an
+external source. It can be useful for testing or replaying a trigger
+that is difficult or impossible to recreate at the source. The
+arguments to ``enqueue-ref`` will vary depending on the source and
+type of trigger. Some familiarity with the arguments emitted by
+``gerrit`` `update hooks
+<https://gerrit-review.googlesource.com/admin/projects/plugins/hooks>`__
+such as ``patchset-created`` and ``ref-updated`` is recommended. Some
+examples of common operations are provided below.
+
+Manual enqueue examples
+***********************
+
+It is common to have a ``release`` pipeline that listens for new tags
+coming from ``gerrit`` and performs a range of code packaging jobs.
+If there is an unexpected issue in the release jobs, the same tag can
+not be recreated in ``gerrit`` and the user must either tag a new
+release or request a manual re-triggering of the jobs. To re-trigger
+the jobs, pass the failed tag as the ``ref`` argument and set
+``newrev`` to the change associated with the tag in the project
+repository (i.e. what you see from ``git show X.Y.Z``)::
+
+ zuul enqueue-ref --tenant openstack --trigger gerrit --pipeline release --project openstack/example_project --ref refs/tags/X.Y.Z --newrev abc123...
+
+The command can also be used asynchronosly trigger a job in a
+``periodic`` pipeline that would usually be run at a specific time by
+the ``timer`` driver. For example, the following command would
+trigger the ``periodic`` jobs against the current ``master`` branch
+top-of-tree for a project::
+
+ zuul enqueue-ref --tenant openstack --trigger timer --pipeline periodic --project openstack/example_project --ref refs/heads/master
+
+Another common pipeline is a ``post`` queue listening for ``gerrit``
+merge results. Triggering here is slightly more complicated as you
+wish to recreate the full ``ref-updated`` event from ``gerrit``. For
+a new commit on ``master``, the gerrit ``ref-updated`` trigger
+expresses "reset ``refs/heads/master`` for the project from ``oldrev``
+to ``newrev``" (``newrev`` being the committed change). Thus to
+replay the event, you could ``git log`` in the project and take the
+current ``HEAD`` and the prior change, then enqueue the event::
+
+ NEW_REF=$(git rev-parse HEAD)
+ OLD_REF=$(git rev-parse HEAD~1)
+
+ zuul enqueue-ref --tenant openstack --trigger gerrit --pipeline post --project openstack/example_project --ref refs/heads/master --newrev $NEW_REF --oldrev $OLD_REF
+
+Note that zero values for ``oldrev`` and ``newrev`` can indicate
+branch creation and deletion; the source code is the best reference
+for these more advanced operations.
+
+
Promote
^^^^^^^
.. program-output:: zuul promote --help
diff --git a/doc/source/admin/components.rst b/doc/source/admin/components.rst
index 99817f7..b3c2e44 100644
--- a/doc/source/admin/components.rst
+++ b/doc/source/admin/components.rst
@@ -522,6 +522,15 @@
The executor will observe system load and determine whether
to accept more jobs every 30 seconds.
+ .. attr:: hostname
+ :default: hostname of the server
+
+ The executor needs to know its hostname under which it is reachable by
+ zuul-web. Otherwise live console log streaming doesn't work. In most cases
+ This is automatically detected correctly. But when running in environments
+ where it cannot determine its hostname correctly this can be overridden
+ here.
+
.. attr:: merger
.. attr:: git_user_email
@@ -592,6 +601,12 @@
Base URL on which the websocket service is exposed, if different
than the base URL of the web app.
+ .. attr:: static_cache_expiry
+ :default: 3600
+
+ The Cache-Control max-age response header value for static files served
+ by the zuul-web. Set to 0 during development to disable Cache-Control.
+
Operation
~~~~~~~~~
diff --git a/doc/source/admin/drivers/gerrit.rst b/doc/source/admin/drivers/gerrit.rst
index ac42bd3..935cb32 100644
--- a/doc/source/admin/drivers/gerrit.rst
+++ b/doc/source/admin/drivers/gerrit.rst
@@ -61,6 +61,17 @@
Path to Gerrit web interface.
+ .. attr:: gitweb_url_template
+ :default: {baseurl}/gitweb?p={project.name}.git;a=commitdiff;h={sha}
+
+ Url template for links to specific git shas. By default this will
+ point at Gerrit's built in gitweb but you can customize this value
+ to point elsewhere (like cgit or github).
+
+ The three values available for string interpolation are baseurl
+ which points back to Gerrit, project and all of its safe attributes,
+ and sha which is the git sha1.
+
.. attr:: user
:default: zuul
diff --git a/doc/source/admin/drivers/sql.rst b/doc/source/admin/drivers/sql.rst
index e208467..a269f5d 100644
--- a/doc/source/admin/drivers/sql.rst
+++ b/doc/source/admin/drivers/sql.rst
@@ -28,6 +28,21 @@
<http://docs.sqlalchemy.org/en/rel_1_0/core/engines.html#database-urls>`_
for more information.
+ The driver will automatically set up the database creating and managing
+ the necesssary tables. Therefore the provided user should have sufficient
+ permissions to manage the database. For example:
+
+ .. code-block:: sql
+
+ GRANT ALL ON my_database TO 'my_user'@'%';
+
+ .. attr:: pool_recycle
+ :default: 1
+
+ Tune the pool_recycle value. See `The SQLAlchemy manual on pooling
+ <http://docs.sqlalchemy.org/en/latest/core/pooling.html#setting-pool-recycle>`_
+ for more information.
+
Reporter Configuration
----------------------
diff --git a/doc/source/admin/monitoring.rst b/doc/source/admin/monitoring.rst
index 55f1908..e6e6139 100644
--- a/doc/source/admin/monitoring.rst
+++ b/doc/source/admin/monitoring.rst
@@ -224,6 +224,49 @@
The number of outstanding nodepool requests from Zuul.
+.. stat:: zuul.mergers
+
+ Holds metrics related to Zuul mergers.
+
+ .. stat:: online
+ :type: gauge
+
+ The number of Zuul merger processes online.
+
+ .. stat:: jobs_running
+ :type: gauge
+
+ The number of merge jobs running.
+
+ .. stat:: jobs_queued
+ :type: gauge
+
+ The number of merge jobs queued.
+
+.. stat:: zuul.executors
+
+ Holds metrics related to Zuul executors.
+
+ .. stat:: online
+ :type: gauge
+
+ The number of Zuul executor processes online.
+
+ .. stat:: accepting
+ :type: gauge
+
+ The number of Zuul executor processes accepting new jobs.
+
+ .. stat:: jobs_running
+ :type: gauge
+
+ The number of executor jobs running.
+
+ .. stat:: jobs_queued
+ :type: gauge
+
+ The number of executor jobs queued.
+
As an example, given a job named `myjob` in `mytenant` triggered by a
change to `myproject` on the `master` branch in the `gate` pipeline
diff --git a/doc/source/user/config.rst b/doc/source/user/config.rst
index c4014e2..3ea20ab 100644
--- a/doc/source/user/config.rst
+++ b/doc/source/user/config.rst
@@ -448,9 +448,7 @@
a base job. Each tenant has a default parent job which will be used
if no explicit parent is specified.
-Jobs also support a concept called variance. The first time a job
-definition appears is called the reference definition of the job.
-Subsequent job definitions with the same name are called variants.
+Multiple job definitions with the same name are called variants.
These may have different selection criteria which indicate to Zuul
that, for instance, the job should behave differently on a different
git branch. Unlike inheritance, all job variants must be defined in
@@ -635,7 +633,7 @@
- job:
name: run-tests
- branch: stable/2.0
+ branches: stable/2.0
nodeset: old-release
In some cases, Zuul uses an implied value for the branch
@@ -645,13 +643,11 @@
branch specifier is used. If no branch specifier appears, the
job applies to all branches.
- * In the case of an :term:`untrusted-project`, no implied branch
- specifier is applied to the reference definition of a job.
- That is to say, that if the first appearance of the job
- definition appears without a branch specifier, then it will
- apply to all branches. Note that when collecting its
- configuration, Zuul reads the ``master`` branch of a given
- project first, then other branches in alphabetical order.
+ * In the case of an :term:`untrusted-project`, if the project
+ has only one branch, no implied branch specifier is applied to
+ :ref:`job` definitions. If the project has more than one
+ branch, the branch containing the job definition is used as an
+ implied branch specifier.
* In the case of a job variant defined within a :ref:`project`,
if the project definition is in a :term:`config-project`, no
@@ -662,19 +658,27 @@
* In the case of a job variant defined within a
:ref:`project-template`, if no branch specifier appears, the
- implied branch specifier for the :ref:`project` definition which
- uses the project-template will be used.
+ implied branch containing the project-template definition is
+ used as an implied branch specifier. This means that
+ definitions of the same project-template on different branches
+ may run different jobs.
- * Any further job variants other than the reference definition
- in an untrusted-project will, if they do not have a branch
- specifier, have an implied branch specifier for the current
- branch applied.
+ When that project-template is used by a :ref:`project`
+ definition within a :term:`untrusted-project`, the branch
+ containing that project definition is combined with the branch
+ specifier of the project-template. This means it is possible
+ for a project to use a template on one branch, but not on
+ another.
This allows for the very simple and expected workflow where if a
project defines a job on the ``master`` branch with no branch
specifier, and then creates a new branch based on ``master``,
any changes to that job definition within the new branch only
- affect that branch.
+ affect that branch, and likewise, changes to the master branch
+ only affect it.
+
+ See :attr:`pragma.implied-branch-matchers` for how to override
+ this behavior on a per-file basis.
.. attr:: files
@@ -749,7 +753,7 @@
If a job has an empty or no nodeset definition, it will still
run and may be able to perform actions on the Zuul executor.
- .. attr:: override-branch
+ .. attr:: override-checkout
When Zuul runs jobs for a proposed change, it normally checks
out the branch associated with that change on every project
@@ -757,13 +761,13 @@
branch tip or tag), then that ref is normally checked out. This
attribute is used to override that behavior and indicate that
this job should, regardless of the branch for the queue item,
- use the indicated branch instead. This can be used, for
- example, to run a previous version of the software (from a
- stable maintenance branch) under test even if the change being
- tested applies to a different branch (this is only likely to be
- useful if there is some cross-branch interaction with some
+ use the indicated ref (i.e., branch or tag) instead. This can
+ be used, for example, to run a previous version of the software
+ (from a stable maintenance branch) under test even if the change
+ being tested applies to a different branch (this is only likely
+ to be useful if there is some cross-branch interaction with some
component of the system being tested). See also the
- project-specific :attr:`job.required-projects.override-branch`
+ project-specific :attr:`job.required-projects.override-checkout`
attribute to apply this behavior to a subset of a job's
projects.
@@ -786,42 +790,55 @@
.. attr:: pre-run
- The name of a playbook or list of playbooks without file
- extension to run before the main body of a job. The full path
- to the playbook in the repo where the job is defined is
- expected.
+ The name of a playbook or list of playbooks to run before the
+ main body of a job. The full path to the playbook in the repo
+ where the job is defined is expected.
When a job inherits from a parent, the child's pre-run playbooks
are run after the parent's. See :ref:`job` for more
information.
+ .. warning::
+
+ If the path as specified does not exist, Zuul will try
+ appending the extensions ``.yaml`` and ``.yml``. This
+ behavior is deprecated and will be removed in the future all
+ playbook paths should include the file extension.
+
.. attr:: post-run
- The name of a playbook or list of playbooks without file
- extension to run after the main body of a job. The full path to
- the playbook in the repo where the job is defined is expected.
+ The name of a playbook or list of playbooks to run after the
+ main body of a job. The full path to the playbook in the repo
+ where the job is defined is expected.
When a job inherits from a parent, the child's post-run
playbooks are run before the parent's. See :ref:`job` for more
information.
+ .. warning::
+
+ If the path as specified does not exist, Zuul will try
+ appending the extensions ``.yaml`` and ``.yml``. This
+ behavior is deprecated and will be removed in the future all
+ playbook paths should include the file extension.
+
.. attr:: run
- The name of the main playbook for this job. This parameter is
- not normally necessary, as it defaults to a playbook with the
- same name as the job inside of the ``playbooks/`` directory
- (e.g., the ``foo`` job would default to ``playbooks/foo``.
- However, if a playbook with a different name is needed, it can
- be specified here. The file extension is not required, but the
- full path within the repo is. When a child inherits from a
- parent, a playbook with the name of the child job is implicitly
- searched first, before falling back on the playbook used by the
- parent job (unless the child job specifies a ``run`` attribute,
- in which case that value is used). Example:
+ The name of the main playbook for this job. If it is not
+ supplied, the parent's playbook will be used (and likewise up
+ the inheritance chain). The full path within the repo is
+ required. Example:
.. code-block:: yaml
- run: playbooks/<name of the job>
+ run: playbooks/job-playbook.yaml
+
+ .. warning::
+
+ If the path as specified does not exist, Zuul will try
+ appending the extensions ``.yaml`` and ``.yml``. This
+ behavior is deprecated and will be removed in the future all
+ playbook paths should include the file extension.
.. attr:: roles
@@ -907,7 +924,7 @@
The name of the required project.
- .. attr:: override-branch
+ .. attr:: override-checkout
When Zuul runs jobs for a proposed change, it normally checks
out the branch associated with that change on every project
@@ -915,9 +932,10 @@
branch tip or tag), then that ref is normally checked out.
This attribute is used to override that behavior and indicate
that this job should, regardless of the branch for the queue
- item, use the indicated branch instead, for only this
- project. See also the :attr:`job.override-branch` attribute
- to apply the same behavior to all projects in a job.
+ item, use the indicated ref (i.e., branch or tag) instead,
+ for only this project. See also the
+ :attr:`job.override-checkout` attribute to apply the same
+ behavior to all projects in a job.
.. attr:: vars
@@ -973,6 +991,12 @@
project-pipeline definition is what determines how a project
participates in a pipeline.
+Multiple project definitions may appear for the same project (for
+example, in a central :term:`config projects <config-project>` as well
+as in a repo's own ``.zuul.yaml``). In this case, all of the project
+definitions are combined (the jobs listed in all of the definitions
+will be run).
+
Consider the following project definition::
- project:
@@ -1019,8 +1043,7 @@
.. attr:: project
- In addition to a project-pipeline definition for one or more
- pipelines, the following attributes may appear in a project:
+ The following attributes may appear in a project:
.. attr:: name
:required:
@@ -1278,3 +1301,41 @@
:default: 1
The maximum number of running jobs which can use this semaphore.
+
+.. _pragma:
+
+Pragma
+~~~~~~
+
+The `pragma` item does not behave like the others. It can not be
+included or excluded from configuration loading by the administrator,
+and does not form part of the final configuration itself. It is used
+to alter how the configuration is processed while loading.
+
+A pragma item only affects the current file. The same file in another
+branch of the same project will not be affected, nor any other files
+or any other projects. The effect is global within that file --
+pragma directives may not be set and then unset within the same file.
+
+.. code-block:: yaml
+
+ - pragma:
+ implied-branch-matchers: False
+
+.. attr:: pragma
+
+ The pragma item currently only supports one attribute:
+
+ .. attr:: implied-branch-matchers
+
+ This is a boolean, which, if set, may be used to enable
+ (``True``) or disable (``False``) the addition of implied branch
+ matchers to job definitions. Normally Zuul decides whether to
+ add these based on heuristics described in :attr:`job.branches`.
+ This attribute overrides that behavior.
+
+ This can be useful if a project has multiple branches, yet the
+ jobs defined in the master branch should apply to all branches.
+
+ Note that if a job contains an explicit branch matcher, it will
+ be used regardless of the value supplied here.
diff --git a/doc/source/user/jobs.rst b/doc/source/user/jobs.rst
index 6962b8f..f65ee19 100644
--- a/doc/source/user/jobs.rst
+++ b/doc/source/user/jobs.rst
@@ -83,9 +83,12 @@
* Job variables
+* Parent job results
+
Meaning that a site-wide variable with the same name as any other will
override its value, and similarly, secrets override job variables of
-the same name. Each of the three sources is described below.
+the same name which override data returned from parent jobs. Each of
+the sources is described below.
Job Variables
@@ -249,6 +252,13 @@
A boolean indicating whether this project appears in the
:attr:`job.required-projects` list for this job.
+ .. var:: _projects
+ :type: dict
+
+ The same as ``projects`` but a dictionary indexed by the
+ ``name`` value of each entry. ``projects`` will be converted to
+ this.
+
.. var:: tenant
The name of the current Zuul tenant.
@@ -475,6 +485,11 @@
<admin_sitewide_variables>` for information on how a site
administrator may define these variables.
+Parent Job Results
+~~~~~~~~~~~~~~~~~~
+
+A job may return data to Zuul for later use by jobs which depend on
+it. For details, see :ref:`return_values`.
SSH Keys
--------
@@ -496,9 +511,9 @@
Return Values
-------------
-The job may return some values to Zuul to affect its behavior. To
-return a value, use the *zuul_return* Ansible module in a job
-playbook. For example:
+A job may return some values to Zuul to affect its behavior and for
+use by other jobs.. To return a value, use the ``zuul_return``
+Ansible module in a job playbook. For example:
.. code-block:: yaml
@@ -507,12 +522,11 @@
data:
foo: bar
-Will return the dictionary "{'foo': 'bar'}" to Zuul.
+Will return the dictionary ``{'foo': 'bar'}`` to Zuul.
.. TODO: xref to section describing formatting
-Several uses of these values are planned, but the only currently
-implemented use is to set the log URL for a build. To do so, set the
+To set the log URL for a build, use *zuul_return* to set the
**zuul.log_url** value. For example:
.. code-block:: yaml
@@ -522,3 +536,10 @@
data:
zuul:
log_url: http://logs.example.com/path/to/build/logs
+
+Any values other than those in the ``zuul`` hierarchy will be supplied
+as Ansible variables to child jobs. These variables have less
+precedence than any other type of variable in Zuul, so be sure their
+names are not shared by any job variables. If more than one parent
+job returns the same variable, the value from the later job in the job
+graph will take precedence.
diff --git a/etc/status/public_html/jquery.zuul.js b/etc/status/public_html/jquery.zuul.js
index 3735004..50dbed5 100644
--- a/etc/status/public_html/jquery.zuul.js
+++ b/etc/status/public_html/jquery.zuul.js
@@ -53,6 +53,7 @@
'msg_id': '#zuul_msg',
'pipelines_id': '#zuul_pipelines',
'queue_events_num': '#zuul_queue_events_num',
+ 'queue_management_events_num': '#zuul_queue_management_events_num',
'queue_results_num': '#zuul_queue_results_num',
}, options);
@@ -713,6 +714,10 @@
data.trigger_event_queue ?
data.trigger_event_queue.length : '0'
);
+ $(options.queue_management_events_num).text(
+ data.management_event_queue ?
+ data.management_event_queue.length : '0'
+ );
$(options.queue_results_num).text(
data.result_event_queue ?
data.result_event_queue.length : '0'
diff --git a/etc/status/public_html/zuul.app.js b/etc/status/public_html/zuul.app.js
index ae950e8..7ceb2dd 100644
--- a/etc/status/public_html/zuul.app.js
+++ b/etc/status/public_html/zuul.app.js
@@ -33,7 +33,7 @@
+ '<div class="zuul-container" id="zuul-container">'
+ '<div style="display: none;" class="alert" id="zuul_msg"></div>'
+ '<button class="btn pull-right zuul-spinner">updating <span class="glyphicon glyphicon-refresh"></span></button>'
- + '<p>Queue lengths: <span id="zuul_queue_events_num">0</span> events, <span id="zuul_queue_results_num">0</span> results.</p>'
+ + '<p>Queue lengths: <span id="zuul_queue_events_num">0</span> events, <span id="zuul_queue_management_events_num">0</span> management events, <span id="zuul_queue_results_num">0</span> results.</p>'
+ '<div id="zuul_controls"></div>'
+ '<div id="zuul_pipelines" class="row"></div>'
+ '<p>Zuul version: <span id="zuul-version-span"></span></p>'
diff --git a/etc/zuul.conf-sample b/etc/zuul.conf-sample
index 76494ad..f0e1765 100644
--- a/etc/zuul.conf-sample
+++ b/etc/zuul.conf-sample
@@ -37,6 +37,7 @@
[web]
listen_address=127.0.0.1
port=9000
+static_cache_expiry=0
[webapp]
listen_address=0.0.0.0
diff --git a/playbooks/zuul-stream/fixtures/test-stream.yaml b/playbooks/zuul-stream/fixtures/test-stream.yaml
index fd28757..c4946e8 100644
--- a/playbooks/zuul-stream/fixtures/test-stream.yaml
+++ b/playbooks/zuul-stream/fixtures/test-stream.yaml
@@ -46,3 +46,6 @@
args:
chdir: /itemloop/somewhere/that/does/not/exist
failed_when: false
+
+ - name: Print binary data
+ command: echo -e '\x80abc'
diff --git a/playbooks/zuul-stream/functional.yaml b/playbooks/zuul-stream/functional.yaml
index 6b67b05..779a102 100644
--- a/playbooks/zuul-stream/functional.yaml
+++ b/playbooks/zuul-stream/functional.yaml
@@ -58,3 +58,8 @@
shell: |
egrep "^.+\| node1 \| OSError.+\/failure-itemloop\/" job-output.txt
egrep "^.+\| node2 \| OSError.+\/failure-itemloop\/" job-output.txt
+
+ - name: Validate output - binary data
+ shell: |
+ egrep "^.*\| node1 \| \\\\x80abc" job-output.txt
+ egrep "^.*\| node2 \| \\\\x80abc" job-output.txt
diff --git a/tests/base.py b/tests/base.py
index a02ee5a..036515d 100755
--- a/tests/base.py
+++ b/tests/base.py
@@ -58,7 +58,6 @@
import zuul.driver.github.githubconnection as githubconnection
import zuul.scheduler
import zuul.webapp
-import zuul.rpclistener
import zuul.executor.server
import zuul.executor.client
import zuul.lib.connections
@@ -68,6 +67,7 @@
import zuul.model
import zuul.nodepool
import zuul.zk
+import zuul.configloader
from zuul.exceptions import MergeFailure
FIXTURE_DIR = os.path.join(os.path.dirname(__file__),
@@ -499,7 +499,7 @@
"refUpdate": {
"oldRev": oldrev,
"newRev": repo.heads[branch].commit.hexsha,
- "refName": branch,
+ "refName": 'refs/heads/' + branch,
"project": project,
}
}
@@ -1713,6 +1713,8 @@
image_id=None,
host_keys=["fake-key1", "fake-key2"],
executor='fake-nodepool')
+ if 'fakeuser' in node_type:
+ data['username'] = 'fakeuser'
data = json.dumps(data).encode('utf8')
path = self.client.create(path, data,
makepath=True,
@@ -2073,6 +2075,7 @@
gerritconnection.GerritEventConnector.delay = 0.0
self.sched = zuul.scheduler.Scheduler(self.config)
+ self.sched._stats_interval = 1
self.webapp = zuul.webapp.WebApp(
self.sched, port=0, listen_address='127.0.0.1')
@@ -2114,11 +2117,8 @@
self.sched.setNodepool(self.nodepool)
self.sched.setZooKeeper(self.zk)
- self.rpc = zuul.rpclistener.RPCListener(self.config, self.sched)
-
self.sched.start()
self.webapp.start()
- self.rpc.start()
self.executor_client.gearman.waitForServer()
# Cleanups are run in reverse order
self.addCleanup(self.assertCleanShutdown)
@@ -2232,8 +2232,14 @@
files={'README': ''},
branch='master', tag='init')
if 'job' in item:
- jobname = item['job']['name']
- files['playbooks/%s.yaml' % jobname] = ''
+ if 'run' in item['job']:
+ files['%s.yaml' % item['job']['run']] = ''
+ for fn in zuul.configloader.as_list(
+ item['job'].get('pre-run', [])):
+ files['%s.yaml' % fn] = ''
+ for fn in zuul.configloader.as_list(
+ item['job'].get('post-run', [])):
+ files['%s.yaml' % fn] = ''
root = os.path.join(self.test_root, "config")
if not os.path.exists(root):
@@ -2374,8 +2380,6 @@
self.statsd.join()
self.webapp.stop()
self.webapp.join()
- self.rpc.stop()
- self.rpc.join()
self.gearman_server.shutdown()
self.fake_nodepool.stop()
self.zk.disconnect()
@@ -2388,6 +2392,7 @@
'pydevd.CommandThread',
'pydevd.Reader',
'pydevd.Writer',
+ 'FingerStreamer',
]
threads = [t for t in threading.enumerate()
if t.name not in whitelist]
diff --git a/tests/encrypt_secret.py b/tests/encrypt_secret.py
index 0b0cf19..4bc514a 100644
--- a/tests/encrypt_secret.py
+++ b/tests/encrypt_secret.py
@@ -12,6 +12,7 @@
# License for the specific language governing permissions and limitations
# under the License.
+import base64
import sys
import os
@@ -27,8 +28,10 @@
private_key, public_key = \
encryption.deserialize_rsa_keypair(f.read())
- ciphertext = encryption.encrypt_pkcs1_oaep(sys.argv[1], public_key)
- print(ciphertext.encode('base64'))
+ plaintext = sys.argv[1].encode('utf-8')
+
+ ciphertext = encryption.encrypt_pkcs1_oaep(plaintext, public_key)
+ print(base64.b64encode(ciphertext).decode('utf-8'))
if __name__ == '__main__':
diff --git a/tests/fixtures/config/ansible/git/common-config/zuul.yaml b/tests/fixtures/config/ansible/git/common-config/zuul.yaml
index 67d1c70..28bfce1 100644
--- a/tests/fixtures/config/ansible/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/ansible/git/common-config/zuul.yaml
@@ -79,10 +79,11 @@
failure-url: https://failure.example.com/zuul-logs/{build.uuid}/
- job:
- parent: base-urls
name: python27
- pre-run: playbooks/pre
- post-run: playbooks/post
+ parent: base-urls
+ run: playbooks/python27.yaml
+ pre-run: playbooks/pre.yaml
+ post-run: playbooks/post.yaml
vars:
flagpath: '{{zuul._test.test_root}}/{{zuul.build}}.flag'
roles:
@@ -93,11 +94,13 @@
- job:
parent: python27
name: timeout
+ run: playbooks/timeout.yaml
timeout: 1
- job:
parent: python27
name: check-vars
+ run: playbooks/check-vars.yaml
nodeset:
nodes:
- name: ubuntu-xenial
@@ -113,6 +116,7 @@
- job:
parent: python27
name: check-secret-names
+ run: playbooks/check-secret-names.yaml
nodeset:
nodes:
- name: ubuntu-xenial
@@ -124,9 +128,11 @@
- job:
parent: base-urls
name: hello
+ run: playbooks/hello-post.yaml
post-run: playbooks/hello-post
- job:
parent: python27
name: failpost
+ run: playbooks/post-broken.yaml
post-run: playbooks/post-broken
diff --git a/tests/fixtures/config/ansible/git/org_project/.zuul.yaml b/tests/fixtures/config/ansible/git/org_project/.zuul.yaml
index e144325..447f6cd 100644
--- a/tests/fixtures/config/ansible/git/org_project/.zuul.yaml
+++ b/tests/fixtures/config/ansible/git/org_project/.zuul.yaml
@@ -1,10 +1,12 @@
- job:
parent: python27
name: faillocal
+ run: playbooks/faillocal.yaml
- job:
parent: hello
name: hello-world
+ run: playbooks/hello-world.yaml
- project:
name: org/project
diff --git a/tests/fixtures/config/base-jobs/git/org_project/.zuul.yaml b/tests/fixtures/config/base-jobs/git/org_project/.zuul.yaml
index 9844c14..a7e3bdb 100644
--- a/tests/fixtures/config/base-jobs/git/org_project/.zuul.yaml
+++ b/tests/fixtures/config/base-jobs/git/org_project/.zuul.yaml
@@ -1,9 +1,11 @@
- job:
name: my-job
+ run: playbooks/my-job.yaml
- job:
name: other-job
parent: other-base
+ run: playbooks/other-job.yaml
- project:
name: org/project
diff --git a/tests/fixtures/config/branch-templates/git/project-config/zuul.yaml b/tests/fixtures/config/branch-templates/git/project-config/zuul.yaml
new file mode 100644
index 0000000..ce08877
--- /dev/null
+++ b/tests/fixtures/config/branch-templates/git/project-config/zuul.yaml
@@ -0,0 +1,26 @@
+- pipeline:
+ name: check
+ manager: independent
+ trigger:
+ gerrit:
+ - event: patchset-created
+ success:
+ gerrit:
+ Verified: 1
+ failure:
+ gerrit:
+ Verified: -1
+
+- job:
+ name: base
+ parent: null
+
+- project:
+ name: project-config
+ check:
+ jobs: []
+
+- project:
+ name: puppet-integration
+ check:
+ jobs: []
diff --git a/tests/fixtures/config/branch-templates/git/puppet-integration/.zuul.yaml b/tests/fixtures/config/branch-templates/git/puppet-integration/.zuul.yaml
new file mode 100644
index 0000000..dfea632
--- /dev/null
+++ b/tests/fixtures/config/branch-templates/git/puppet-integration/.zuul.yaml
@@ -0,0 +1,25 @@
+- job:
+ name: puppet-unit-base
+ run: playbooks/run-unit-tests.yaml
+
+- job:
+ name: puppet-unit-3.8
+ parent: puppet-unit-base
+ branches: ^(stable/(newton|ocata)).*$
+ vars:
+ puppet_gem_version: 3.8
+
+- job:
+ name: puppet-something
+ run: playbooks/run-unit-tests.yaml
+
+- project-template:
+ name: puppet-unit
+ check:
+ jobs:
+ - puppet-unit-3.8
+
+- project:
+ name: puppet-integration
+ templates:
+ - puppet-unit
diff --git a/tests/fixtures/config/branch-templates/git/puppet-integration/README b/tests/fixtures/config/branch-templates/git/puppet-integration/README
new file mode 100644
index 0000000..9daeafb
--- /dev/null
+++ b/tests/fixtures/config/branch-templates/git/puppet-integration/README
@@ -0,0 +1 @@
+test
diff --git a/tests/fixtures/config/branch-templates/git/puppet-integration/playbooks/run-unit-tests.yaml b/tests/fixtures/config/branch-templates/git/puppet-integration/playbooks/run-unit-tests.yaml
new file mode 100644
index 0000000..f679dce
--- /dev/null
+++ b/tests/fixtures/config/branch-templates/git/puppet-integration/playbooks/run-unit-tests.yaml
@@ -0,0 +1,2 @@
+- hosts: all
+ tasks: []
diff --git a/tests/fixtures/config/branch-templates/git/puppet-tripleo/.zuul.yaml b/tests/fixtures/config/branch-templates/git/puppet-tripleo/.zuul.yaml
new file mode 100644
index 0000000..4be8146
--- /dev/null
+++ b/tests/fixtures/config/branch-templates/git/puppet-tripleo/.zuul.yaml
@@ -0,0 +1,4 @@
+- project:
+ name: puppet-tripleo
+ templates:
+ - puppet-unit
diff --git a/tests/fixtures/config/branch-templates/git/puppet-tripleo/README b/tests/fixtures/config/branch-templates/git/puppet-tripleo/README
new file mode 100644
index 0000000..9daeafb
--- /dev/null
+++ b/tests/fixtures/config/branch-templates/git/puppet-tripleo/README
@@ -0,0 +1 @@
+test
diff --git a/tests/fixtures/config/branch-templates/main.yaml b/tests/fixtures/config/branch-templates/main.yaml
new file mode 100644
index 0000000..f7677a3
--- /dev/null
+++ b/tests/fixtures/config/branch-templates/main.yaml
@@ -0,0 +1,9 @@
+- tenant:
+ name: tenant-one
+ source:
+ gerrit:
+ config-projects:
+ - project-config
+ untrusted-projects:
+ - puppet-integration
+ - puppet-tripleo
diff --git a/tests/fixtures/config/branch-variants/git/project-config/zuul.yaml b/tests/fixtures/config/branch-variants/git/project-config/zuul.yaml
index 89d98a9..161e5a1 100644
--- a/tests/fixtures/config/branch-variants/git/project-config/zuul.yaml
+++ b/tests/fixtures/config/branch-variants/git/project-config/zuul.yaml
@@ -11,6 +11,26 @@
gerrit:
Verified: -1
+- pipeline:
+ name: gate
+ manager: dependent
+ trigger:
+ gerrit:
+ - event: comment-added
+ approval:
+ - Approved: 1
+ success:
+ gerrit:
+ Verified: 2
+ submit: true
+ failure:
+ gerrit:
+ Verified: -2
+ start:
+ gerrit:
+ Verified: 0
+ precedence: high
+
- job:
name: base
parent: null
@@ -23,3 +43,14 @@
name: project-config
check:
jobs: []
+ gate:
+ jobs:
+ - noop
+
+- project:
+ name: puppet-integration
+ check:
+ jobs: []
+ gate:
+ jobs:
+ - noop
diff --git a/tests/fixtures/config/branch-variants/git/puppet-integration/.zuul.yaml b/tests/fixtures/config/branch-variants/git/puppet-integration/.zuul.yaml
index 2545208..322927f 100644
--- a/tests/fixtures/config/branch-variants/git/puppet-integration/.zuul.yaml
+++ b/tests/fixtures/config/branch-variants/git/puppet-integration/.zuul.yaml
@@ -11,6 +11,8 @@
name: puppet-lint
parent: puppet-module-base
run: playbooks/run-lint
+ tags:
+ - master
- project-template:
name: puppet-check-jobs
diff --git a/tests/fixtures/config/branch-variants/git/puppet-integration/stable.zuul.yaml b/tests/fixtures/config/branch-variants/git/puppet-integration/stable.zuul.yaml
new file mode 100644
index 0000000..4701b80
--- /dev/null
+++ b/tests/fixtures/config/branch-variants/git/puppet-integration/stable.zuul.yaml
@@ -0,0 +1,26 @@
+- job:
+ name: puppet-base
+ pre-run: playbooks/prepare-node-common
+
+- job:
+ name: puppet-module-base
+ parent: puppet-base
+ pre-run: playbooks/prepare-node-unit
+
+- job:
+ name: puppet-lint
+ parent: puppet-module-base
+ run: playbooks/run-lint
+ tags:
+ - stable
+
+- project-template:
+ name: puppet-check-jobs
+ check:
+ jobs:
+ - puppet-lint
+
+- project:
+ name: puppet-integration
+ templates:
+ - puppet-check-jobs
diff --git a/tests/fixtures/config/central-jobs/git/central-jobs/README b/tests/fixtures/config/central-jobs/git/central-jobs/README
new file mode 100644
index 0000000..9daeafb
--- /dev/null
+++ b/tests/fixtures/config/central-jobs/git/central-jobs/README
@@ -0,0 +1 @@
+test
diff --git a/tests/fixtures/config/central-jobs/git/central-jobs/playbooks/central-job.yaml b/tests/fixtures/config/central-jobs/git/central-jobs/playbooks/central-job.yaml
new file mode 100644
index 0000000..f679dce
--- /dev/null
+++ b/tests/fixtures/config/central-jobs/git/central-jobs/playbooks/central-job.yaml
@@ -0,0 +1,2 @@
+- hosts: all
+ tasks: []
diff --git a/tests/fixtures/config/central-jobs/git/central-jobs/zuul.yaml b/tests/fixtures/config/central-jobs/git/central-jobs/zuul.yaml
new file mode 100644
index 0000000..2bf782e
--- /dev/null
+++ b/tests/fixtures/config/central-jobs/git/central-jobs/zuul.yaml
@@ -0,0 +1,9 @@
+- job:
+ name: central-job
+ run: playbooks/central-job.yaml
+
+- project-template:
+ name: central-jobs
+ check:
+ jobs:
+ - central-job
diff --git a/tests/fixtures/config/central-jobs/git/common-config/zuul.yaml b/tests/fixtures/config/central-jobs/git/common-config/zuul.yaml
new file mode 100644
index 0000000..c31af45
--- /dev/null
+++ b/tests/fixtures/config/central-jobs/git/common-config/zuul.yaml
@@ -0,0 +1,52 @@
+- pipeline:
+ name: check
+ manager: independent
+ trigger:
+ gerrit:
+ - event: patchset-created
+ success:
+ gerrit:
+ Verified: 1
+ failure:
+ gerrit:
+ Verified: -1
+
+- pipeline:
+ name: gate
+ manager: dependent
+ trigger:
+ gerrit:
+ - event: comment-added
+ approval:
+ - Approved: 1
+ success:
+ gerrit:
+ Verified: 2
+ submit: true
+ failure:
+ gerrit:
+ Verified: -2
+ start:
+ gerrit:
+ Verified: 0
+ precedence: high
+
+- job:
+ name: base
+ parent: null
+
+- project:
+ name: common-config
+ check:
+ jobs: []
+ gate:
+ jobs:
+ - noop
+
+- project:
+ name: org/project
+ check:
+ jobs: []
+ gate:
+ jobs:
+ - noop
diff --git a/tests/fixtures/config/central-jobs/git/org_project/README b/tests/fixtures/config/central-jobs/git/org_project/README
new file mode 100644
index 0000000..9daeafb
--- /dev/null
+++ b/tests/fixtures/config/central-jobs/git/org_project/README
@@ -0,0 +1 @@
+test
diff --git a/tests/fixtures/config/central-jobs/main.yaml b/tests/fixtures/config/central-jobs/main.yaml
new file mode 100644
index 0000000..08f4d5d
--- /dev/null
+++ b/tests/fixtures/config/central-jobs/main.yaml
@@ -0,0 +1,9 @@
+- tenant:
+ name: tenant-one
+ source:
+ gerrit:
+ config-projects:
+ - common-config
+ untrusted-projects:
+ - central-jobs
+ - org/project
diff --git a/tests/fixtures/config/data-return/git/common-config/playbooks/child.yaml b/tests/fixtures/config/data-return/git/common-config/playbooks/child.yaml
new file mode 100644
index 0000000..d147e13
--- /dev/null
+++ b/tests/fixtures/config/data-return/git/common-config/playbooks/child.yaml
@@ -0,0 +1,7 @@
+- hosts: localhost
+ tasks:
+ - name: Assert returned variables are valid
+ assert:
+ that:
+ - child.value1 == 'data-return-relative'
+ - child.value2 == 'data-return'
diff --git a/tests/fixtures/config/data-return/git/common-config/playbooks/data-return-relative.yaml b/tests/fixtures/config/data-return/git/common-config/playbooks/data-return-relative.yaml
new file mode 100644
index 0000000..e9193a8
--- /dev/null
+++ b/tests/fixtures/config/data-return/git/common-config/playbooks/data-return-relative.yaml
@@ -0,0 +1,8 @@
+- hosts: localhost
+ tasks:
+ - zuul_return:
+ data:
+ zuul:
+ log_url: http://example.com/test/log/url/
+ child:
+ value1: data-return-relative
diff --git a/tests/fixtures/config/data-return/git/common-config/playbooks/data-return.yaml b/tests/fixtures/config/data-return/git/common-config/playbooks/data-return.yaml
index 5e412c3..db54277 100644
--- a/tests/fixtures/config/data-return/git/common-config/playbooks/data-return.yaml
+++ b/tests/fixtures/config/data-return/git/common-config/playbooks/data-return.yaml
@@ -4,3 +4,6 @@
data:
zuul:
log_url: http://example.com/test/log/url/
+ child:
+ value1: data-return
+ value2: data-return
diff --git a/tests/fixtures/config/data-return/git/common-config/zuul.yaml b/tests/fixtures/config/data-return/git/common-config/zuul.yaml
index 906dc5b..97b6b28 100644
--- a/tests/fixtures/config/data-return/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/data-return/git/common-config/zuul.yaml
@@ -18,11 +18,16 @@
- job:
name: data-return
+ run: playbooks/data-return.yaml
- job:
name: data-return-relative
- run: playbooks/data-return
success-url: docs/index.html
+ run: playbooks/data-return-relative.yaml
+
+- job:
+ name: child
+ run: playbooks/child.yaml
- project:
name: org/project
@@ -30,3 +35,7 @@
jobs:
- data-return
- data-return-relative
+ - child:
+ dependencies:
+ - data-return
+ - data-return-relative
diff --git a/tests/fixtures/config/dependency-graph/git/common-config/zuul.yaml b/tests/fixtures/config/dependency-graph/git/common-config/zuul.yaml
index 4179226..6a96b50 100644
--- a/tests/fixtures/config/dependency-graph/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/dependency-graph/git/common-config/zuul.yaml
@@ -25,24 +25,31 @@
- job:
name: A
+ run: playbooks/A.yaml
- job:
name: B
+ run: playbooks/B.yaml
- job:
name: C
+ run: playbooks/C.yaml
- job:
name: D
+ run: playbooks/D.yaml
- job:
name: E
+ run: playbooks/E.yaml
- job:
name: F
+ run: playbooks/F.yaml
- job:
name: G
+ run: playbooks/G.yaml
- project:
name: org/project
diff --git a/tests/fixtures/config/disk-accountant/git/common-config/zuul.yaml b/tests/fixtures/config/disk-accountant/git/common-config/zuul.yaml
index 893ea05..9ad8de5 100644
--- a/tests/fixtures/config/disk-accountant/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/disk-accountant/git/common-config/zuul.yaml
@@ -18,6 +18,7 @@
- job:
name: dd-big-empty-file
+ run: playbooks/dd-big-empty-file.yaml
- project:
name: org/project
diff --git a/tests/fixtures/config/duplicate-pipeline/git/common-config/zuul.yaml b/tests/fixtures/config/duplicate-pipeline/git/common-config/zuul.yaml
index 117e381..dbd63c5 100755
--- a/tests/fixtures/config/duplicate-pipeline/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/duplicate-pipeline/git/common-config/zuul.yaml
@@ -32,6 +32,7 @@
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
- project:
name: org/project
diff --git a/tests/fixtures/config/final/git/common-config/zuul.yaml b/tests/fixtures/config/final/git/common-config/zuul.yaml
index f08d66e..944626c 100644
--- a/tests/fixtures/config/final/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/final/git/common-config/zuul.yaml
@@ -20,9 +20,9 @@
final: true
vars:
dont_override_this: dummy
+ run: playbooks/job-final.yaml
- project:
name: org/project
check:
jobs: []
-
diff --git a/tests/fixtures/config/git-driver/git/common-config/zuul.yaml b/tests/fixtures/config/git-driver/git/common-config/zuul.yaml
index 34d1136..784b5f2 100644
--- a/tests/fixtures/config/git-driver/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/git-driver/git/common-config/zuul.yaml
@@ -17,6 +17,7 @@
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
- project:
name: org/project
diff --git a/tests/fixtures/config/implicit-roles/git/org_norole-project/.zuul.yaml b/tests/fixtures/config/implicit-roles/git/org_norole-project/.zuul.yaml
index 74c8e8e..f66c616 100644
--- a/tests/fixtures/config/implicit-roles/git/org_norole-project/.zuul.yaml
+++ b/tests/fixtures/config/implicit-roles/git/org_norole-project/.zuul.yaml
@@ -1,11 +1,13 @@
- job:
name: implicit-role-fail
+ run: playbooks/implicit-role-fail.yaml
- job:
name: explicit-role-fail
attempts: 1
roles:
- zuul: org/norole-project
+ run: playbooks/explicit-role-fail.yaml
- project:
name: org/norole-project
diff --git a/tests/fixtures/config/implicit-roles/git/org_role-project/.zuul.yaml b/tests/fixtures/config/implicit-roles/git/org_role-project/.zuul.yaml
index 42cae95..e6e902e 100644
--- a/tests/fixtures/config/implicit-roles/git/org_role-project/.zuul.yaml
+++ b/tests/fixtures/config/implicit-roles/git/org_role-project/.zuul.yaml
@@ -1,11 +1,13 @@
- job:
name: implicit-role-ok
+ run: playbooks/implicit-role-ok.yaml
- job:
name: explicit-role-ok
roles:
- zuul: org/role-project
name: role-name
+ run: playbooks/explicit-role-ok.yaml
- project:
name: org/role-project
diff --git a/tests/fixtures/config/in-repo-join/git/common-config/zuul.yaml b/tests/fixtures/config/in-repo-join/git/common-config/zuul.yaml
index 561fc39..a8ee256 100644
--- a/tests/fixtures/config/in-repo-join/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/in-repo-join/git/common-config/zuul.yaml
@@ -38,6 +38,7 @@
- job:
name: common-config-test
+ run: playbooks/common-config-test.yaml
- project:
name: org/project
diff --git a/tests/fixtures/config/in-repo-join/git/org_project/.zuul.yaml b/tests/fixtures/config/in-repo-join/git/org_project/.zuul.yaml
index 280342c..3845b26 100644
--- a/tests/fixtures/config/in-repo-join/git/org_project/.zuul.yaml
+++ b/tests/fixtures/config/in-repo-join/git/org_project/.zuul.yaml
@@ -1,2 +1,3 @@
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
diff --git a/tests/fixtures/config/in-repo/git/common-config/playbooks/template-job.yaml b/tests/fixtures/config/in-repo/git/common-config/playbooks/template-job.yaml
new file mode 100644
index 0000000..f679dce
--- /dev/null
+++ b/tests/fixtures/config/in-repo/git/common-config/playbooks/template-job.yaml
@@ -0,0 +1,2 @@
+- hosts: all
+ tasks: []
diff --git a/tests/fixtures/config/in-repo/git/common-config/zuul.yaml b/tests/fixtures/config/in-repo/git/common-config/zuul.yaml
index 5623467..c98651c 100644
--- a/tests/fixtures/config/in-repo/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/in-repo/git/common-config/zuul.yaml
@@ -62,10 +62,9 @@
approval:
- Code-Review: 2
username: maintainer
- require:
github:
review:
- - username: '^(herp|derp)$'
+ - username: ^(herp|derp)$
type: approved
trigger: {}
@@ -75,6 +74,17 @@
- job:
name: common-config-test
+ run: playbooks/common-config-test.yaml
+
+- job:
+ name: template-job
+ run: playbooks/template-job.yaml
+
+- project-template:
+ name: common-config-template
+ check:
+ jobs:
+ - template-job
- project:
name: common-config
diff --git a/tests/fixtures/config/in-repo/git/org_project/.zuul.yaml b/tests/fixtures/config/in-repo/git/org_project/.zuul.yaml
index e1c27bb..2c39a10 100644
--- a/tests/fixtures/config/in-repo/git/org_project/.zuul.yaml
+++ b/tests/fixtures/config/in-repo/git/org_project/.zuul.yaml
@@ -1,5 +1,6 @@
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
- project:
name: org/project
diff --git a/tests/fixtures/config/inventory/git/common-config/playbooks/hostvars-inventory.yaml b/tests/fixtures/config/inventory/git/common-config/playbooks/hostvars-inventory.yaml
new file mode 100644
index 0000000..f679dce
--- /dev/null
+++ b/tests/fixtures/config/inventory/git/common-config/playbooks/hostvars-inventory.yaml
@@ -0,0 +1,2 @@
+- hosts: all
+ tasks: []
diff --git a/tests/fixtures/config/inventory/git/common-config/zuul.yaml b/tests/fixtures/config/inventory/git/common-config/zuul.yaml
index e5727a2..74ddf2d 100644
--- a/tests/fixtures/config/inventory/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/inventory/git/common-config/zuul.yaml
@@ -31,6 +31,14 @@
- compute1
- compute2
+- nodeset:
+ name: nodeset2
+ nodes:
+ - name: default
+ label: default-label
+ - name: fakeuser
+ label: fakeuser-label
+
- job:
name: base
parent: null
@@ -41,7 +49,14 @@
nodes:
- name: ubuntu-xenial
label: ubuntu-xenial
+ run: playbooks/single-inventory.yaml
- job:
name: group-inventory
nodeset: nodeset1
+ run: playbooks/group-inventory.yaml
+
+- job:
+ name: hostvars-inventory
+ run: playbooks/hostvars-inventory.yaml
+ nodeset: nodeset2
diff --git a/tests/fixtures/config/inventory/git/org_project/.zuul.yaml b/tests/fixtures/config/inventory/git/org_project/.zuul.yaml
index 26310a0..1a8bf5d 100644
--- a/tests/fixtures/config/inventory/git/org_project/.zuul.yaml
+++ b/tests/fixtures/config/inventory/git/org_project/.zuul.yaml
@@ -4,3 +4,4 @@
jobs:
- single-inventory
- group-inventory
+ - hostvars-inventory
diff --git a/tests/fixtures/config/job-output/git/common-config/zuul.yaml b/tests/fixtures/config/job-output/git/common-config/zuul.yaml
index f182d8d..4df0020 100644
--- a/tests/fixtures/config/job-output/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/job-output/git/common-config/zuul.yaml
@@ -19,6 +19,7 @@
- job:
parent: base
name: job-output
+ run: playbooks/job-output.yaml
- job:
name: job-output-failure
diff --git a/tests/fixtures/config/merges/git/common-config/zuul.yaml b/tests/fixtures/config/merges/git/common-config/zuul.yaml
index 1ea5048..94dbca7 100644
--- a/tests/fixtures/config/merges/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/merges/git/common-config/zuul.yaml
@@ -38,13 +38,16 @@
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
- job:
name: project-test2
+ run: playbooks/project-test2.yaml
- job:
name: project-merge
hold-following-changes: true
+ run: playbooks/project-merge.yaml
- project:
name: org/project-merge
diff --git a/tests/fixtures/config/multi-driver/git/org_common-config/zuul.yaml b/tests/fixtures/config/multi-driver/git/org_common-config/zuul.yaml
index 7a5c190..94a9ecb 100644
--- a/tests/fixtures/config/multi-driver/git/org_common-config/zuul.yaml
+++ b/tests/fixtures/config/multi-driver/git/org_common-config/zuul.yaml
@@ -12,12 +12,12 @@
- event: patchset-created
success:
github:
- status: 'success'
+ status: success
gerrit:
Verified: 1
failure:
github:
- status: 'failure'
+ status: failure
gerrit:
Verified: 1
start:
@@ -32,9 +32,11 @@
- job:
name: project-gerrit
+ run: playbooks/project-gerrit.yaml
- job:
name: project1-github
+ run: playbooks/project1-github.yaml
- project:
name: org/project
diff --git a/tests/fixtures/config/multi-tenant-semaphore/git/tenant-one-config/zuul.yaml b/tests/fixtures/config/multi-tenant-semaphore/git/tenant-one-config/zuul.yaml
index 5e377e7..ca936b9 100644
--- a/tests/fixtures/config/multi-tenant-semaphore/git/tenant-one-config/zuul.yaml
+++ b/tests/fixtures/config/multi-tenant-semaphore/git/tenant-one-config/zuul.yaml
@@ -1,6 +1,7 @@
- job:
name: project1-test1
semaphore: test-semaphore
+ run: playbooks/project1-test1.yaml
- project:
name: org/project1
diff --git a/tests/fixtures/config/multi-tenant-semaphore/git/tenant-two-config/zuul.yaml b/tests/fixtures/config/multi-tenant-semaphore/git/tenant-two-config/zuul.yaml
index a310532..a2866b3 100644
--- a/tests/fixtures/config/multi-tenant-semaphore/git/tenant-two-config/zuul.yaml
+++ b/tests/fixtures/config/multi-tenant-semaphore/git/tenant-two-config/zuul.yaml
@@ -1,6 +1,7 @@
- job:
name: project2-test1
semaphore: test-semaphore
+ run: playbooks/project2-test1.yaml
- project:
name: org/project2
diff --git a/tests/fixtures/config/multi-tenant/git/common-config/zuul.yaml b/tests/fixtures/config/multi-tenant/git/common-config/zuul.yaml
index 273469c..31f1e27 100644
--- a/tests/fixtures/config/multi-tenant/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/multi-tenant/git/common-config/zuul.yaml
@@ -21,3 +21,4 @@
nodes:
- name: controller
label: ubuntu-trusty
+ run: playbooks/python27.yaml
diff --git a/tests/fixtures/config/multi-tenant/git/tenant-one-config/zuul.yaml b/tests/fixtures/config/multi-tenant/git/tenant-one-config/zuul.yaml
index 9a1b928..278c12c 100644
--- a/tests/fixtures/config/multi-tenant/git/tenant-one-config/zuul.yaml
+++ b/tests/fixtures/config/multi-tenant/git/tenant-one-config/zuul.yaml
@@ -27,6 +27,7 @@
- job:
name: project1-test1
+ run: playbooks/project1-test1.yaml
- project:
name: org/project1
diff --git a/tests/fixtures/config/multi-tenant/git/tenant-two-config/zuul.yaml b/tests/fixtures/config/multi-tenant/git/tenant-two-config/zuul.yaml
index 9496a49..2b795b0 100644
--- a/tests/fixtures/config/multi-tenant/git/tenant-two-config/zuul.yaml
+++ b/tests/fixtures/config/multi-tenant/git/tenant-two-config/zuul.yaml
@@ -27,6 +27,7 @@
- job:
name: project2-test1
+ run: playbooks/project2-test1.yaml
- project:
name: org/project2
diff --git a/tests/fixtures/config/openstack/git/project-config/zuul.yaml b/tests/fixtures/config/openstack/git/project-config/zuul.yaml
index de6321d..93bdb11 100644
--- a/tests/fixtures/config/openstack/git/project-config/zuul.yaml
+++ b/tests/fixtures/config/openstack/git/project-config/zuul.yaml
@@ -41,10 +41,12 @@
nodes:
- name: controller
label: ubuntu-xenial
+ run: playbooks/base.yaml
- job:
name: python27
parent: base
+ run: playbooks/python27.yaml
- job:
name: python27
@@ -54,10 +56,12 @@
nodes:
- name: controller
label: ubuntu-trusty
+ run: playbooks/python27.yaml
- job:
name: python35
parent: base
+ run: playbooks/python35.yaml
- project-template:
name: python-jobs
@@ -72,6 +76,7 @@
required-projects:
- openstack/keystone
- openstack/nova
+ run: playbooks/dsvm.yaml
- project:
name: openstack/nova
diff --git a/tests/fixtures/config/post-playbook/git/common-config/zuul.yaml b/tests/fixtures/config/post-playbook/git/common-config/zuul.yaml
index 92a5515..16d7dee 100644
--- a/tests/fixtures/config/post-playbook/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/post-playbook/git/common-config/zuul.yaml
@@ -22,3 +22,4 @@
post-run: playbooks/post
vars:
waitpath: '{{zuul._test.test_root}}/{{zuul.build}}/test_wait'
+ run: playbooks/python27.yaml
diff --git a/tests/fixtures/config/pragma/git/common-config/zuul.yaml b/tests/fixtures/config/pragma/git/common-config/zuul.yaml
new file mode 100644
index 0000000..7a8b45e
--- /dev/null
+++ b/tests/fixtures/config/pragma/git/common-config/zuul.yaml
@@ -0,0 +1,53 @@
+- pipeline:
+ name: check
+ manager: independent
+ trigger:
+ gerrit:
+ - event: patchset-created
+ success:
+ gerrit:
+ Verified: 1
+ failure:
+ gerrit:
+ Verified: -1
+
+- pipeline:
+ name: gate
+ manager: dependent
+ post-review: True
+ trigger:
+ gerrit:
+ - event: comment-added
+ approval:
+ - Approved: 1
+ success:
+ gerrit:
+ Verified: 2
+ submit: true
+ failure:
+ gerrit:
+ Verified: -2
+ start:
+ gerrit:
+ Verified: 0
+ precedence: high
+
+- job:
+ name: base
+ parent: null
+
+- project:
+ name: common-config
+ check:
+ jobs: []
+ gate:
+ jobs:
+ - noop
+
+- project:
+ name: org/project
+ check:
+ jobs: []
+ gate:
+ jobs:
+ - noop
diff --git a/tests/fixtures/config/pragma/git/org_project/README b/tests/fixtures/config/pragma/git/org_project/README
new file mode 100644
index 0000000..9daeafb
--- /dev/null
+++ b/tests/fixtures/config/pragma/git/org_project/README
@@ -0,0 +1 @@
+test
diff --git a/tests/fixtures/config/pragma/git/org_project/nopragma.yaml b/tests/fixtures/config/pragma/git/org_project/nopragma.yaml
new file mode 100644
index 0000000..95a306b
--- /dev/null
+++ b/tests/fixtures/config/pragma/git/org_project/nopragma.yaml
@@ -0,0 +1,2 @@
+- job:
+ name: test-job
diff --git a/tests/fixtures/config/pragma/git/org_project/playbooks/test-job.yaml b/tests/fixtures/config/pragma/git/org_project/playbooks/test-job.yaml
new file mode 100644
index 0000000..f679dce
--- /dev/null
+++ b/tests/fixtures/config/pragma/git/org_project/playbooks/test-job.yaml
@@ -0,0 +1,2 @@
+- hosts: all
+ tasks: []
diff --git a/tests/fixtures/config/pragma/git/org_project/pragma.yaml b/tests/fixtures/config/pragma/git/org_project/pragma.yaml
new file mode 100644
index 0000000..89852b0
--- /dev/null
+++ b/tests/fixtures/config/pragma/git/org_project/pragma.yaml
@@ -0,0 +1,5 @@
+- pragma:
+ implied-branch-matchers: False
+
+- job:
+ name: test-job
diff --git a/tests/fixtures/config/pragma/main.yaml b/tests/fixtures/config/pragma/main.yaml
new file mode 100644
index 0000000..208e274
--- /dev/null
+++ b/tests/fixtures/config/pragma/main.yaml
@@ -0,0 +1,8 @@
+- tenant:
+ name: tenant-one
+ source:
+ gerrit:
+ config-projects:
+ - common-config
+ untrusted-projects:
+ - org/project
diff --git a/tests/fixtures/config/pre-playbook/git/common-config/zuul.yaml b/tests/fixtures/config/pre-playbook/git/common-config/zuul.yaml
index 16d1966..7817745 100644
--- a/tests/fixtures/config/pre-playbook/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/pre-playbook/git/common-config/zuul.yaml
@@ -20,3 +20,4 @@
name: python27
pre-run: playbooks/pre
post-run: playbooks/post
+ run: playbooks/python27.yaml
diff --git a/tests/fixtures/config/push-reqs/git/org_common-config/zuul.yaml b/tests/fixtures/config/push-reqs/git/org_common-config/zuul.yaml
index 63af1c9..e7d52d9 100644
--- a/tests/fixtures/config/push-reqs/git/org_common-config/zuul.yaml
+++ b/tests/fixtures/config/push-reqs/git/org_common-config/zuul.yaml
@@ -47,7 +47,7 @@
manager: independent
require:
github:
- status: 'zuul:check:success'
+ status: zuul:check:success
trigger:
github:
- event: push
@@ -82,6 +82,7 @@
- job:
name: job1
+ run: playbooks/job1.yaml
- project:
name: org/project1
diff --git a/tests/fixtures/config/requirements/email/git/common-config/zuul.yaml b/tests/fixtures/config/requirements/email/git/common-config/zuul.yaml
index 90c9ac2..e41cfad 100644
--- a/tests/fixtures/config/requirements/email/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/requirements/email/git/common-config/zuul.yaml
@@ -36,9 +36,11 @@
- job:
name: project1-job
+ run: playbooks/project1-job.yaml
- job:
name: project2-job
+ run: playbooks/project2-job.yaml
- project:
name: org/project1
diff --git a/tests/fixtures/config/requirements/newer-than/git/common-config/zuul.yaml b/tests/fixtures/config/requirements/newer-than/git/common-config/zuul.yaml
index 5f266a4..96e04d9 100644
--- a/tests/fixtures/config/requirements/newer-than/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/requirements/newer-than/git/common-config/zuul.yaml
@@ -38,9 +38,11 @@
- job:
name: project1-job
+ run: playbooks/project1-job.yaml
- job:
name: project2-job
+ run: playbooks/project2-job.yaml
- project:
name: org/project1
diff --git a/tests/fixtures/config/requirements/older-than/git/common-config/zuul.yaml b/tests/fixtures/config/requirements/older-than/git/common-config/zuul.yaml
index 4287a94..182a036 100644
--- a/tests/fixtures/config/requirements/older-than/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/requirements/older-than/git/common-config/zuul.yaml
@@ -38,9 +38,11 @@
- job:
name: project1-job
+ run: playbooks/project1-job.yaml
- job:
name: project2-job
+ run: playbooks/project2-job.yaml
- project:
name: org/project1
diff --git a/tests/fixtures/config/requirements/reject-username/git/common-config/zuul.yaml b/tests/fixtures/config/requirements/reject-username/git/common-config/zuul.yaml
index aabfb6a..7cbba03 100644
--- a/tests/fixtures/config/requirements/reject-username/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/requirements/reject-username/git/common-config/zuul.yaml
@@ -36,9 +36,11 @@
- job:
name: project1-job
+ run: playbooks/project1-job.yaml
- job:
name: project2-job
+ run: playbooks/project2-job.yaml
- project:
name: org/project1
diff --git a/tests/fixtures/config/requirements/reject/git/common-config/zuul.yaml b/tests/fixtures/config/requirements/reject/git/common-config/zuul.yaml
index 2661eed..0ce66de 100644
--- a/tests/fixtures/config/requirements/reject/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/requirements/reject/git/common-config/zuul.yaml
@@ -52,9 +52,11 @@
- job:
name: project1-job
+ run: playbooks/project1-job.yaml
- job:
name: project2-job
+ run: playbooks/project2-job.yaml
- project:
name: org/project1
diff --git a/tests/fixtures/config/requirements/state/git/common-config/zuul.yaml b/tests/fixtures/config/requirements/state/git/common-config/zuul.yaml
index 715b89f..9f018c9 100644
--- a/tests/fixtures/config/requirements/state/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/requirements/state/git/common-config/zuul.yaml
@@ -55,6 +55,7 @@
- job:
name: project-job
+ run: playbooks/project-job.yaml
- project:
name: current-project
diff --git a/tests/fixtures/config/requirements/username/git/common-config/zuul.yaml b/tests/fixtures/config/requirements/username/git/common-config/zuul.yaml
index 778ac16..05203ab 100644
--- a/tests/fixtures/config/requirements/username/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/requirements/username/git/common-config/zuul.yaml
@@ -36,9 +36,11 @@
- job:
name: project1-job
+ run: playbooks/project1-job.yaml
- job:
name: project2-job
+ run: playbooks/project2-job.yaml
- project:
name: org/project1
diff --git a/tests/fixtures/config/requirements/vote1/git/common-config/zuul.yaml b/tests/fixtures/config/requirements/vote1/git/common-config/zuul.yaml
index b5d7498..88f64e3 100644
--- a/tests/fixtures/config/requirements/vote1/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/requirements/vote1/git/common-config/zuul.yaml
@@ -38,9 +38,11 @@
- job:
name: project1-job
+ run: playbooks/project1-job.yaml
- job:
name: project2-job
+ run: playbooks/project2-job.yaml
- project:
name: org/project1
diff --git a/tests/fixtures/config/requirements/vote2/git/common-config/zuul.yaml b/tests/fixtures/config/requirements/vote2/git/common-config/zuul.yaml
index 3f41868..31b2329 100644
--- a/tests/fixtures/config/requirements/vote2/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/requirements/vote2/git/common-config/zuul.yaml
@@ -42,9 +42,11 @@
- job:
name: project1-job
+ run: playbooks/project1-job.yaml
- job:
name: project2-job
+ run: playbooks/project2-job.yaml
- project:
name: org/project1
diff --git a/tests/fixtures/config/roles/git/common-config/zuul.yaml b/tests/fixtures/config/roles/git/common-config/zuul.yaml
index 7ae6263..ba34ad6 100644
--- a/tests/fixtures/config/roles/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/roles/git/common-config/zuul.yaml
@@ -38,6 +38,7 @@
- job:
name: common-config-test
+ run: playbooks/common-config-test.yaml
- project:
name: common-config
diff --git a/tests/fixtures/config/roles/git/org_project/.zuul.yaml b/tests/fixtures/config/roles/git/org_project/.zuul.yaml
index 35c2153..0986b82 100644
--- a/tests/fixtures/config/roles/git/org_project/.zuul.yaml
+++ b/tests/fixtures/config/roles/git/org_project/.zuul.yaml
@@ -2,6 +2,7 @@
name: project-test
roles:
- zuul: bare-role
+ run: playbooks/project-test.yaml
- project:
name: org/project
diff --git a/tests/fixtures/config/secret-inheritance/git/common-config/playbooks/trusted-secrets-trusted-child.yaml b/tests/fixtures/config/secret-inheritance/git/common-config/playbooks/trusted-secrets-trusted-child.yaml
new file mode 100644
index 0000000..f679dce
--- /dev/null
+++ b/tests/fixtures/config/secret-inheritance/git/common-config/playbooks/trusted-secrets-trusted-child.yaml
@@ -0,0 +1,2 @@
+- hosts: all
+ tasks: []
diff --git a/tests/fixtures/config/secret-inheritance/git/common-config/playbooks/trusted-secrets.yaml b/tests/fixtures/config/secret-inheritance/git/common-config/playbooks/trusted-secrets.yaml
new file mode 100644
index 0000000..f679dce
--- /dev/null
+++ b/tests/fixtures/config/secret-inheritance/git/common-config/playbooks/trusted-secrets.yaml
@@ -0,0 +1,2 @@
+- hosts: all
+ tasks: []
diff --git a/tests/fixtures/config/secret-inheritance/git/common-config/playbooks/untrusted-secrets-trusted-child.yaml b/tests/fixtures/config/secret-inheritance/git/common-config/playbooks/untrusted-secrets-trusted-child.yaml
new file mode 100644
index 0000000..f679dce
--- /dev/null
+++ b/tests/fixtures/config/secret-inheritance/git/common-config/playbooks/untrusted-secrets-trusted-child.yaml
@@ -0,0 +1,2 @@
+- hosts: all
+ tasks: []
diff --git a/tests/fixtures/config/secret-inheritance/git/common-config/zuul.yaml b/tests/fixtures/config/secret-inheritance/git/common-config/zuul.yaml
new file mode 100644
index 0000000..ad16d4e
--- /dev/null
+++ b/tests/fixtures/config/secret-inheritance/git/common-config/zuul.yaml
@@ -0,0 +1,106 @@
+- pipeline:
+ name: check
+ manager: independent
+ trigger:
+ gerrit:
+ - event: patchset-created
+ success:
+ gerrit:
+ Verified: 1
+ failure:
+ gerrit:
+ Verified: -1
+
+- pipeline:
+ name: gate
+ manager: dependent
+ post-review: True
+ trigger:
+ gerrit:
+ - event: comment-added
+ approval:
+ - Approved: 1
+ success:
+ gerrit:
+ Verified: 2
+ submit: true
+ failure:
+ gerrit:
+ Verified: -2
+ start:
+ gerrit:
+ Verified: 0
+ precedence: high
+
+- job:
+ name: base
+ parent: null
+
+- job:
+ name: trusted-secrets
+ run: playbooks/trusted-secrets.yaml
+ secrets:
+ - trusted-secret
+
+- job:
+ name: trusted-secrets-trusted-child
+ run: playbooks/trusted-secrets-trusted-child.yaml
+ parent: trusted-secrets
+
+- job:
+ name: untrusted-secrets-trusted-child
+ run: playbooks/untrusted-secrets-trusted-child.yaml
+ parent: untrusted-secrets
+
+- project:
+ name: common-config
+ check:
+ jobs:
+ - trusted-secrets
+ - trusted-secrets-trusted-child
+ - trusted-secrets-untrusted-child
+ gate:
+ jobs:
+ - untrusted-secrets
+ - untrusted-secrets-trusted-child
+ - untrusted-secrets-untrusted-child
+
+- secret:
+ name: trusted-secret
+ data:
+ username: test-username
+ longpassword: !encrypted/pkcs1-oaep
+ - BFhtdnm8uXx7kn79RFL/zJywmzLkT1GY78P3bOtp4WghUFWobkifSu7ZpaV4NeO0s71Y
+ Usi1wGZZL0LveZjUN0t6OU1VZKSG8R5Ly7urjaSo1pPVIq5Rtt/H7W14Lecd+cUeKb4j
+ oeusC9drN3AA8a4oykcVpt1wVqUnTbMGC9ARMCQP6eopcs1l7tzMseprW4RDNhIuz3CR
+ gd0QBMPl6VDoFgBPB8vxtJw+3m0rqBYZCLZgCXekqlny8s2s92nJMuUABbJOEcDRarzi
+ bDsSXsfJt1y+5n7yOURsC7lovMg4GF/vCl/0YMKjBO5bpv9EM5fToeKYyPGSKQoHOnCY
+ ceb3cAVcv5UawcCic8XjhEhp4K7WPdYf2HVAC/qtxhbpjTxG4U5Q/SoppOJ60WqEkQvb
+ Xs6n5Dvy7xmph6GWmU/bAv3eUK3pdD3xa2Ue1lHWz3U+rsYraI+AKYsMYx3RBlfAmCeC
+ 1ve2BXPrqnOo7G8tnUvfdYPbK4Aakk0ds/AVqFHEZN+S6hRBmBjLaRFWZ3QSO1NjbBxW
+ naHKZYT7nkrJm8AMCgZU0ZArFLpaufKCeiK5ECSsDxic4FIsY1OkWT42qEUfL0Wd+150
+ AKGNZpPJnnP3QYY4W/MWcKH/zdO400+zWN52WevbSqZy90tqKDJrBkMl1ydqbuw1E4ZH
+ vIs=
+ - BFhtdnm8uXx7kn79RFL/zJywmzLkT1GY78P3bOtp4WghUFWobkifSu7ZpaV4NeO0s71Y
+ Usi1wGZZL0LveZjUN0t6OU1VZKSG8R5Ly7urjaSo1pPVIq5Rtt/H7W14Lecd+cUeKb4j
+ oeusC9drN3AA8a4oykcVpt1wVqUnTbMGC9ARMCQP6eopcs1l7tzMseprW4RDNhIuz3CR
+ gd0QBMPl6VDoFgBPB8vxtJw+3m0rqBYZCLZgCXekqlny8s2s92nJMuUABbJOEcDRarzi
+ bDsSXsfJt1y+5n7yOURsC7lovMg4GF/vCl/0YMKjBO5bpv9EM5fToeKYyPGSKQoHOnCY
+ ceb3cAVcv5UawcCic8XjhEhp4K7WPdYf2HVAC/qtxhbpjTxG4U5Q/SoppOJ60WqEkQvb
+ Xs6n5Dvy7xmph6GWmU/bAv3eUK3pdD3xa2Ue1lHWz3U+rsYraI+AKYsMYx3RBlfAmCeC
+ 1ve2BXPrqnOo7G8tnUvfdYPbK4Aakk0ds/AVqFHEZN+S6hRBmBjLaRFWZ3QSO1NjbBxW
+ naHKZYT7nkrJm8AMCgZU0ZArFLpaufKCeiK5ECSsDxic4FIsY1OkWT42qEUfL0Wd+150
+ AKGNZpPJnnP3QYY4W/MWcKH/zdO400+zWN52WevbSqZy90tqKDJrBkMl1ydqbuw1E4ZH
+ vIs=
+ password: !encrypted/pkcs1-oaep |
+ BFhtdnm8uXx7kn79RFL/zJywmzLkT1GY78P3bOtp4WghUFWobkifSu7ZpaV4NeO0s71Y
+ Usi1wGZZL0LveZjUN0t6OU1VZKSG8R5Ly7urjaSo1pPVIq5Rtt/H7W14Lecd+cUeKb4j
+ oeusC9drN3AA8a4oykcVpt1wVqUnTbMGC9ARMCQP6eopcs1l7tzMseprW4RDNhIuz3CR
+ gd0QBMPl6VDoFgBPB8vxtJw+3m0rqBYZCLZgCXekqlny8s2s92nJMuUABbJOEcDRarzi
+ bDsSXsfJt1y+5n7yOURsC7lovMg4GF/vCl/0YMKjBO5bpv9EM5fToeKYyPGSKQoHOnCY
+ ceb3cAVcv5UawcCic8XjhEhp4K7WPdYf2HVAC/qtxhbpjTxG4U5Q/SoppOJ60WqEkQvb
+ Xs6n5Dvy7xmph6GWmU/bAv3eUK3pdD3xa2Ue1lHWz3U+rsYraI+AKYsMYx3RBlfAmCeC
+ 1ve2BXPrqnOo7G8tnUvfdYPbK4Aakk0ds/AVqFHEZN+S6hRBmBjLaRFWZ3QSO1NjbBxW
+ naHKZYT7nkrJm8AMCgZU0ZArFLpaufKCeiK5ECSsDxic4FIsY1OkWT42qEUfL0Wd+150
+ AKGNZpPJnnP3QYY4W/MWcKH/zdO400+zWN52WevbSqZy90tqKDJrBkMl1ydqbuw1E4ZH
+ vIs=
diff --git a/tests/fixtures/config/secret-inheritance/git/org_project/.zuul.yaml b/tests/fixtures/config/secret-inheritance/git/org_project/.zuul.yaml
new file mode 100644
index 0000000..b384669
--- /dev/null
+++ b/tests/fixtures/config/secret-inheritance/git/org_project/.zuul.yaml
@@ -0,0 +1,66 @@
+- job:
+ name: untrusted-secrets
+ run: playbooks/untrusted-secrets.yaml
+ secrets:
+ - untrusted-secret
+
+- job:
+ name: trusted-secrets-untrusted-child
+ run: playbooks/trusted-secrets-untrusted-child.yaml
+ parent: trusted-secrets
+
+- job:
+ name: untrusted-secrets-untrusted-child
+ run: playbooks/untrusted-secrets-untrusted-child.yaml
+ parent: untrusted-secrets
+
+- project:
+ name: org/project
+ check:
+ jobs:
+ - trusted-secrets
+ - trusted-secrets-trusted-child
+ - trusted-secrets-untrusted-child
+ - untrusted-secrets
+ - untrusted-secrets-trusted-child
+ - untrusted-secrets-untrusted-child
+
+- secret:
+ name: untrusted-secret
+ data:
+ username: test-username
+ longpassword: !encrypted/pkcs1-oaep
+ - BFhtdnm8uXx7kn79RFL/zJywmzLkT1GY78P3bOtp4WghUFWobkifSu7ZpaV4NeO0s71Y
+ Usi1wGZZL0LveZjUN0t6OU1VZKSG8R5Ly7urjaSo1pPVIq5Rtt/H7W14Lecd+cUeKb4j
+ oeusC9drN3AA8a4oykcVpt1wVqUnTbMGC9ARMCQP6eopcs1l7tzMseprW4RDNhIuz3CR
+ gd0QBMPl6VDoFgBPB8vxtJw+3m0rqBYZCLZgCXekqlny8s2s92nJMuUABbJOEcDRarzi
+ bDsSXsfJt1y+5n7yOURsC7lovMg4GF/vCl/0YMKjBO5bpv9EM5fToeKYyPGSKQoHOnCY
+ ceb3cAVcv5UawcCic8XjhEhp4K7WPdYf2HVAC/qtxhbpjTxG4U5Q/SoppOJ60WqEkQvb
+ Xs6n5Dvy7xmph6GWmU/bAv3eUK3pdD3xa2Ue1lHWz3U+rsYraI+AKYsMYx3RBlfAmCeC
+ 1ve2BXPrqnOo7G8tnUvfdYPbK4Aakk0ds/AVqFHEZN+S6hRBmBjLaRFWZ3QSO1NjbBxW
+ naHKZYT7nkrJm8AMCgZU0ZArFLpaufKCeiK5ECSsDxic4FIsY1OkWT42qEUfL0Wd+150
+ AKGNZpPJnnP3QYY4W/MWcKH/zdO400+zWN52WevbSqZy90tqKDJrBkMl1ydqbuw1E4ZH
+ vIs=
+ - BFhtdnm8uXx7kn79RFL/zJywmzLkT1GY78P3bOtp4WghUFWobkifSu7ZpaV4NeO0s71Y
+ Usi1wGZZL0LveZjUN0t6OU1VZKSG8R5Ly7urjaSo1pPVIq5Rtt/H7W14Lecd+cUeKb4j
+ oeusC9drN3AA8a4oykcVpt1wVqUnTbMGC9ARMCQP6eopcs1l7tzMseprW4RDNhIuz3CR
+ gd0QBMPl6VDoFgBPB8vxtJw+3m0rqBYZCLZgCXekqlny8s2s92nJMuUABbJOEcDRarzi
+ bDsSXsfJt1y+5n7yOURsC7lovMg4GF/vCl/0YMKjBO5bpv9EM5fToeKYyPGSKQoHOnCY
+ ceb3cAVcv5UawcCic8XjhEhp4K7WPdYf2HVAC/qtxhbpjTxG4U5Q/SoppOJ60WqEkQvb
+ Xs6n5Dvy7xmph6GWmU/bAv3eUK3pdD3xa2Ue1lHWz3U+rsYraI+AKYsMYx3RBlfAmCeC
+ 1ve2BXPrqnOo7G8tnUvfdYPbK4Aakk0ds/AVqFHEZN+S6hRBmBjLaRFWZ3QSO1NjbBxW
+ naHKZYT7nkrJm8AMCgZU0ZArFLpaufKCeiK5ECSsDxic4FIsY1OkWT42qEUfL0Wd+150
+ AKGNZpPJnnP3QYY4W/MWcKH/zdO400+zWN52WevbSqZy90tqKDJrBkMl1ydqbuw1E4ZH
+ vIs=
+ password: !encrypted/pkcs1-oaep |
+ BFhtdnm8uXx7kn79RFL/zJywmzLkT1GY78P3bOtp4WghUFWobkifSu7ZpaV4NeO0s71Y
+ Usi1wGZZL0LveZjUN0t6OU1VZKSG8R5Ly7urjaSo1pPVIq5Rtt/H7W14Lecd+cUeKb4j
+ oeusC9drN3AA8a4oykcVpt1wVqUnTbMGC9ARMCQP6eopcs1l7tzMseprW4RDNhIuz3CR
+ gd0QBMPl6VDoFgBPB8vxtJw+3m0rqBYZCLZgCXekqlny8s2s92nJMuUABbJOEcDRarzi
+ bDsSXsfJt1y+5n7yOURsC7lovMg4GF/vCl/0YMKjBO5bpv9EM5fToeKYyPGSKQoHOnCY
+ ceb3cAVcv5UawcCic8XjhEhp4K7WPdYf2HVAC/qtxhbpjTxG4U5Q/SoppOJ60WqEkQvb
+ Xs6n5Dvy7xmph6GWmU/bAv3eUK3pdD3xa2Ue1lHWz3U+rsYraI+AKYsMYx3RBlfAmCeC
+ 1ve2BXPrqnOo7G8tnUvfdYPbK4Aakk0ds/AVqFHEZN+S6hRBmBjLaRFWZ3QSO1NjbBxW
+ naHKZYT7nkrJm8AMCgZU0ZArFLpaufKCeiK5ECSsDxic4FIsY1OkWT42qEUfL0Wd+150
+ AKGNZpPJnnP3QYY4W/MWcKH/zdO400+zWN52WevbSqZy90tqKDJrBkMl1ydqbuw1E4ZH
+ vIs=
diff --git a/tests/fixtures/config/secret-inheritance/git/org_project/README b/tests/fixtures/config/secret-inheritance/git/org_project/README
new file mode 100644
index 0000000..9daeafb
--- /dev/null
+++ b/tests/fixtures/config/secret-inheritance/git/org_project/README
@@ -0,0 +1 @@
+test
diff --git a/tests/fixtures/config/secret-inheritance/git/org_project/playbooks/trusted-secrets-untrusted-child.yaml b/tests/fixtures/config/secret-inheritance/git/org_project/playbooks/trusted-secrets-untrusted-child.yaml
new file mode 100644
index 0000000..f679dce
--- /dev/null
+++ b/tests/fixtures/config/secret-inheritance/git/org_project/playbooks/trusted-secrets-untrusted-child.yaml
@@ -0,0 +1,2 @@
+- hosts: all
+ tasks: []
diff --git a/tests/fixtures/config/secret-inheritance/git/org_project/playbooks/untrusted-secrets-untrusted-child.yaml b/tests/fixtures/config/secret-inheritance/git/org_project/playbooks/untrusted-secrets-untrusted-child.yaml
new file mode 100644
index 0000000..f679dce
--- /dev/null
+++ b/tests/fixtures/config/secret-inheritance/git/org_project/playbooks/untrusted-secrets-untrusted-child.yaml
@@ -0,0 +1,2 @@
+- hosts: all
+ tasks: []
diff --git a/tests/fixtures/config/secret-inheritance/git/org_project/playbooks/untrusted-secrets.yaml b/tests/fixtures/config/secret-inheritance/git/org_project/playbooks/untrusted-secrets.yaml
new file mode 100644
index 0000000..f679dce
--- /dev/null
+++ b/tests/fixtures/config/secret-inheritance/git/org_project/playbooks/untrusted-secrets.yaml
@@ -0,0 +1,2 @@
+- hosts: all
+ tasks: []
diff --git a/tests/fixtures/config/secret-inheritance/main.yaml b/tests/fixtures/config/secret-inheritance/main.yaml
new file mode 100644
index 0000000..208e274
--- /dev/null
+++ b/tests/fixtures/config/secret-inheritance/main.yaml
@@ -0,0 +1,8 @@
+- tenant:
+ name: tenant-one
+ source:
+ gerrit:
+ config-projects:
+ - common-config
+ untrusted-projects:
+ - org/project
diff --git a/tests/fixtures/config/secret-leaks/git/common-config/zuul.yaml b/tests/fixtures/config/secret-leaks/git/common-config/zuul.yaml
index 4ab198f..9321df8 100644
--- a/tests/fixtures/config/secret-leaks/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/secret-leaks/git/common-config/zuul.yaml
@@ -55,12 +55,14 @@
- job:
parent: base
name: secret-file
+ run: playbooks/secret-file.yaml
secrets:
- test_secret
- job:
parent: base
name: secret-file-fail
+ run: playbooks/secret-file-fail.yaml
secrets:
- test_secret
diff --git a/tests/fixtures/config/semaphore/git/common-config/zuul.yaml b/tests/fixtures/config/semaphore/git/common-config/zuul.yaml
index c8bd322..52a0e7d 100644
--- a/tests/fixtures/config/semaphore/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/semaphore/git/common-config/zuul.yaml
@@ -25,22 +25,27 @@
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
- job:
name: semaphore-one-test1
semaphore: test-semaphore
+ run: playbooks/semaphore-one-test1.yaml
- job:
name: semaphore-one-test2
semaphore: test-semaphore
+ run: playbooks/semaphore-one-test2.yaml
- job:
name: semaphore-two-test1
semaphore: test-semaphore-two
+ run: playbooks/semaphore-two-test1.yaml
- job:
name: semaphore-two-test2
semaphore: test-semaphore-two
+ run: playbooks/semaphore-two-test2.yaml
- project:
name: org/project
diff --git a/tests/fixtures/config/shadow/git/local-config/zuul.yaml b/tests/fixtures/config/shadow/git/local-config/zuul.yaml
index 87f46b7..8935d8a 100644
--- a/tests/fixtures/config/shadow/git/local-config/zuul.yaml
+++ b/tests/fixtures/config/shadow/git/local-config/zuul.yaml
@@ -14,9 +14,11 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: test2
+ run: playbooks/test2.yaml
- project:
name: org/project
diff --git a/tests/fixtures/config/shadow/git/stdlib/.zuul.yaml b/tests/fixtures/config/shadow/git/stdlib/.zuul.yaml
index 6a6f9c9..5132653 100644
--- a/tests/fixtures/config/shadow/git/stdlib/.zuul.yaml
+++ b/tests/fixtures/config/shadow/git/stdlib/.zuul.yaml
@@ -1,10 +1,13 @@
- job:
name: base
+ run: playbooks/base.yaml
- job:
name: test1
parent: base
+ run: playbooks/test1.yaml
- job:
name: test2
parent: base
+ run: playbooks/test2.yaml
diff --git a/tests/fixtures/config/single-tenant/git/common-config/zuul.yaml b/tests/fixtures/config/single-tenant/git/common-config/zuul.yaml
index 2160ef9..b2f15f9 100644
--- a/tests/fixtures/config/single-tenant/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/single-tenant/git/common-config/zuul.yaml
@@ -52,6 +52,7 @@
nodes:
- name: controller
label: label1
+ run: playbooks/project-merge.yaml
- job:
name: project-test1
@@ -60,6 +61,7 @@
nodes:
- name: controller
label: label1
+ run: playbooks/project-test1.yaml
- job:
name: project-test1
@@ -68,6 +70,7 @@
nodes:
- name: controller
label: label2
+ run: playbooks/project-test1.yaml
- job:
name: project-post
@@ -75,6 +78,7 @@
nodes:
- name: static
label: ubuntu-xenial
+ run: playbooks/project-post.yaml
- job:
name: project-test2
@@ -82,6 +86,7 @@
nodes:
- name: controller
label: label1
+ run: playbooks/project-test2.yaml
- job:
name: project1-project2-integration
@@ -89,11 +94,13 @@
nodes:
- name: controller
label: label1
+ run: playbooks/project1-project2-integration.yaml
- job:
name: project-testfile
files:
- .*-requires
+ run: playbooks/project-testfile.yaml
- project:
name: org/project
diff --git a/tests/fixtures/config/split-config/git/common-config/zuul.d/jobs.yaml b/tests/fixtures/config/split-config/git/common-config/zuul.d/jobs.yaml
index 9d15599..12e1c24 100644
--- a/tests/fixtures/config/split-config/git/common-config/zuul.d/jobs.yaml
+++ b/tests/fixtures/config/split-config/git/common-config/zuul.d/jobs.yaml
@@ -4,3 +4,4 @@
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
diff --git a/tests/fixtures/config/split-config/git/org_project1/.zuul.d/jobs.yaml b/tests/fixtures/config/split-config/git/org_project1/.zuul.d/jobs.yaml
index 33d74f3..20cd16a 100644
--- a/tests/fixtures/config/split-config/git/org_project1/.zuul.d/jobs.yaml
+++ b/tests/fixtures/config/split-config/git/org_project1/.zuul.d/jobs.yaml
@@ -1,2 +1,3 @@
- job:
name: project1-project2-integration
+ run: playbooks/project1-project2-integration.yaml
diff --git a/tests/fixtures/config/sql-driver/git/common-config/zuul.yaml b/tests/fixtures/config/sql-driver/git/common-config/zuul.yaml
index 8fce9e7..82c85c7 100644
--- a/tests/fixtures/config/sql-driver/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/sql-driver/git/common-config/zuul.yaml
@@ -7,12 +7,12 @@
success:
gerrit:
Verified: 1
- resultsdb:
+ resultsdb: null
failure:
gerrit:
Verified: -1
- resultsdb:
- resultsdb_failures:
+ resultsdb: null
+ resultsdb_failures: null
- job:
name: base
@@ -20,15 +20,19 @@
- job:
name: project-merge
+ run: playbooks/project-merge.yaml
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
- job:
name: project-test2
+ run: playbooks/project-test2.yaml
- job:
name: project-test3
+ run: playbooks/project-test3.yaml
- project:
name: org/project
diff --git a/tests/fixtures/config/streamer/git/common-config/zuul.yaml b/tests/fixtures/config/streamer/git/common-config/zuul.yaml
index f9925fe..8e67bfb 100644
--- a/tests/fixtures/config/streamer/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/streamer/git/common-config/zuul.yaml
@@ -19,3 +19,4 @@
name: python27
vars:
waitpath: '{{zuul._test.test_root}}/{{zuul.build}}/test_wait'
+ run: playbooks/python27.yaml
diff --git a/tests/fixtures/config/success-url/git/common-config/zuul.yaml b/tests/fixtures/config/success-url/git/common-config/zuul.yaml
index 8929240..b9f4bff 100644
--- a/tests/fixtures/config/success-url/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/success-url/git/common-config/zuul.yaml
@@ -23,10 +23,12 @@
- job:
name: docs-draft-test
success-url: http://docs-draft.example.org/{change.number:.2}/{change.number}/{change.patchset}/{pipeline.name}/{job.name}/{build.uuid:.7}/publish-docs/
+ run: playbooks/docs-draft-test.yaml
- job:
name: docs-draft-test2
success-url: http://docs-draft.example.org/{NOPE}/{build.parameters[BAD]}/publish-docs/
+ run: playbooks/docs-draft-test2.yaml
- project:
name: org/docs
diff --git a/tests/fixtures/config/templated-project/git/common-config/zuul.d/jobs.yaml b/tests/fixtures/config/templated-project/git/common-config/zuul.d/jobs.yaml
index f9de1ad..7ad791d 100644
--- a/tests/fixtures/config/templated-project/git/common-config/zuul.d/jobs.yaml
+++ b/tests/fixtures/config/templated-project/git/common-config/zuul.d/jobs.yaml
@@ -4,18 +4,24 @@
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
- job:
name: project-test2
+ run: playbooks/project-test2.yaml
- job:
name: layered-project-test3
+ run: playbooks/layered-project-test3.yaml
- job:
name: layered-project-test4
+ run: playbooks/layered-project-test4.yaml
- job:
name: layered-project-foo-test5
+ run: playbooks/layered-project-foo-test5.yaml
- job:
name: project-test6
+ run: playbooks/project-test6.yaml
diff --git a/tests/fixtures/config/unprotected-branches/git/org_project1/zuul.yaml b/tests/fixtures/config/unprotected-branches/git/org_project1/zuul.yaml
index 31abadf..1deffb3 100644
--- a/tests/fixtures/config/unprotected-branches/git/org_project1/zuul.yaml
+++ b/tests/fixtures/config/unprotected-branches/git/org_project1/zuul.yaml
@@ -1,5 +1,6 @@
- job:
name: project-test
+ run: playbooks/project-test.yaml
- project:
name: org/project1
diff --git a/tests/fixtures/config/zuul-connections-multiple-gerrits/git/common-config/zuul.yaml b/tests/fixtures/config/zuul-connections-multiple-gerrits/git/common-config/zuul.yaml
index d70a384..6a0865e 100644
--- a/tests/fixtures/config/zuul-connections-multiple-gerrits/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/zuul-connections-multiple-gerrits/git/common-config/zuul.yaml
@@ -49,9 +49,11 @@
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
- job:
name: project-test2
+ run: playbooks/project-test2.yaml
- project:
name: review.example.com/org/project1
@@ -65,7 +67,6 @@
jobs:
- project-test2
-
- project:
name: review.example.com/org/project2
common_check:
diff --git a/tests/fixtures/config/zuul-connections-same-gerrit/git/common-config/zuul.yaml b/tests/fixtures/config/zuul-connections-same-gerrit/git/common-config/zuul.yaml
index eb65279..14a9b11 100644
--- a/tests/fixtures/config/zuul-connections-same-gerrit/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/zuul-connections-same-gerrit/git/common-config/zuul.yaml
@@ -17,9 +17,11 @@
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
- job:
name: project-test2
+ run: playbooks/project-test2.yaml
- project:
name: org/project
diff --git a/tests/fixtures/config/zuultrigger/parent-change-enqueued/git/common-config/zuul.yaml b/tests/fixtures/config/zuultrigger/parent-change-enqueued/git/common-config/zuul.yaml
index 3dd8324..3fcc43c 100644
--- a/tests/fixtures/config/zuultrigger/parent-change-enqueued/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/zuultrigger/parent-change-enqueued/git/common-config/zuul.yaml
@@ -51,9 +51,11 @@
- job:
name: project-check
+ run: playbooks/project-check.yaml
- job:
name: project-gate
+ run: playbooks/project-gate.yaml
- project:
name: org/project
diff --git a/tests/fixtures/config/zuultrigger/project-change-merged/git/common-config/zuul.yaml b/tests/fixtures/config/zuultrigger/project-change-merged/git/common-config/zuul.yaml
index a5c5a1c..045e0a9 100644
--- a/tests/fixtures/config/zuultrigger/project-change-merged/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/zuultrigger/project-change-merged/git/common-config/zuul.yaml
@@ -49,9 +49,11 @@
- job:
name: project-check
+ run: playbooks/project-check.yaml
- job:
name: project-gate
+ run: playbooks/project-gate.yaml
- project:
name: org/project
diff --git a/tests/fixtures/layouts/autohold.yaml b/tests/fixtures/layouts/autohold.yaml
index 578f886..32b6822 100644
--- a/tests/fixtures/layouts/autohold.yaml
+++ b/tests/fixtures/layouts/autohold.yaml
@@ -14,6 +14,7 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-test2
@@ -21,6 +22,7 @@
nodes:
- name: controller
label: label1
+ run: playbooks/project-test2.yaml
- project:
name: org/project
diff --git a/tests/fixtures/layouts/basic-github.yaml b/tests/fixtures/layouts/basic-github.yaml
index d7b323a..217e874 100644
--- a/tests/fixtures/layouts/basic-github.yaml
+++ b/tests/fixtures/layouts/basic-github.yaml
@@ -8,10 +8,10 @@
- opened
- changed
- reopened
- branch: '^master$'
+ branch: ^master$
- event: pull_request
action: comment
- comment: 'test me'
+ comment: test me
success:
github: {}
failure:
@@ -20,12 +20,15 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
- job:
name: project-test2
+ run: playbooks/project-test2.yaml
- project:
name: org/project
diff --git a/tests/fixtures/layouts/crd-github.yaml b/tests/fixtures/layouts/crd-github.yaml
index 9696226..6ef881f 100644
--- a/tests/fixtures/layouts/crd-github.yaml
+++ b/tests/fixtures/layouts/crd-github.yaml
@@ -30,24 +30,31 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project1-test
+ run: playbooks/project1-test.yaml
- job:
name: project2-test
+ run: playbooks/project2-test.yaml
- job:
name: project3-test
+ run: playbooks/project3-test.yaml
- job:
name: project4-test
+ run: playbooks/project4-test.yaml
- job:
name: project5-test
+ run: playbooks/project5-test.yaml
- job:
name: project6-test
+ run: playbooks/project6-test.yaml
- project:
name: org/project1
diff --git a/tests/fixtures/layouts/delayed-repo-init.yaml b/tests/fixtures/layouts/delayed-repo-init.yaml
index c89e2fa..0c9a152 100644
--- a/tests/fixtures/layouts/delayed-repo-init.yaml
+++ b/tests/fixtures/layouts/delayed-repo-init.yaml
@@ -43,18 +43,23 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-merge
+ run: playbooks/project-merge.yaml
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
- job:
name: project-test2
+ run: playbooks/project-test2.yaml
- job:
name: project-post
+ run: playbooks/project-post.yaml
- project:
name: org/new-project
diff --git a/tests/fixtures/layouts/dependent-github.yaml b/tests/fixtures/layouts/dependent-github.yaml
index eb74163..6ad6bd2 100644
--- a/tests/fixtures/layouts/dependent-github.yaml
+++ b/tests/fixtures/layouts/dependent-github.yaml
@@ -6,29 +6,33 @@
github:
- event: pull_request
action: labeled
- label: 'merge'
+ label: merge
success:
github:
merge: true
- unlabel: 'merge'
+ unlabel: merge
failure:
github:
- unlabel: 'merge'
+ unlabel: merge
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
- job:
name: project-test2
+ run: playbooks/project-test2.yaml
- job:
name: project-merge
failure-message: Unable to merge change
hold-following-changes: true
+ run: playbooks/project-merge.yaml
- project:
name: org/project
diff --git a/tests/fixtures/layouts/dequeue-github.yaml b/tests/fixtures/layouts/dequeue-github.yaml
index ae61cd5..72d8145 100644
--- a/tests/fixtures/layouts/dequeue-github.yaml
+++ b/tests/fixtures/layouts/dequeue-github.yaml
@@ -11,9 +11,11 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: one-job-project-merge
+ run: playbooks/one-job-project-merge.yaml
- project:
name: org/one-job-project
diff --git a/tests/fixtures/layouts/disable_at.yaml b/tests/fixtures/layouts/disable_at.yaml
index 8c24c1b..a090d11 100644
--- a/tests/fixtures/layouts/disable_at.yaml
+++ b/tests/fixtures/layouts/disable_at.yaml
@@ -18,6 +18,7 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-test1
@@ -25,6 +26,7 @@
nodes:
- name: controller
label: label1
+ run: playbooks/project-test1.yaml
- project:
name: org/project
diff --git a/tests/fixtures/layouts/dont-ignore-ref-deletes.yaml b/tests/fixtures/layouts/dont-ignore-ref-deletes.yaml
index bb98b57..9a9e592 100644
--- a/tests/fixtures/layouts/dont-ignore-ref-deletes.yaml
+++ b/tests/fixtures/layouts/dont-ignore-ref-deletes.yaml
@@ -10,6 +10,7 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-post
@@ -17,6 +18,7 @@
nodes:
- name: static
label: ubuntu-xenial
+ run: playbooks/project-post.yaml
- project:
name: org/project
diff --git a/tests/fixtures/layouts/files-github.yaml b/tests/fixtures/layouts/files-github.yaml
index ec35259..ed053f9 100644
--- a/tests/fixtures/layouts/files-github.yaml
+++ b/tests/fixtures/layouts/files-github.yaml
@@ -9,11 +9,13 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-test1
files:
- - '.*-requires'
+ - .*-requires
+ run: playbooks/project-test1.yaml
- project:
name: org/project
diff --git a/tests/fixtures/layouts/footer-message.yaml b/tests/fixtures/layouts/footer-message.yaml
index 4ee25f6..746d384 100644
--- a/tests/fixtures/layouts/footer-message.yaml
+++ b/tests/fixtures/layouts/footer-message.yaml
@@ -28,10 +28,11 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-test1
-# success-url: http://logs.exxxample.com/{change.number}/{change.patchset}/{pipeline.name}/{job.name}
+ run: playbooks/project-test1.yaml
- project:
name: org/project
diff --git a/tests/fixtures/layouts/idle.yaml b/tests/fixtures/layouts/idle.yaml
index 4cc07ae..4f3efe4 100644
--- a/tests/fixtures/layouts/idle.yaml
+++ b/tests/fixtures/layouts/idle.yaml
@@ -8,6 +8,7 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-bitrot
@@ -15,10 +16,10 @@
nodes:
- name: static
label: ubuntu-xenial
+ run: playbooks/project-bitrot.yaml
- project:
name: org/project
periodic:
jobs:
- project-bitrot
-
diff --git a/tests/fixtures/layouts/ignore-dependencies.yaml b/tests/fixtures/layouts/ignore-dependencies.yaml
index 89a82b3..b869dab 100644
--- a/tests/fixtures/layouts/ignore-dependencies.yaml
+++ b/tests/fixtures/layouts/ignore-dependencies.yaml
@@ -15,27 +15,35 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project1-merge
+ run: playbooks/project1-merge.yaml
- job:
name: project1-test1
+ run: playbooks/project1-test1.yaml
- job:
name: project1-test2
+ run: playbooks/project1-test2.yaml
- job:
name: project2-merge
+ run: playbooks/project2-merge.yaml
- job:
name: project2-test1
+ run: playbooks/project2-test1.yaml
- job:
name: project2-test2
+ run: playbooks/project2-test2.yaml
- job:
name: project1-project2-integration
+ run: playbooks/project1-project2-integration.yaml
- project:
name: org/project1
diff --git a/tests/fixtures/layouts/inheritance.yaml b/tests/fixtures/layouts/inheritance.yaml
index 3fe7fd4..35f1402 100644
--- a/tests/fixtures/layouts/inheritance.yaml
+++ b/tests/fixtures/layouts/inheritance.yaml
@@ -14,23 +14,28 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-test-irrelevant-starts-empty
+ run: playbooks/project-test-irrelevant-starts-empty.yaml
- job:
name: project-test-irrelevant-starts-full
irrelevant-files:
- ^README$
- ^ignoreme$
+ run: playbooks/project-test-irrelevant-starts-full.yaml
- job:
name: project-test-nomatch-starts-empty
+ run: playbooks/project-test-nomatch-starts-empty.yaml
- job:
name: project-test-nomatch-starts-full
irrelevant-files:
- ^README$
+ run: playbooks/project-test-nomatch-starts-full.yaml
- project:
name: org/project
diff --git a/tests/fixtures/layouts/irrelevant-files.yaml b/tests/fixtures/layouts/irrelevant-files.yaml
index 97f58e7..80be9af 100644
--- a/tests/fixtures/layouts/irrelevant-files.yaml
+++ b/tests/fixtures/layouts/irrelevant-files.yaml
@@ -14,9 +14,11 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-test-irrelevant-files
+ run: playbooks/project-test-irrelevant-files.yaml
- project:
name: org/project
diff --git a/tests/fixtures/layouts/job-variants.yaml b/tests/fixtures/layouts/job-variants.yaml
new file mode 100644
index 0000000..356034f
--- /dev/null
+++ b/tests/fixtures/layouts/job-variants.yaml
@@ -0,0 +1,64 @@
+- pipeline:
+ name: check
+ manager: independent
+ trigger:
+ gerrit:
+ - event: patchset-created
+ success:
+ gerrit:
+ Verified: 1
+ failure:
+ gerrit:
+ Verified: -1
+
+- job:
+ name: base
+ parent: null
+ pre-run: base-pre
+ post-run: base-post
+ nodeset:
+ nodes:
+ - name: controller
+ label: base
+ run: playbooks/base.yaml
+
+- job:
+ name: python27
+ parent: base
+ timeout: 40
+ pre-run: py27-pre
+ post-run:
+ - py27-post-a
+ - py27-post-b
+ nodeset:
+ nodes:
+ - name: controller
+ label: new
+ run: playbooks/python27.yaml
+
+- job:
+ name: python27
+ timeout: 50
+ branches:
+ - stable/diablo
+ pre-run: py27-diablo-pre
+ run: py27-diablo
+ post-run: py27-diablo-post
+ nodeset:
+ nodes:
+ - name: controller
+ label: old
+
+- job:
+ name: python27
+ branches:
+ - stable/essex
+ pre-run: py27-essex-pre
+ post-run: py27-essex-post
+ run: playbooks/python27.yaml
+
+- project:
+ name: org/project
+ check:
+ jobs:
+ - python27
diff --git a/tests/fixtures/layouts/job-vars.yaml b/tests/fixtures/layouts/job-vars.yaml
index 22fc5c2..e46f084 100644
--- a/tests/fixtures/layouts/job-vars.yaml
+++ b/tests/fixtures/layouts/job-vars.yaml
@@ -14,6 +14,7 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: parentjob
@@ -24,6 +25,7 @@
override: 0
child1override: 0
parent: 0
+ run: playbooks/parentjob.yaml
- job:
name: child1
@@ -34,6 +36,7 @@
override: 1
child1override: 1
child1: 1
+ run: playbooks/child1.yaml
- job:
name: child2
@@ -43,10 +46,12 @@
vars:
override: 2
child2: 2
+ run: playbooks/child2.yaml
- job:
name: child3
parent: parentjob
+ run: playbooks/child3.yaml
- project:
name: org/project
diff --git a/tests/fixtures/layouts/labeling-github.yaml b/tests/fixtures/layouts/labeling-github.yaml
index 2441a9c..fbbf068 100644
--- a/tests/fixtures/layouts/labeling-github.yaml
+++ b/tests/fixtures/layouts/labeling-github.yaml
@@ -7,24 +7,26 @@
- event: pull_request
action: labeled
label:
- - 'test'
+ - test
- event: pull_request
action: unlabeled
label:
- - 'do not test'
+ - do not test
success:
github:
label:
- - 'tests passed'
+ - tests passed
unlabel:
- - 'test'
+ - test
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-labels
+ run: playbooks/project-labels.yaml
- project:
name: org/project
diff --git a/tests/fixtures/layouts/live-reconfiguration-add-job.yaml b/tests/fixtures/layouts/live-reconfiguration-add-job.yaml
index 57d2a5f..c871f1d 100644
--- a/tests/fixtures/layouts/live-reconfiguration-add-job.yaml
+++ b/tests/fixtures/layouts/live-reconfiguration-add-job.yaml
@@ -22,40 +22,46 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-merge
hold-following-changes: true
+ run: playbooks/project-merge.yaml
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
- job:
name: project-test2
+ run: playbooks/project-test2.yaml
- job:
name: project-test3
+ run: playbooks/project-test3.yaml
- job:
name: project-testfile
files:
- - '.*-requires'
+ - .*-requires
+ run: playbooks/project-testfile.yaml
- project:
name: org/project
merge-mode: cherry-pick
gate:
jobs:
- - project-merge
- - project-test1:
- dependencies:
- - project-merge
- - project-test2:
- dependencies:
- - project-merge
- - project-test3:
- dependencies:
- - project-merge
- - project-testfile:
- dependencies:
- - project-merge
+ - project-merge
+ - project-test1:
+ dependencies:
+ - project-merge
+ - project-test2:
+ dependencies:
+ - project-merge
+ - project-test3:
+ dependencies:
+ - project-merge
+ - project-testfile:
+ dependencies:
+ - project-merge
diff --git a/tests/fixtures/layouts/live-reconfiguration-del-project.yaml b/tests/fixtures/layouts/live-reconfiguration-del-project.yaml
index b149af0..259de84 100644
--- a/tests/fixtures/layouts/live-reconfiguration-del-project.yaml
+++ b/tests/fixtures/layouts/live-reconfiguration-del-project.yaml
@@ -14,19 +14,24 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-merge
hold-following-changes: true
+ run: playbooks/project-merge.yaml
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
- job:
name: project-test2
+ run: playbooks/project-test2.yaml
- job:
name: project-testfile
+ run: playbooks/project-testfile.yaml
- project:
name: org/project
diff --git a/tests/fixtures/layouts/live-reconfiguration-failed-job.yaml b/tests/fixtures/layouts/live-reconfiguration-failed-job.yaml
index c4719f4..e5cc651 100644
--- a/tests/fixtures/layouts/live-reconfiguration-failed-job.yaml
+++ b/tests/fixtures/layouts/live-reconfiguration-failed-job.yaml
@@ -14,26 +14,30 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-merge
hold-following-changes: true
+ run: playbooks/project-merge.yaml
- job:
name: project-test2
+ run: playbooks/project-test2.yaml
- job:
name: project-testfile
+ run: playbooks/project-testfile.yaml
- project:
name: org/project
merge-mode: cherry-pick
check:
jobs:
- - project-merge
- - project-test2:
- dependencies:
- - project-merge
- - project-testfile:
- dependencies:
- - project-merge
+ - project-merge
+ - project-test2:
+ dependencies:
+ - project-merge
+ - project-testfile:
+ dependencies:
+ - project-merge
diff --git a/tests/fixtures/layouts/live-reconfiguration-shared-queue.yaml b/tests/fixtures/layouts/live-reconfiguration-shared-queue.yaml
index e363b4c..49a1c2c 100644
--- a/tests/fixtures/layouts/live-reconfiguration-shared-queue.yaml
+++ b/tests/fixtures/layouts/live-reconfiguration-shared-queue.yaml
@@ -35,19 +35,24 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-merge
hold-following-changes: true
+ run: playbooks/project-merge.yaml
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
- job:
name: project-test2
+ run: playbooks/project-test2.yaml
- job:
name: project1-project2-integration
+ run: playbooks/project1-project2-integration.yaml
- project:
name: org/project1
diff --git a/tests/fixtures/layouts/matcher-test.yaml b/tests/fixtures/layouts/matcher-test.yaml
index b511a2f..3239b52 100644
--- a/tests/fixtures/layouts/matcher-test.yaml
+++ b/tests/fixtures/layouts/matcher-test.yaml
@@ -35,6 +35,7 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-test1
@@ -42,6 +43,7 @@
nodes:
- name: controller
label: label1
+ run: playbooks/project-test1.yaml
- job:
name: ignore-branch
@@ -50,6 +52,7 @@
nodes:
- name: controller
label: label2
+ run: playbooks/ignore-branch.yaml
- project:
name: org/project
diff --git a/tests/fixtures/layouts/merge-failure.yaml b/tests/fixtures/layouts/merge-failure.yaml
index 7c5121c..3828a06 100644
--- a/tests/fixtures/layouts/merge-failure.yaml
+++ b/tests/fixtures/layouts/merge-failure.yaml
@@ -23,7 +23,7 @@
name: gate
manager: dependent
failure-message: Build failed. For information on how to proceed, see http://wiki.example.org/Test_Failures
- merge-failure-message: "The merge failed! For more information..."
+ merge-failure-message: The merge failed! For more information...
trigger:
gerrit:
- event: comment-added
@@ -49,16 +49,20 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-merge
hold-following-changes: true
+ run: playbooks/project-merge.yaml
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
- job:
name: project-test2
+ run: playbooks/project-test2.yaml
- project:
name: org/project
diff --git a/tests/fixtures/layouts/merging-github.yaml b/tests/fixtures/layouts/merging-github.yaml
index c9673b9..20df0d4 100644
--- a/tests/fixtures/layouts/merging-github.yaml
+++ b/tests/fixtures/layouts/merging-github.yaml
@@ -2,12 +2,12 @@
name: merge
description: Pipeline for merging the pull request
manager: independent
- merge-failure-message: 'Merge failed'
+ merge-failure-message: Merge failed
trigger:
github:
- event: pull_request
action: comment
- comment: 'merge me'
+ comment: merge me
success:
github:
merge: true
@@ -16,6 +16,7 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- project:
name: org/project
diff --git a/tests/fixtures/layouts/multiple-templates.yaml b/tests/fixtures/layouts/multiple-templates.yaml
index 7272cad..ece8396 100644
--- a/tests/fixtures/layouts/multiple-templates.yaml
+++ b/tests/fixtures/layouts/multiple-templates.yaml
@@ -14,9 +14,11 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: py27
+ run: playbooks/py27.yaml
- project-template:
name: python-jobs
diff --git a/tests/fixtures/layouts/no-jobs-project.yaml b/tests/fixtures/layouts/no-jobs-project.yaml
index 8f965e2..e23f36c 100644
--- a/tests/fixtures/layouts/no-jobs-project.yaml
+++ b/tests/fixtures/layouts/no-jobs-project.yaml
@@ -14,11 +14,13 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-testfile
files:
- .*-requires
+ run: playbooks/project-testfile.yaml
- project:
name: org/no-jobs-project
diff --git a/tests/fixtures/layouts/no-jobs.yaml b/tests/fixtures/layouts/no-jobs.yaml
index 301b27a..7d483ec 100644
--- a/tests/fixtures/layouts/no-jobs.yaml
+++ b/tests/fixtures/layouts/no-jobs.yaml
@@ -35,9 +35,11 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: gate-noop
+ run: playbooks/gate-noop.yaml
- project:
name: org/project
diff --git a/tests/fixtures/layouts/no-run.yaml b/tests/fixtures/layouts/no-run.yaml
new file mode 100644
index 0000000..bccee9c
--- /dev/null
+++ b/tests/fixtures/layouts/no-run.yaml
@@ -0,0 +1,22 @@
+- pipeline:
+ name: check
+ manager: independent
+ trigger:
+ gerrit:
+ - event: patchset-created
+ success:
+ gerrit:
+ Verified: 1
+ failure:
+ gerrit:
+ Verified: -1
+
+- job:
+ name: base
+ parent: null
+
+- project:
+ name: org/project
+ check:
+ jobs:
+ - base
diff --git a/tests/fixtures/layouts/no-timer.yaml b/tests/fixtures/layouts/no-timer.yaml
index 7aaa1ed..67a3244 100644
--- a/tests/fixtures/layouts/no-timer.yaml
+++ b/tests/fixtures/layouts/no-timer.yaml
@@ -23,9 +23,11 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
- job:
name: project-bitrot
@@ -33,6 +35,7 @@
nodes:
- name: static
label: ubuntu-xenial
+ run: playbooks/project-bitrot.yaml
- project:
name: org/project
diff --git a/tests/fixtures/layouts/nonvoting-job.yaml b/tests/fixtures/layouts/nonvoting-job.yaml
index 6a912bf..5b8e9be 100644
--- a/tests/fixtures/layouts/nonvoting-job.yaml
+++ b/tests/fixtures/layouts/nonvoting-job.yaml
@@ -22,17 +22,21 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: nonvoting-project-merge
hold-following-changes: true
+ run: playbooks/nonvoting-project-merge.yaml
- job:
name: nonvoting-project-test1
+ run: playbooks/nonvoting-project-test1.yaml
- job:
name: nonvoting-project-test2
voting: false
+ run: playbooks/nonvoting-project-test2.yaml
- project:
name: org/nonvoting-project
diff --git a/tests/fixtures/layouts/nonvoting-pipeline.yaml b/tests/fixtures/layouts/nonvoting-pipeline.yaml
index d8468dd..afe0528 100644
--- a/tests/fixtures/layouts/nonvoting-pipeline.yaml
+++ b/tests/fixtures/layouts/nonvoting-pipeline.yaml
@@ -12,13 +12,16 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-merge
hold-following-changes: true
+ run: playbooks/project-merge.yaml
- job:
name: experimental-project-test
+ run: playbooks/experimental-project-test.yaml
- project:
name: org/experimental-project
diff --git a/tests/fixtures/layouts/one-job-project.yaml b/tests/fixtures/layouts/one-job-project.yaml
index 4b682d3..d5346f6 100644
--- a/tests/fixtures/layouts/one-job-project.yaml
+++ b/tests/fixtures/layouts/one-job-project.yaml
@@ -43,13 +43,16 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: one-job-project-merge
hold-following-changes: true
+ run: playbooks/one-job-project-merge.yaml
- job:
name: one-job-project-post
+ run: playbooks/one-job-project-post.yaml
- project:
name: org/one-job-project
diff --git a/tests/fixtures/layouts/parent-matchers.yaml b/tests/fixtures/layouts/parent-matchers.yaml
new file mode 100644
index 0000000..2080215
--- /dev/null
+++ b/tests/fixtures/layouts/parent-matchers.yaml
@@ -0,0 +1,38 @@
+- pipeline:
+ name: check
+ manager: independent
+ trigger:
+ gerrit:
+ - event: patchset-created
+ success:
+ gerrit:
+ Verified: 1
+ failure:
+ gerrit:
+ Verified: -1
+
+- job:
+ name: base
+ parent: null
+ run: playbooks/base.yaml
+
+- job:
+ name: parent-job
+ files: foo.txt
+ run: playbooks/parent-job.yaml
+
+- job:
+ name: parent-job
+ files: bar.txt
+ run: playbooks/parent-job.yaml
+
+- job:
+ name: child-job
+ parent: parent-job
+ run: playbooks/child-job.yaml
+
+- project:
+ name: org/project
+ check:
+ jobs:
+ - child-job
diff --git a/tests/fixtures/layouts/push-tag-github.yaml b/tests/fixtures/layouts/push-tag-github.yaml
index 5805127..d689201 100644
--- a/tests/fixtures/layouts/push-tag-github.yaml
+++ b/tests/fixtures/layouts/push-tag-github.yaml
@@ -4,7 +4,7 @@
trigger:
github:
- event: push
- ref: '^refs/heads/master$'
+ ref: ^refs/heads/master$
- pipeline:
name: tag
@@ -17,12 +17,15 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-post
+ run: playbooks/project-post.yaml
- job:
name: project-tag
+ run: playbooks/project-tag.yaml
- project:
name: org/project
diff --git a/tests/fixtures/layouts/rate-limit.yaml b/tests/fixtures/layouts/rate-limit.yaml
index 1f32dbf..b432d51 100644
--- a/tests/fixtures/layouts/rate-limit.yaml
+++ b/tests/fixtures/layouts/rate-limit.yaml
@@ -27,15 +27,19 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-merge
+ run: playbooks/project-merge.yaml
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
- job:
name: project-test2
+ run: playbooks/project-test2.yaml
- project:
name: org/project
diff --git a/tests/fixtures/layouts/reconfigure-failed-head.yaml b/tests/fixtures/layouts/reconfigure-failed-head.yaml
new file mode 100644
index 0000000..3347c9b
--- /dev/null
+++ b/tests/fixtures/layouts/reconfigure-failed-head.yaml
@@ -0,0 +1,56 @@
+- pipeline:
+ name: check
+ manager: independent
+ trigger:
+ gerrit:
+ - event: patchset-created
+ success:
+ gerrit:
+ Verified: 1
+ failure:
+ gerrit:
+ Verified: -1
+
+- pipeline:
+ name: gate
+ manager: dependent
+ success-message: Build succeeded (gate).
+ trigger:
+ gerrit:
+ - event: comment-added
+ approval:
+ - Approved: 1
+ success:
+ gerrit:
+ Verified: 2
+ submit: true
+ failure:
+ gerrit:
+ Verified: -2
+ start:
+ gerrit:
+ Verified: 0
+ precedence: high
+
+- job:
+ name: base
+ parent: null
+
+- job:
+ name: job1
+ run: playbooks/job1.yaml
+
+- job:
+ name: job2
+ run: playbooks/job2.yaml
+
+- project:
+ name: org/project
+ check:
+ jobs:
+ - job1
+ - job2
+ gate:
+ jobs:
+ - job1
+ - job2
diff --git a/tests/fixtures/layouts/repo-checkout-four-project.yaml b/tests/fixtures/layouts/repo-checkout-four-project.yaml
index 17303f5..11212e8 100644
--- a/tests/fixtures/layouts/repo-checkout-four-project.yaml
+++ b/tests/fixtures/layouts/repo-checkout-four-project.yaml
@@ -35,6 +35,7 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: integration
@@ -43,6 +44,7 @@
- org/project2
- org/project3
- org/project4
+ run: playbooks/integration.yaml
- project:
name: org/project1
diff --git a/tests/fixtures/layouts/repo-checkout-no-timer-override.yaml b/tests/fixtures/layouts/repo-checkout-no-timer-override.yaml
index 4680869..89d2b93 100644
--- a/tests/fixtures/layouts/repo-checkout-no-timer-override.yaml
+++ b/tests/fixtures/layouts/repo-checkout-no-timer-override.yaml
@@ -10,13 +10,15 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: integration
branches: master
- override-branch: stable/havana
+ override-checkout: stable/havana
required-projects:
- org/project1
+ run: playbooks/integration.yaml
- project:
name: org/project1
diff --git a/tests/fixtures/layouts/repo-checkout-no-timer.yaml b/tests/fixtures/layouts/repo-checkout-no-timer.yaml
index ed20bb1..0374897 100644
--- a/tests/fixtures/layouts/repo-checkout-no-timer.yaml
+++ b/tests/fixtures/layouts/repo-checkout-no-timer.yaml
@@ -10,12 +10,14 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: integration
- override-branch: stable/havana
+ override-checkout: stable/havana
required-projects:
- org/project1
+ run: playbooks/integration.yaml
- project:
name: org/project1
diff --git a/tests/fixtures/layouts/repo-checkout-post.yaml b/tests/fixtures/layouts/repo-checkout-post.yaml
index 191569c..2e702bc 100644
--- a/tests/fixtures/layouts/repo-checkout-post.yaml
+++ b/tests/fixtures/layouts/repo-checkout-post.yaml
@@ -9,12 +9,14 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: integration
required-projects:
- org/project1
- org/project2
+ run: playbooks/integration.yaml
- project:
name: org/project1
diff --git a/tests/fixtures/layouts/repo-checkout-six-project.yaml b/tests/fixtures/layouts/repo-checkout-six-project.yaml
index 9a81eae..4878665 100644
--- a/tests/fixtures/layouts/repo-checkout-six-project.yaml
+++ b/tests/fixtures/layouts/repo-checkout-six-project.yaml
@@ -35,6 +35,7 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: integration
@@ -43,9 +44,10 @@
- org/project2
- org/project3
- name: org/project4
- override-branch: master
+ override-checkout: master
- org/project5
- org/project6
+ run: playbooks/integration.yaml
- project:
name: org/project1
diff --git a/tests/fixtures/layouts/repo-checkout-tag.yaml b/tests/fixtures/layouts/repo-checkout-tag.yaml
new file mode 100644
index 0000000..3f1af1c
--- /dev/null
+++ b/tests/fixtures/layouts/repo-checkout-tag.yaml
@@ -0,0 +1,36 @@
+- pipeline:
+ name: check
+ manager: independent
+ trigger:
+ gerrit:
+ - event: patchset-created
+ success:
+ gerrit:
+ Verified: 1
+ failure:
+ gerrit:
+ Verified: -1
+
+- job:
+ name: base
+ parent: null
+
+- job:
+ name: integration
+ required-projects:
+ - org/project1
+ - name: org/project2
+ override-checkout: test-tag
+ run: playbooks/integration.yaml
+
+- project:
+ name: org/project1
+ check:
+ jobs:
+ - integration
+
+- project:
+ name: org/project2
+ check:
+ jobs:
+ - integration
diff --git a/tests/fixtures/layouts/repo-checkout-timer-override.yaml b/tests/fixtures/layouts/repo-checkout-timer-override.yaml
index 99fc4f5..4aacfee 100644
--- a/tests/fixtures/layouts/repo-checkout-timer-override.yaml
+++ b/tests/fixtures/layouts/repo-checkout-timer-override.yaml
@@ -8,13 +8,15 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: integration
branches: master
- override-branch: stable/havana
+ override-checkout: stable/havana
required-projects:
- org/project1
+ run: playbooks/integration.yaml
- project:
name: org/project1
diff --git a/tests/fixtures/layouts/repo-checkout-timer.yaml b/tests/fixtures/layouts/repo-checkout-timer.yaml
index e707732..739c066 100644
--- a/tests/fixtures/layouts/repo-checkout-timer.yaml
+++ b/tests/fixtures/layouts/repo-checkout-timer.yaml
@@ -8,11 +8,13 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: integration
required-projects:
- org/project1
+ run: playbooks/integration.yaml
- project:
name: org/project1
diff --git a/tests/fixtures/layouts/repo-checkout-two-project.yaml b/tests/fixtures/layouts/repo-checkout-two-project.yaml
index 7910ae7..64c6ee9 100644
--- a/tests/fixtures/layouts/repo-checkout-two-project.yaml
+++ b/tests/fixtures/layouts/repo-checkout-two-project.yaml
@@ -35,12 +35,14 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: integration
required-projects:
- org/project1
- org/project2
+ run: playbooks/integration.yaml
- project:
name: org/project1
diff --git a/tests/fixtures/layouts/repo-deleted.yaml b/tests/fixtures/layouts/repo-deleted.yaml
index 3a7f6b3..2ee8ebd 100644
--- a/tests/fixtures/layouts/repo-deleted.yaml
+++ b/tests/fixtures/layouts/repo-deleted.yaml
@@ -35,10 +35,12 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-merge
hold-following-changes: true
+ run: playbooks/project-merge.yaml
- job:
name: project-test1
@@ -46,6 +48,7 @@
nodes:
- name: controller
label: label1
+ run: playbooks/project-test1.yaml
- job:
name: project-test1
@@ -54,9 +57,11 @@
nodes:
- name: controller
label: label2
+ run: playbooks/project-test1.yaml
- job:
name: project-test2
+ run: playbooks/project-test2.yaml
- project:
name: org/delete-project
diff --git a/tests/fixtures/layouts/reporting-github.yaml b/tests/fixtures/layouts/reporting-github.yaml
index 159f205..c909cf4 100644
--- a/tests/fixtures/layouts/reporting-github.yaml
+++ b/tests/fixtures/layouts/reporting-github.yaml
@@ -8,11 +8,11 @@
action: opened
start:
github:
- status: 'pending'
+ status: pending
comment: false
success:
github:
- status: 'success'
+ status: success
- pipeline:
name: reporting
@@ -22,13 +22,13 @@
github:
- event: pull_request
action: comment
- comment: 'reporting check'
+ comment: reporting check
start:
github: {}
success:
github:
comment: false
- status: 'success'
+ status: success
status-url: http://logs.example.com/{tenant.name}/{pipeline.name}/{change.project}/{change.number}/{buildset.uuid}/
failure:
github:
@@ -42,14 +42,14 @@
github:
- event: pull_request
action: comment
- comment: 'long pipeline'
+ comment: long pipeline
start:
github:
- status: 'pending'
+ status: pending
success:
github:
comment: false
- status: 'success'
+ status: success
status-url: http://logs.example.com/{tenant.name}/{pipeline.name}/{change.project}/{change.number}/{buildset.uuid}/
failure:
github:
@@ -67,23 +67,25 @@
start:
github:
comment: true
- status: 'pending'
+ status: pending
success:
github:
comment: true
- status: 'success'
+ status: success
merge: true
failure:
github:
comment: true
- status: 'failure'
+ status: failure
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
- project:
name: org/project
diff --git a/tests/fixtures/layouts/reporting-multiple-github.yaml b/tests/fixtures/layouts/reporting-multiple-github.yaml
index 0126ec5..67a237a 100644
--- a/tests/fixtures/layouts/reporting-multiple-github.yaml
+++ b/tests/fixtures/layouts/reporting-multiple-github.yaml
@@ -11,26 +11,29 @@
action: opened
start:
github:
- status: 'pending'
+ status: pending
comment: false
github_ent:
- status: 'pending'
+ status: pending
comment: false
success:
github:
- status: 'success'
+ status: success
github_ent:
- status: 'success'
+ status: success
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project1-test1
+ run: playbooks/project1-test1.yaml
- job:
name: project2-test2
+ run: playbooks/project2-test2.yaml
- project:
name: org/project1
diff --git a/tests/fixtures/layouts/requirements-github.yaml b/tests/fixtures/layouts/requirements-github.yaml
index f2ecd16..92bd9cb 100644
--- a/tests/fixtures/layouts/requirements-github.yaml
+++ b/tests/fixtures/layouts/requirements-github.yaml
@@ -3,12 +3,12 @@
manager: independent
require:
github:
- status: "zuul:check:success"
+ status: zuul:check:success
trigger:
github:
- event: pull_request
action: comment
- comment: 'test me'
+ comment: test me
success:
github:
comment: true
@@ -20,8 +20,8 @@
github:
- event: pull_request
action: comment
- comment: 'trigger me'
- require-status: "zuul:check:success"
+ comment: trigger me
+ require-status: zuul:check:success
success:
github:
comment: true
@@ -33,13 +33,13 @@
github:
- event: pull_request
action: status
- status: 'zuul:check:success'
+ status: zuul:check:success
success:
github:
- status: 'success'
+ status: success
failure:
github:
- status: 'failure'
+ status: failure
- pipeline:
name: reviewusername
@@ -47,13 +47,13 @@
require:
github:
review:
- - username: '^(herp|derp)$'
+ - username: ^(herp|derp)$
type: approved
trigger:
github:
- event: pull_request
action: comment
- comment: 'test me'
+ comment: test me
success:
github:
comment: true
@@ -75,7 +75,7 @@
github:
- event: pull_request
action: comment
- comment: 'test me'
+ comment: test me
success:
github:
comment: true
@@ -86,7 +86,7 @@
require:
github:
review:
- - username: 'derp'
+ - username: derp
type: approved
permission: write
reject:
@@ -98,7 +98,7 @@
github:
- event: pull_request
action: comment
- comment: 'test me'
+ comment: test me
success:
github:
comment: true
@@ -116,7 +116,7 @@
github:
- event: pull_request
action: comment
- comment: 'test me'
+ comment: test me
success:
github:
comment: true
@@ -134,7 +134,7 @@
github:
- event: pull_request
action: comment
- comment: 'test me'
+ comment: test me
success:
github:
comment: true
@@ -149,7 +149,7 @@
github:
- event: pull_request
action: comment
- comment: 'test me'
+ comment: test me
success:
github:
comment: true
@@ -178,7 +178,7 @@
github:
- event: pull_request
action: comment
- comment: 'test me'
+ comment: test me
success:
github:
comment: true
@@ -186,36 +186,47 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project1-pipeline
+ run: playbooks/project1-pipeline.yaml
- job:
name: project2-trigger
+ run: playbooks/project2-trigger.yaml
- job:
name: project3-reviewusername
+ run: playbooks/project3-reviewusername.yaml
- job:
name: project4-reviewreq
+ run: playbooks/project4-reviewreq.yaml
- job:
name: project5-reviewuserstate
+ run: playbooks/project5-reviewuserstate.yaml
- job:
name: project6-newerthan
+ run: playbooks/project6-newerthan.yaml
- job:
name: project7-olderthan
+ run: playbooks/project7-olderthan.yaml
- job:
name: project8-requireopen
+ run: playbooks/project8-requireopen.yaml
- job:
name: project9-requirecurrent
+ run: playbooks/project9-requirecurrent.yaml
- job:
name: project10-label
+ run: playbooks/project10-label.yaml
- project:
name: org/project1
diff --git a/tests/fixtures/layouts/reviews-github.yaml b/tests/fixtures/layouts/reviews-github.yaml
index f186fbe..abc3d99 100644
--- a/tests/fixtures/layouts/reviews-github.yaml
+++ b/tests/fixtures/layouts/reviews-github.yaml
@@ -5,18 +5,20 @@
github:
- event: pull_request_review
action: submitted
- state: 'approve'
+ state: approve
success:
github:
label:
- - 'tests passed'
+ - tests passed
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-reviews
+ run: playbooks/project-reviews.yaml
- project:
name: org/project
diff --git a/tests/fixtures/layouts/smtp.yaml b/tests/fixtures/layouts/smtp.yaml
index 0654448..77391a0 100644
--- a/tests/fixtures/layouts/smtp.yaml
+++ b/tests/fixtures/layouts/smtp.yaml
@@ -41,10 +41,12 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-merge
hold-following-changes: true
+ run: playbooks/project-merge.yaml
- job:
name: project-test1
@@ -52,6 +54,7 @@
nodes:
- name: controller
label: label1
+ run: playbooks/project-test1.yaml
- job:
name: project-test1
@@ -60,9 +63,11 @@
nodes:
- name: controller
label: label2
+ run: playbooks/project-test1.yaml
- job:
name: project-test2
+ run: playbooks/project-test2.yaml
- project:
name: org/project
diff --git a/tests/fixtures/layouts/tags.yaml b/tests/fixtures/layouts/tags.yaml
index f86f5ab..2fda2db 100644
--- a/tests/fixtures/layouts/tags.yaml
+++ b/tests/fixtures/layouts/tags.yaml
@@ -14,11 +14,13 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: merge
tags:
- merge
+ run: playbooks/merge.yaml
- project:
name: org/project1
diff --git a/tests/fixtures/layouts/three-projects.yaml b/tests/fixtures/layouts/three-projects.yaml
index 51cd406..33e81ac 100644
--- a/tests/fixtures/layouts/three-projects.yaml
+++ b/tests/fixtures/layouts/three-projects.yaml
@@ -35,19 +35,24 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-merge
hold-following-changes: true
+ run: playbooks/project-merge.yaml
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
- job:
name: project-test2
+ run: playbooks/project-test2.yaml
- job:
name: project1-project2-integration
+ run: playbooks/project1-project2-integration.yaml
- project:
name: org/project1
diff --git a/tests/fixtures/layouts/timer-smtp.yaml b/tests/fixtures/layouts/timer-smtp.yaml
index a27b183..d9e4282 100644
--- a/tests/fixtures/layouts/timer-smtp.yaml
+++ b/tests/fixtures/layouts/timer-smtp.yaml
@@ -13,14 +13,17 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-bitrot-stable-old
success-url: http://logs.example.com/{job.name}/{build.number}
+ run: playbooks/project-bitrot-stable-old.yaml
- job:
name: project-bitrot-stable-older
success-url: http://logs.example.com/{job.name}/{build.number}
+ run: playbooks/project-bitrot-stable-older.yaml
- project:
name: org/project
diff --git a/tests/fixtures/layouts/timer.yaml b/tests/fixtures/layouts/timer.yaml
index 8c0cc2b..e9e9a17 100644
--- a/tests/fixtures/layouts/timer.yaml
+++ b/tests/fixtures/layouts/timer.yaml
@@ -21,12 +21,15 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
- job:
name: project-test2
+ run: playbooks/project-test2.yaml
- job:
name: project-bitrot
@@ -34,6 +37,7 @@
nodes:
- name: static
label: ubuntu-xenial
+ run: playbooks/project-bitrot.yaml
- project:
name: org/project
diff --git a/tests/fixtures/layouts/untrusted-secrets.yaml b/tests/fixtures/layouts/untrusted-secrets.yaml
index b90d3d7..337587a 100644
--- a/tests/fixtures/layouts/untrusted-secrets.yaml
+++ b/tests/fixtures/layouts/untrusted-secrets.yaml
@@ -14,10 +14,12 @@
- job:
name: base
parent: null
+ run: playbooks/base.yaml
- job:
name: project1-test
post-review: true
+ run: playbooks/project1-test.yaml
- project:
name: org/project1
diff --git a/tests/fixtures/zuul-connections-cgit.conf b/tests/fixtures/zuul-connections-cgit.conf
new file mode 100644
index 0000000..39dc0bb
--- /dev/null
+++ b/tests/fixtures/zuul-connections-cgit.conf
@@ -0,0 +1,27 @@
+[gearman]
+server=127.0.0.1
+
+[scheduler]
+tenant_config=main.yaml
+
+[merger]
+git_dir=/tmp/zuul-test/merger-git
+git_user_email=zuul@example.com
+git_user_name=zuul
+
+[executor]
+git_dir=/tmp/zuul-test/executor-git
+
+[connection gerrit]
+driver=gerrit
+server=review.example.com
+user=jenkins
+sshkey=none
+gitweb_url_template=https://cgit.example.com/cgit/{project.name}/commit/?id={sha}
+
+[connection outgoing_smtp]
+driver=smtp
+server=localhost
+port=25
+default_from=zuul@example.com
+default_to=you@example.com
diff --git a/tests/fixtures/zuul-connections-gitweb.conf b/tests/fixtures/zuul-connections-gitweb.conf
new file mode 100644
index 0000000..172208e
--- /dev/null
+++ b/tests/fixtures/zuul-connections-gitweb.conf
@@ -0,0 +1,26 @@
+[gearman]
+server=127.0.0.1
+
+[scheduler]
+tenant_config=main.yaml
+
+[merger]
+git_dir=/tmp/zuul-test/merger-git
+git_user_email=zuul@example.com
+git_user_name=zuul
+
+[executor]
+git_dir=/tmp/zuul-test/executor-git
+
+[connection gerrit]
+driver=gerrit
+server=review.example.com
+user=jenkins
+sshkey=none
+
+[connection outgoing_smtp]
+driver=smtp
+server=localhost
+port=25
+default_from=zuul@example.com
+default_to=you@example.com
diff --git a/tests/fixtures/zuul-executor-hostname.conf b/tests/fixtures/zuul-executor-hostname.conf
new file mode 100644
index 0000000..8199aba
--- /dev/null
+++ b/tests/fixtures/zuul-executor-hostname.conf
@@ -0,0 +1,32 @@
+[gearman]
+server=127.0.0.1
+
+[statsd]
+# note, use 127.0.0.1 rather than localhost to avoid getting ipv6
+# see: https://github.com/jsocol/pystatsd/issues/61
+server=127.0.0.1
+
+[scheduler]
+tenant_config=main.yaml
+
+[merger]
+git_dir=/tmp/zuul-test/merger-git
+git_user_email=zuul@example.com
+git_user_name=zuul
+
+[executor]
+git_dir=/tmp/zuul-test/executor-git
+hostname=test-executor-hostname.example.com
+
+[connection gerrit]
+driver=gerrit
+server=review.example.com
+user=jenkins
+sshkey=fake_id_rsa_path
+
+[connection smtp]
+driver=smtp
+server=localhost
+port=25
+default_from=zuul@example.com
+default_to=you@example.com
diff --git a/tests/unit/test_connection.py b/tests/unit/test_connection.py
index 1e12bce..c882d3a 100644
--- a/tests/unit/test_connection.py
+++ b/tests/unit/test_connection.py
@@ -69,7 +69,7 @@
insp = sa.engine.reflection.Inspector(
self.connections.connections['resultsdb'].engine)
- self.assertEqual(10, len(insp.get_columns(buildset_table)))
+ self.assertEqual(13, len(insp.get_columns(buildset_table)))
self.assertEqual(10, len(insp.get_columns(build_table)))
def test_sql_results(self):
@@ -114,6 +114,8 @@
self.assertEqual('SUCCESS', buildset0['result'])
self.assertEqual('Build succeeded.', buildset0['message'])
self.assertEqual('tenant-one', buildset0['tenant'])
+ self.assertEqual('https://hostname/%d' % buildset0['change'],
+ buildset0['ref_url'])
buildset0_builds = conn.execute(
sa.sql.select([reporter.connection.zuul_build_table]).
@@ -336,3 +338,32 @@
self.assertNotIn("sql", self.connections.connections)
self.assertNotIn("timer", self.connections.connections)
self.assertNotIn("zuul", self.connections.connections)
+
+
+class TestConnectionsCgit(ZuulTestCase):
+ config_file = 'zuul-connections-cgit.conf'
+ tenant_config_file = 'config/single-tenant/main.yaml'
+
+ def test_cgit_web_url(self):
+ self.assertIn("gerrit", self.connections.connections)
+ conn = self.connections.connections['gerrit']
+ source = conn.source
+ proj = source.getProject('foo/bar')
+ url = conn._getWebUrl(proj, '1')
+ self.assertEqual(url,
+ 'https://cgit.example.com/cgit/foo/bar/commit/?id=1')
+
+
+class TestConnectionsGitweb(ZuulTestCase):
+ config_file = 'zuul-connections-gitweb.conf'
+ tenant_config_file = 'config/single-tenant/main.yaml'
+
+ def test_gitweb_url(self):
+ self.assertIn("gerrit", self.connections.connections)
+ conn = self.connections.connections['gerrit']
+ source = conn.source
+ proj = source.getProject('foo/bar')
+ url = conn._getWebUrl(proj, '1')
+ url_should_be = 'https://review.example.com/' \
+ 'gitweb?p=foo/bar.git;a=commitdiff;h=1'
+ self.assertEqual(url, url_should_be)
diff --git a/tests/unit/test_executor.py b/tests/unit/test_executor.py
index 9c45645..5d27663 100755
--- a/tests/unit/test_executor.py
+++ b/tests/unit/test_executor.py
@@ -378,6 +378,32 @@
self.assertBuildStates(states, projects)
+ @simple_layout('layouts/repo-checkout-tag.yaml')
+ def test_tag_checkout(self):
+ self.executor_server.hold_jobs_in_build = True
+ p1 = "review.example.com/org/project1"
+ p2 = "review.example.com/org/project2"
+ projects = [p1, p2]
+ upstream = self.getUpstreamRepos(projects)
+
+ self.create_branch('org/project2', 'stable/havana')
+ files = {'README': 'tagged readme'}
+ self.addCommitToRepo('org/project2', 'tagged commit',
+ files, branch='stable/havana', tag='test-tag')
+
+ A = self.fake_gerrit.addFakeChange('org/project1', 'master', 'A')
+ self.fake_gerrit.addEvent(A.getPatchsetCreatedEvent(1))
+ self.waitUntilSettled()
+
+ states = [
+ {p1: dict(present=[A], branch='master'),
+ p2: dict(commit=str(upstream[p2].commit('test-tag')),
+ absent=[A]),
+ },
+ ]
+
+ self.assertBuildStates(states, projects)
+
class TestAnsibleJob(ZuulTestCase):
tenant_config_file = 'config/ansible/main.yaml'
@@ -401,3 +427,12 @@
node['ssh_port'] = 22022
keys = self.test_job.getHostList({'nodes': [node]})[0]['host_keys']
self.assertEqual(keys[0], '[localhost]:22022 fake-host-key')
+
+
+class TestExecutorHostname(ZuulTestCase):
+ config_file = 'zuul-executor-hostname.conf'
+ tenant_config_file = 'config/single-tenant/main.yaml'
+
+ def test_executor_hostname(self):
+ self.assertEqual('test-executor-hostname.example.com',
+ self.executor_server.hostname)
diff --git a/tests/unit/test_inventory.py b/tests/unit/test_inventory.py
index 2835d30..04dcb05 100644
--- a/tests/unit/test_inventory.py
+++ b/tests/unit/test_inventory.py
@@ -80,3 +80,24 @@
self.executor_server.release()
self.waitUntilSettled()
+
+ def test_hostvars_inventory(self):
+
+ inventory = self._get_build_inventory('hostvars-inventory')
+
+ all_nodes = ('default', 'fakeuser')
+ self.assertIn('all', inventory)
+ self.assertIn('hosts', inventory['all'])
+ self.assertIn('vars', inventory['all'])
+ for node_name in all_nodes:
+ self.assertIn(node_name, inventory['all']['hosts'])
+ # check if the nodes use the correct username
+ if node_name == 'fakeuser':
+ username = 'fakeuser'
+ else:
+ username = 'zuul'
+ self.assertEqual(
+ inventory['all']['hosts'][node_name]['ansible_user'], username)
+
+ self.executor_server.release()
+ self.waitUntilSettled()
diff --git a/tests/unit/test_model.py b/tests/unit/test_model.py
index 628a45c..784fcb3 100644
--- a/tests/unit/test_model.py
+++ b/tests/unit/test_model.py
@@ -96,30 +96,6 @@
def test_job_sets_defaults_for_boolean_attributes(self):
self.assertIsNotNone(self.job.voting)
- def test_job_inheritance(self):
- # This is standard job inheritance.
-
- base_pre = model.PlaybookContext(self.context, 'base-pre', [], [])
- base_run = model.PlaybookContext(self.context, 'base-run', [], [])
- base_post = model.PlaybookContext(self.context, 'base-post', [], [])
-
- base = model.Job('base')
- base.timeout = 30
- base.pre_run = [base_pre]
- base.run = [base_run]
- base.post_run = [base_post]
-
- py27 = model.Job('py27')
- self.assertIsNone(py27.timeout)
- py27.inheritFrom(base)
- self.assertEqual(30, py27.timeout)
- self.assertEqual(['base-pre'],
- [x.path for x in py27.pre_run])
- self.assertEqual(['base-run'],
- [x.path for x in py27.run])
- self.assertEqual(['base-post'],
- [x.path for x in py27.post_run])
-
def test_job_variants(self):
# This simulates freezing a job.
@@ -170,342 +146,6 @@
"Unable to modify final job"):
job.applyVariant(bad_final)
- def test_job_inheritance_configloader(self):
- # TODO(jeblair): move this to a configloader test
- tenant = model.Tenant('tenant')
- layout = model.Layout(tenant)
-
- pipeline = model.Pipeline('gate', layout)
- layout.addPipeline(pipeline)
- queue = model.ChangeQueue(pipeline)
- project = model.Project('project', self.source)
- tpc = model.TenantProjectConfig(project)
- tenant.addUntrustedProject(tpc)
-
- base = configloader.JobParser.fromYaml(tenant, layout, {
- '_source_context': self.context,
- '_start_mark': self.start_mark,
- 'name': 'base',
- 'parent': None,
- 'timeout': 30,
- 'pre-run': 'base-pre',
- 'post-run': 'base-post',
- 'nodeset': {
- 'nodes': [{
- 'name': 'controller',
- 'label': 'base',
- }],
- },
- })
- layout.addJob(base)
- python27 = configloader.JobParser.fromYaml(tenant, layout, {
- '_source_context': self.context,
- '_start_mark': self.start_mark,
- 'name': 'python27',
- 'parent': 'base',
- 'pre-run': 'py27-pre',
- 'post-run': ['py27-post-a', 'py27-post-b'],
- 'nodeset': {
- 'nodes': [{
- 'name': 'controller',
- 'label': 'new',
- }],
- },
- 'timeout': 40,
- })
- layout.addJob(python27)
- python27diablo = configloader.JobParser.fromYaml(tenant, layout, {
- '_source_context': self.context,
- '_start_mark': self.start_mark,
- 'name': 'python27',
- 'branches': [
- 'stable/diablo'
- ],
- 'pre-run': 'py27-diablo-pre',
- 'run': 'py27-diablo',
- 'post-run': 'py27-diablo-post',
- 'nodeset': {
- 'nodes': [{
- 'name': 'controller',
- 'label': 'old',
- }],
- },
- 'timeout': 50,
- })
- layout.addJob(python27diablo)
-
- python27essex = configloader.JobParser.fromYaml(tenant, layout, {
- '_source_context': self.context,
- '_start_mark': self.start_mark,
- 'name': 'python27',
- 'branches': [
- 'stable/essex'
- ],
- 'pre-run': 'py27-essex-pre',
- 'post-run': 'py27-essex-post',
- })
- layout.addJob(python27essex)
-
- project_template_parser = configloader.ProjectTemplateParser(
- tenant, layout)
- project_parser = configloader.ProjectParser(
- tenant, layout, project_template_parser)
- project_config = project_parser.fromYaml([{
- '_source_context': self.context,
- '_start_mark': self.start_mark,
- 'name': 'project',
- 'gate': {
- 'jobs': [
- 'python27'
- ]
- }
- }])
- layout.addProjectConfig(project_config)
-
- change = model.Change(project)
- # Test master
- change.branch = 'master'
- item = queue.enqueueChange(change)
- item.layout = layout
-
- self.assertTrue(base.changeMatches(change))
- self.assertTrue(python27.changeMatches(change))
- self.assertFalse(python27diablo.changeMatches(change))
- self.assertFalse(python27essex.changeMatches(change))
-
- item.freezeJobGraph()
- self.assertEqual(len(item.getJobs()), 1)
- job = item.getJobs()[0]
- self.assertEqual(job.name, 'python27')
- self.assertEqual(job.timeout, 40)
- nodes = job.nodeset.getNodes()
- self.assertEqual(len(nodes), 1)
- self.assertEqual(nodes[0].label, 'new')
- self.assertEqual([x.path for x in job.pre_run],
- ['base-pre',
- 'py27-pre'])
- self.assertEqual([x.path for x in job.post_run],
- ['py27-post-a',
- 'py27-post-b',
- 'base-post'])
- self.assertEqual([x.path for x in job.run],
- ['playbooks/python27',
- 'playbooks/base'])
-
- # Test diablo
- change.branch = 'stable/diablo'
- item = queue.enqueueChange(change)
- item.layout = layout
-
- self.assertTrue(base.changeMatches(change))
- self.assertTrue(python27.changeMatches(change))
- self.assertTrue(python27diablo.changeMatches(change))
- self.assertFalse(python27essex.changeMatches(change))
-
- item.freezeJobGraph()
- self.assertEqual(len(item.getJobs()), 1)
- job = item.getJobs()[0]
- self.assertEqual(job.name, 'python27')
- self.assertEqual(job.timeout, 50)
- nodes = job.nodeset.getNodes()
- self.assertEqual(len(nodes), 1)
- self.assertEqual(nodes[0].label, 'old')
- self.assertEqual([x.path for x in job.pre_run],
- ['base-pre',
- 'py27-pre',
- 'py27-diablo-pre'])
- self.assertEqual([x.path for x in job.post_run],
- ['py27-diablo-post',
- 'py27-post-a',
- 'py27-post-b',
- 'base-post'])
- self.assertEqual([x.path for x in job.run],
- ['py27-diablo']),
-
- # Test essex
- change.branch = 'stable/essex'
- item = queue.enqueueChange(change)
- item.layout = layout
-
- self.assertTrue(base.changeMatches(change))
- self.assertTrue(python27.changeMatches(change))
- self.assertFalse(python27diablo.changeMatches(change))
- self.assertTrue(python27essex.changeMatches(change))
-
- item.freezeJobGraph()
- self.assertEqual(len(item.getJobs()), 1)
- job = item.getJobs()[0]
- self.assertEqual(job.name, 'python27')
- self.assertEqual([x.path for x in job.pre_run],
- ['base-pre',
- 'py27-pre',
- 'py27-essex-pre'])
- self.assertEqual([x.path for x in job.post_run],
- ['py27-essex-post',
- 'py27-post-a',
- 'py27-post-b',
- 'base-post'])
- self.assertEqual([x.path for x in job.run],
- ['playbooks/python27',
- 'playbooks/base'])
-
- def test_job_auth_inheritance(self):
- tenant = self.tenant
- layout = self.layout
-
- conf = yaml.safe_load('''
-- secret:
- name: trusted-secret
- data:
- username: test-username
- longpassword: !encrypted/pkcs1-oaep
- - BFhtdnm8uXx7kn79RFL/zJywmzLkT1GY78P3bOtp4WghUFWobkifSu7ZpaV4NeO0s71Y
- Usi1wGZZL0LveZjUN0t6OU1VZKSG8R5Ly7urjaSo1pPVIq5Rtt/H7W14Lecd+cUeKb4j
- oeusC9drN3AA8a4oykcVpt1wVqUnTbMGC9ARMCQP6eopcs1l7tzMseprW4RDNhIuz3CR
- gd0QBMPl6VDoFgBPB8vxtJw+3m0rqBYZCLZgCXekqlny8s2s92nJMuUABbJOEcDRarzi
- bDsSXsfJt1y+5n7yOURsC7lovMg4GF/vCl/0YMKjBO5bpv9EM5fToeKYyPGSKQoHOnCY
- ceb3cAVcv5UawcCic8XjhEhp4K7WPdYf2HVAC/qtxhbpjTxG4U5Q/SoppOJ60WqEkQvb
- Xs6n5Dvy7xmph6GWmU/bAv3eUK3pdD3xa2Ue1lHWz3U+rsYraI+AKYsMYx3RBlfAmCeC
- 1ve2BXPrqnOo7G8tnUvfdYPbK4Aakk0ds/AVqFHEZN+S6hRBmBjLaRFWZ3QSO1NjbBxW
- naHKZYT7nkrJm8AMCgZU0ZArFLpaufKCeiK5ECSsDxic4FIsY1OkWT42qEUfL0Wd+150
- AKGNZpPJnnP3QYY4W/MWcKH/zdO400+zWN52WevbSqZy90tqKDJrBkMl1ydqbuw1E4ZH
- vIs=
- - BFhtdnm8uXx7kn79RFL/zJywmzLkT1GY78P3bOtp4WghUFWobkifSu7ZpaV4NeO0s71Y
- Usi1wGZZL0LveZjUN0t6OU1VZKSG8R5Ly7urjaSo1pPVIq5Rtt/H7W14Lecd+cUeKb4j
- oeusC9drN3AA8a4oykcVpt1wVqUnTbMGC9ARMCQP6eopcs1l7tzMseprW4RDNhIuz3CR
- gd0QBMPl6VDoFgBPB8vxtJw+3m0rqBYZCLZgCXekqlny8s2s92nJMuUABbJOEcDRarzi
- bDsSXsfJt1y+5n7yOURsC7lovMg4GF/vCl/0YMKjBO5bpv9EM5fToeKYyPGSKQoHOnCY
- ceb3cAVcv5UawcCic8XjhEhp4K7WPdYf2HVAC/qtxhbpjTxG4U5Q/SoppOJ60WqEkQvb
- Xs6n5Dvy7xmph6GWmU/bAv3eUK3pdD3xa2Ue1lHWz3U+rsYraI+AKYsMYx3RBlfAmCeC
- 1ve2BXPrqnOo7G8tnUvfdYPbK4Aakk0ds/AVqFHEZN+S6hRBmBjLaRFWZ3QSO1NjbBxW
- naHKZYT7nkrJm8AMCgZU0ZArFLpaufKCeiK5ECSsDxic4FIsY1OkWT42qEUfL0Wd+150
- AKGNZpPJnnP3QYY4W/MWcKH/zdO400+zWN52WevbSqZy90tqKDJrBkMl1ydqbuw1E4ZH
- vIs=
- password: !encrypted/pkcs1-oaep |
- BFhtdnm8uXx7kn79RFL/zJywmzLkT1GY78P3bOtp4WghUFWobkifSu7ZpaV4NeO0s71Y
- Usi1wGZZL0LveZjUN0t6OU1VZKSG8R5Ly7urjaSo1pPVIq5Rtt/H7W14Lecd+cUeKb4j
- oeusC9drN3AA8a4oykcVpt1wVqUnTbMGC9ARMCQP6eopcs1l7tzMseprW4RDNhIuz3CR
- gd0QBMPl6VDoFgBPB8vxtJw+3m0rqBYZCLZgCXekqlny8s2s92nJMuUABbJOEcDRarzi
- bDsSXsfJt1y+5n7yOURsC7lovMg4GF/vCl/0YMKjBO5bpv9EM5fToeKYyPGSKQoHOnCY
- ceb3cAVcv5UawcCic8XjhEhp4K7WPdYf2HVAC/qtxhbpjTxG4U5Q/SoppOJ60WqEkQvb
- Xs6n5Dvy7xmph6GWmU/bAv3eUK3pdD3xa2Ue1lHWz3U+rsYraI+AKYsMYx3RBlfAmCeC
- 1ve2BXPrqnOo7G8tnUvfdYPbK4Aakk0ds/AVqFHEZN+S6hRBmBjLaRFWZ3QSO1NjbBxW
- naHKZYT7nkrJm8AMCgZU0ZArFLpaufKCeiK5ECSsDxic4FIsY1OkWT42qEUfL0Wd+150
- AKGNZpPJnnP3QYY4W/MWcKH/zdO400+zWN52WevbSqZy90tqKDJrBkMl1ydqbuw1E4ZH
- vIs=
-''')[0]['secret']
-
- conf['_source_context'] = self.context
- conf['_start_mark'] = self.start_mark
-
- trusted_secret = configloader.SecretParser.fromYaml(layout, conf)
- layout.addSecret(trusted_secret)
-
- conf['name'] = 'untrusted-secret'
- conf['_source_context'] = self.untrusted_context
-
- untrusted_secret = configloader.SecretParser.fromYaml(layout, conf)
- layout.addSecret(untrusted_secret)
-
- base = configloader.JobParser.fromYaml(self.tenant, self.layout, {
- '_source_context': self.context,
- '_start_mark': self.start_mark,
- 'name': 'base',
- 'parent': None,
- 'timeout': 30,
- })
- layout.addJob(base)
-
- trusted_secrets_job = configloader.JobParser.fromYaml(
- tenant, layout, {
- '_source_context': self.context,
- '_start_mark': self.start_mark,
- 'name': 'trusted-secrets',
- 'parent': 'base',
- 'timeout': 40,
- 'secrets': [
- 'trusted-secret',
- ]
- })
- layout.addJob(trusted_secrets_job)
- untrusted_secrets_job = configloader.JobParser.fromYaml(
- tenant, layout, {
- '_source_context': self.untrusted_context,
- '_start_mark': self.start_mark,
- 'name': 'untrusted-secrets',
- 'parent': 'base',
- 'timeout': 40,
- 'secrets': [
- 'untrusted-secret',
- ]
- })
- layout.addJob(untrusted_secrets_job)
- trusted_secrets_trusted_child_job = configloader.JobParser.fromYaml(
- tenant, layout, {
- '_source_context': self.context,
- '_start_mark': self.start_mark,
- 'name': 'trusted-secrets-trusted-child',
- 'parent': 'trusted-secrets',
- })
- layout.addJob(trusted_secrets_trusted_child_job)
- trusted_secrets_untrusted_child_job = configloader.JobParser.fromYaml(
- tenant, layout, {
- '_source_context': self.untrusted_context,
- '_start_mark': self.start_mark,
- 'name': 'trusted-secrets-untrusted-child',
- 'parent': 'trusted-secrets',
- })
- layout.addJob(trusted_secrets_untrusted_child_job)
- untrusted_secrets_trusted_child_job = configloader.JobParser.fromYaml(
- tenant, layout, {
- '_source_context': self.context,
- '_start_mark': self.start_mark,
- 'name': 'untrusted-secrets-trusted-child',
- 'parent': 'untrusted-secrets',
- })
- layout.addJob(untrusted_secrets_trusted_child_job)
- untrusted_secrets_untrusted_child_job = \
- configloader.JobParser.fromYaml(
- tenant, layout, {
- '_source_context': self.untrusted_context,
- '_start_mark': self.start_mark,
- 'name': 'untrusted-secrets-untrusted-child',
- 'parent': 'untrusted-secrets',
- })
- layout.addJob(untrusted_secrets_untrusted_child_job)
-
- self.assertIsNone(trusted_secrets_job.post_review)
- self.assertTrue(untrusted_secrets_job.post_review)
- self.assertIsNone(
- trusted_secrets_trusted_child_job.post_review)
- self.assertIsNone(
- trusted_secrets_untrusted_child_job.post_review)
- self.assertTrue(
- untrusted_secrets_trusted_child_job.post_review)
- self.assertTrue(
- untrusted_secrets_untrusted_child_job.post_review)
-
- self.assertEqual(trusted_secrets_job.implied_run[0].secrets[0].name,
- 'trusted-secret')
- self.assertEqual(trusted_secrets_job.implied_run[0].secrets[0].
- secret_data['longpassword'],
- 'test-passwordtest-password')
- self.assertEqual(trusted_secrets_job.implied_run[0].secrets[0].
- secret_data['password'],
- 'test-password')
- self.assertEqual(
- len(trusted_secrets_trusted_child_job.implied_run[0].secrets), 0)
- self.assertEqual(
- len(trusted_secrets_untrusted_child_job.implied_run[0].secrets), 0)
-
- self.assertEqual(untrusted_secrets_job.implied_run[0].secrets[0].name,
- 'untrusted-secret')
- self.assertEqual(
- len(untrusted_secrets_trusted_child_job.implied_run[0].secrets), 0)
- self.assertEqual(
- len(untrusted_secrets_untrusted_child_job.implied_run[0].secrets),
- 0)
-
def test_job_inheritance_job_tree(self):
tenant = model.Tenant('tenant')
layout = model.Layout(tenant)
@@ -554,7 +194,8 @@
'name': 'project',
'gate': {
'jobs': [
- {'python27': {'timeout': 70}}
+ {'python27': {'timeout': 70,
+ 'run': 'playbooks/python27.yaml'}}
]
}
}])
diff --git a/tests/unit/test_scheduler.py b/tests/unit/test_scheduler.py
index 6efc43f..53a20ff 100755
--- a/tests/unit/test_scheduler.py
+++ b/tests/unit/test_scheduler.py
@@ -47,7 +47,6 @@
def test_jobs_executed(self):
"Test that jobs are executed and a change is merged"
-
A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A')
A.addApproval('Code-Review', 2)
self.fake_gerrit.addEvent(A.addApproval('Approved', 1))
@@ -127,15 +126,20 @@
self.assertReportedStat('zuul.nodepool.requested.size.1', value='1|c')
self.assertReportedStat('zuul.nodepool.fulfilled.size.1', value='1|c')
self.assertReportedStat('zuul.nodepool.current_requests', value='1|g')
+ self.assertReportedStat('zuul.executors.online', value='1|g')
+ self.assertReportedStat('zuul.executors.accepting', value='1|g')
+ self.assertReportedStat('zuul.mergers.online', value='1|g')
for build in self.history:
self.assertTrue(build.parameters['zuul']['voting'])
def test_initial_pipeline_gauges(self):
"Test that each pipeline reported its length on start"
- self.assertReportedStat('zuul.pipeline.gate.current_changes',
+ self.assertReportedStat('zuul.tenant.tenant-one.pipeline.gate.'
+ 'current_changes',
value='0|g')
- self.assertReportedStat('zuul.pipeline.check.current_changes',
+ self.assertReportedStat('zuul.tenant.tenant-one.pipeline.check.'
+ 'current_changes',
value='0|g')
def test_job_branch(self):
@@ -1461,8 +1465,6 @@
@simple_layout('layouts/autohold.yaml')
def test_autohold(self):
- A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A')
-
client = zuul.rpcclient.RPCClient('127.0.0.1',
self.gearman_server.port)
self.addCleanup(client.shutdown)
@@ -1470,15 +1472,36 @@
"reason text", 1)
self.assertTrue(r)
- self.executor_server.failJob('project-test2', A)
+ # First check that successful jobs do not autohold
+ A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A')
self.fake_gerrit.addEvent(A.getPatchsetCreatedEvent(1))
self.waitUntilSettled()
self.assertEqual(A.data['status'], 'NEW')
self.assertEqual(A.reported, 1)
- self.assertEqual(self.getJobFromHistory('project-test2').result,
- 'FAILURE')
+ # project-test2
+ self.assertEqual(self.history[0].result, 'SUCCESS')
+
+ # Check nodepool for a held node
+ held_node = None
+ for node in self.fake_nodepool.getNodes():
+ if node['state'] == zuul.model.STATE_HOLD:
+ held_node = node
+ break
+ self.assertIsNone(held_node)
+
+ # Now test that failed jobs are autoheld
+ B = self.fake_gerrit.addFakeChange('org/project', 'master', 'B')
+ self.executor_server.failJob('project-test2', B)
+ self.fake_gerrit.addEvent(B.getPatchsetCreatedEvent(1))
+
+ self.waitUntilSettled()
+
+ self.assertEqual(B.data['status'], 'NEW')
+ self.assertEqual(B.reported, 1)
+ # project-test2
+ self.assertEqual(self.history[1].result, 'FAILURE')
# Check nodepool for a held node
held_node = None
@@ -1498,14 +1521,14 @@
self.assertEqual(held_node['comment'], "reason text")
# Another failed change should not hold any more nodes
- B = self.fake_gerrit.addFakeChange('org/project', 'master', 'B')
- self.executor_server.failJob('project-test2', B)
- self.fake_gerrit.addEvent(B.getPatchsetCreatedEvent(1))
+ C = self.fake_gerrit.addFakeChange('org/project', 'master', 'C')
+ self.executor_server.failJob('project-test2', C)
+ self.fake_gerrit.addEvent(C.getPatchsetCreatedEvent(1))
self.waitUntilSettled()
- self.assertEqual(B.data['status'], 'NEW')
- self.assertEqual(B.reported, 1)
- self.assertEqual(self.getJobFromHistory('project-test2').result,
- 'FAILURE')
+ self.assertEqual(C.data['status'], 'NEW')
+ self.assertEqual(C.reported, 1)
+ # project-test2
+ self.assertEqual(self.history[2].result, 'FAILURE')
held_nodes = 0
for node in self.fake_nodepool.getNodes():
@@ -1514,6 +1537,49 @@
self.assertEqual(held_nodes, 1)
@simple_layout('layouts/autohold.yaml')
+ def test_autohold_ignores_aborted_jobs(self):
+ client = zuul.rpcclient.RPCClient('127.0.0.1',
+ self.gearman_server.port)
+ self.addCleanup(client.shutdown)
+ r = client.autohold('tenant-one', 'org/project', 'project-test2',
+ "reason text", 1)
+ self.assertTrue(r)
+
+ self.executor_server.hold_jobs_in_build = True
+
+ # Create a change that will have its job aborted
+ A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A')
+ self.fake_gerrit.addEvent(A.getPatchsetCreatedEvent(1))
+ self.waitUntilSettled()
+
+ # Creating new patchset on change A will abort A,1's job because
+ # a new patchset arrived replacing A,1 with A,2.
+ A.addPatchset()
+ self.fake_gerrit.addEvent(A.getPatchsetCreatedEvent(2))
+
+ self.waitUntilSettled()
+ self.executor_server.hold_jobs_in_build = False
+ self.executor_server.release()
+ self.waitUntilSettled()
+
+ self.assertEqual(A.data['status'], 'NEW')
+ # Note only the successful job for A,2 will report as we don't
+ # report aborted builds for old patchsets.
+ self.assertEqual(A.reported, 1)
+ # A,1 project-test2
+ self.assertEqual(self.history[0].result, 'ABORTED')
+ # A,2 project-test2
+ self.assertEqual(self.history[1].result, 'SUCCESS')
+
+ # Check nodepool for a held node
+ held_node = None
+ for node in self.fake_nodepool.getNodes():
+ if node['state'] == zuul.model.STATE_HOLD:
+ held_node = node
+ break
+ self.assertIsNone(held_node)
+
+ @simple_layout('layouts/autohold.yaml')
def test_autohold_list(self):
client = zuul.rpcclient.RPCClient('127.0.0.1',
self.gearman_server.port)
@@ -2364,6 +2430,71 @@
self.assertEqual(rp, set(['org/project', 'org/project0',
'org/project3']))
+ @simple_layout('layouts/job-variants.yaml')
+ def test_job_branch_variants(self):
+ self.create_branch('org/project', 'stable/diablo')
+ self.create_branch('org/project', 'stable/essex')
+ A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A')
+ self.fake_gerrit.addEvent(A.getPatchsetCreatedEvent(1))
+ self.waitUntilSettled()
+
+ B = self.fake_gerrit.addFakeChange('org/project', 'stable/diablo', 'B')
+ self.fake_gerrit.addEvent(B.getPatchsetCreatedEvent(1))
+ self.waitUntilSettled()
+
+ C = self.fake_gerrit.addFakeChange('org/project', 'stable/essex', 'C')
+ self.fake_gerrit.addEvent(C.getPatchsetCreatedEvent(1))
+ self.waitUntilSettled()
+ self.assertHistory([
+ dict(name='python27', result='SUCCESS'),
+ dict(name='python27', result='SUCCESS'),
+ dict(name='python27', result='SUCCESS'),
+ ])
+
+ p = self.history[0].parameters
+ self.assertEqual(p['timeout'], 40)
+ self.assertEqual(len(p['nodes']), 1)
+ self.assertEqual(p['nodes'][0]['label'], 'new')
+ self.assertEqual([x['path'] for x in p['pre_playbooks']],
+ ['base-pre', 'py27-pre'])
+ self.assertEqual([x['path'] for x in p['post_playbooks']],
+ ['py27-post-a', 'py27-post-b', 'base-post'])
+ self.assertEqual([x['path'] for x in p['playbooks']],
+ ['playbooks/python27.yaml'])
+
+ p = self.history[1].parameters
+ self.assertEqual(p['timeout'], 50)
+ self.assertEqual(len(p['nodes']), 1)
+ self.assertEqual(p['nodes'][0]['label'], 'old')
+ self.assertEqual([x['path'] for x in p['pre_playbooks']],
+ ['base-pre', 'py27-pre', 'py27-diablo-pre'])
+ self.assertEqual([x['path'] for x in p['post_playbooks']],
+ ['py27-diablo-post', 'py27-post-a', 'py27-post-b',
+ 'base-post'])
+ self.assertEqual([x['path'] for x in p['playbooks']],
+ ['py27-diablo'])
+
+ p = self.history[2].parameters
+ self.assertEqual(p['timeout'], 40)
+ self.assertEqual(len(p['nodes']), 1)
+ self.assertEqual(p['nodes'][0]['label'], 'new')
+ self.assertEqual([x['path'] for x in p['pre_playbooks']],
+ ['base-pre', 'py27-pre', 'py27-essex-pre'])
+ self.assertEqual([x['path'] for x in p['post_playbooks']],
+ ['py27-essex-post', 'py27-post-a', 'py27-post-b',
+ 'base-post'])
+ self.assertEqual([x['path'] for x in p['playbooks']],
+ ['playbooks/python27.yaml'])
+
+ @simple_layout("layouts/no-run.yaml")
+ def test_job_without_run(self):
+ "Test that a job without a run playbook errors"
+ A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A')
+ self.fake_gerrit.addEvent(A.getPatchsetCreatedEvent(1))
+ self.waitUntilSettled()
+ self.assertIn('Job base does not specify a run playbook',
+ A.messages[-1])
+
def test_queue_names(self):
"Test shared change queue names"
tenant = self.sched.abide.tenants.get('tenant-one')
@@ -2506,6 +2637,28 @@
self.assertIn('project-merge', status_jobs[1]['dependencies'])
self.assertIn('project-merge', status_jobs[2]['dependencies'])
+ def test_reconfigure_merge(self):
+ """Test that two reconfigure events are merged"""
+
+ tenant = self.sched.abide.tenants['tenant-one']
+ (trusted, project) = tenant.getProject('org/project')
+
+ self.sched.run_handler_lock.acquire()
+ self.assertEqual(self.sched.management_event_queue.qsize(), 0)
+
+ self.sched.reconfigureTenant(tenant, project)
+ self.assertEqual(self.sched.management_event_queue.qsize(), 1)
+
+ self.sched.reconfigureTenant(tenant, project)
+ # The second event should have been combined with the first
+ # so we should still only have one entry.
+ self.assertEqual(self.sched.management_event_queue.qsize(), 1)
+
+ self.sched.run_handler_lock.release()
+ self.waitUntilSettled()
+
+ self.assertEqual(self.sched.management_event_queue.qsize(), 0)
+
def test_live_reconfiguration(self):
"Test that live reconfiguration works"
self.executor_server.hold_jobs_in_build = True
@@ -2881,6 +3034,70 @@
self.assertEqual(len(tenant.layout.pipelines['check'].queues), 0)
self.assertIn('Build succeeded', A.messages[0])
+ @simple_layout("layouts/reconfigure-failed-head.yaml")
+ def test_live_reconfiguration_failed_change_at_head(self):
+ # Test that if we reconfigure with a failed change at head,
+ # that the change behind it isn't reparented onto it.
+
+ self.executor_server.hold_jobs_in_build = True
+ A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A')
+ A.addApproval('Code-Review', 2)
+ B = self.fake_gerrit.addFakeChange('org/project', 'master', 'B')
+ B.addApproval('Code-Review', 2)
+
+ self.executor_server.failJob('job1', A)
+
+ self.fake_gerrit.addEvent(A.addApproval('Approved', 1))
+ self.fake_gerrit.addEvent(B.addApproval('Approved', 1))
+
+ self.waitUntilSettled()
+
+ self.assertBuilds([
+ dict(name='job1', changes='1,1'),
+ dict(name='job2', changes='1,1'),
+ dict(name='job1', changes='1,1 2,1'),
+ dict(name='job2', changes='1,1 2,1'),
+ ])
+
+ self.release(self.builds[0])
+ self.waitUntilSettled()
+
+ self.assertBuilds([
+ dict(name='job2', changes='1,1'),
+ dict(name='job1', changes='2,1'),
+ dict(name='job2', changes='2,1'),
+ ])
+
+ # Unordered history comparison because the aborts can finish
+ # in any order.
+ self.assertHistory([
+ dict(name='job1', result='FAILURE', changes='1,1'),
+ dict(name='job1', result='ABORTED', changes='1,1 2,1'),
+ dict(name='job2', result='ABORTED', changes='1,1 2,1'),
+ ], ordered=False)
+
+ self.sched.reconfigure(self.config)
+ self.waitUntilSettled()
+
+ self.executor_server.hold_jobs_in_build = False
+ self.executor_server.release()
+ self.waitUntilSettled()
+
+ self.assertBuilds([])
+
+ self.assertHistory([
+ dict(name='job1', result='FAILURE', changes='1,1'),
+ dict(name='job1', result='ABORTED', changes='1,1 2,1'),
+ dict(name='job2', result='ABORTED', changes='1,1 2,1'),
+ dict(name='job2', result='SUCCESS', changes='1,1'),
+ dict(name='job1', result='SUCCESS', changes='2,1'),
+ dict(name='job2', result='SUCCESS', changes='2,1'),
+ ], ordered=False)
+ self.assertEqual(A.data['status'], 'NEW')
+ self.assertEqual(B.data['status'], 'MERGED')
+ self.assertEqual(A.reported, 2)
+ self.assertEqual(B.reported, 2)
+
def test_delayed_repo_init(self):
self.init_repo("org/new-project")
files = {'README': ''}
@@ -4824,6 +5041,39 @@
self.gearman_server.release()
self.waitUntilSettled()
+ @simple_layout('layouts/parent-matchers.yaml')
+ def test_parent_matchers(self):
+ "Test that if a job's parent does not match, the job does not run"
+ A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A')
+ self.fake_gerrit.addEvent(A.getPatchsetCreatedEvent(1))
+ self.waitUntilSettled()
+
+ self.assertHistory([])
+
+ files = {'foo.txt': ''}
+ B = self.fake_gerrit.addFakeChange('org/project', 'master', 'B',
+ files=files)
+ self.fake_gerrit.addEvent(B.getPatchsetCreatedEvent(1))
+ self.waitUntilSettled()
+
+ files = {'bar.txt': ''}
+ C = self.fake_gerrit.addFakeChange('org/project', 'master', 'C',
+ files=files)
+ self.fake_gerrit.addEvent(C.getPatchsetCreatedEvent(1))
+ self.waitUntilSettled()
+
+ files = {'foo.txt': '', 'bar.txt': ''}
+ D = self.fake_gerrit.addFakeChange('org/project', 'master', 'D',
+ files=files)
+ self.fake_gerrit.addEvent(D.getPatchsetCreatedEvent(1))
+ self.waitUntilSettled()
+
+ self.assertHistory([
+ dict(name='child-job', result='SUCCESS', changes='2,1'),
+ dict(name='child-job', result='SUCCESS', changes='3,1'),
+ dict(name='child-job', result='SUCCESS', changes='4,1'),
+ ])
+
class TestExecutor(ZuulTestCase):
tenant_config_file = 'config/single-tenant/main.yaml'
@@ -5702,6 +5952,7 @@
- job:
name: project-test2
+ run: playbooks/project-test2.yaml
semaphore: test-semaphore
- project:
diff --git a/tests/unit/test_v3.py b/tests/unit/test_v3.py
index b30d710..e2da808 100755
--- a/tests/unit/test_v3.py
+++ b/tests/unit/test_v3.py
@@ -128,6 +128,7 @@
- job:
name: project-test
parent: job-final
+ run: playbooks/project-test.yaml
- project:
name: org/project
@@ -153,7 +154,116 @@
# Thus it should fail.
self.assertEqual(A.reported, 1)
self.assertEqual(A.patchsets[-1]['approvals'][0]['value'], '-1')
- self.assertIn('Unable to inherit from final job', A.messages[0])
+ self.assertIn('Unable to modify final job', A.messages[0])
+
+
+class TestBranchTemplates(ZuulTestCase):
+ tenant_config_file = 'config/branch-templates/main.yaml'
+
+ def test_template_removal_from_branch(self):
+ # Test that a template can be removed from one branch but not
+ # another.
+ # This creates a new branch with a copy of the config in master
+ self.create_branch('puppet-integration', 'stable/newton')
+ self.create_branch('puppet-integration', 'stable/ocata')
+ self.create_branch('puppet-tripleo', 'stable/newton')
+ self.create_branch('puppet-tripleo', 'stable/ocata')
+ self.fake_gerrit.addEvent(
+ self.fake_gerrit.getFakeBranchCreatedEvent(
+ 'puppet-integration', 'stable/newton'))
+ self.fake_gerrit.addEvent(
+ self.fake_gerrit.getFakeBranchCreatedEvent(
+ 'puppet-integration', 'stable/ocata'))
+ self.fake_gerrit.addEvent(
+ self.fake_gerrit.getFakeBranchCreatedEvent(
+ 'puppet-tripleo', 'stable/newton'))
+ self.fake_gerrit.addEvent(
+ self.fake_gerrit.getFakeBranchCreatedEvent(
+ 'puppet-tripleo', 'stable/ocata'))
+ self.waitUntilSettled()
+
+ in_repo_conf = textwrap.dedent(
+ """
+ - project:
+ name: puppet-tripleo
+ check:
+ jobs:
+ - puppet-something
+ """)
+
+ file_dict = {'.zuul.yaml': in_repo_conf}
+ A = self.fake_gerrit.addFakeChange('puppet-tripleo', 'stable/newton',
+ 'A', files=file_dict)
+ self.fake_gerrit.addEvent(A.getPatchsetCreatedEvent(1))
+ self.waitUntilSettled()
+ self.assertHistory([
+ dict(name='puppet-something', result='SUCCESS', changes='1,1')])
+
+ def test_template_change_on_branch(self):
+ # Test that the contents of a template can be changed on one
+ # branch without affecting another.
+
+ # This creates a new branch with a copy of the config in master
+ self.create_branch('puppet-integration', 'stable/newton')
+ self.create_branch('puppet-integration', 'stable/ocata')
+ self.create_branch('puppet-tripleo', 'stable/newton')
+ self.create_branch('puppet-tripleo', 'stable/ocata')
+ self.fake_gerrit.addEvent(
+ self.fake_gerrit.getFakeBranchCreatedEvent(
+ 'puppet-integration', 'stable/newton'))
+ self.fake_gerrit.addEvent(
+ self.fake_gerrit.getFakeBranchCreatedEvent(
+ 'puppet-integration', 'stable/ocata'))
+ self.fake_gerrit.addEvent(
+ self.fake_gerrit.getFakeBranchCreatedEvent(
+ 'puppet-tripleo', 'stable/newton'))
+ self.fake_gerrit.addEvent(
+ self.fake_gerrit.getFakeBranchCreatedEvent(
+ 'puppet-tripleo', 'stable/ocata'))
+ self.waitUntilSettled()
+
+ in_repo_conf = textwrap.dedent("""
+ - job:
+ name: puppet-unit-base
+ run: playbooks/run-unit-tests.yaml
+
+ - job:
+ name: puppet-unit-3.8
+ parent: puppet-unit-base
+ branches: ^(stable/(newton|ocata)).*$
+ vars:
+ puppet_gem_version: 3.8
+
+ - job:
+ name: puppet-something
+ run: playbooks/run-unit-tests.yaml
+
+ - project-template:
+ name: puppet-unit
+ check:
+ jobs:
+ - puppet-something
+
+ - project:
+ name: puppet-integration
+ templates:
+ - puppet-unit
+ """)
+
+ file_dict = {'.zuul.yaml': in_repo_conf}
+ A = self.fake_gerrit.addFakeChange('puppet-integration',
+ 'stable/newton',
+ 'A', files=file_dict)
+ B = self.fake_gerrit.addFakeChange('puppet-tripleo',
+ 'stable/newton',
+ 'B')
+ B.data['commitMessage'] = '%s\n\nDepends-On: %s\n' % (
+ B.subject, A.data['id'])
+ self.fake_gerrit.addEvent(B.getPatchsetCreatedEvent(1))
+ self.waitUntilSettled()
+ self.assertHistory([
+ dict(name='puppet-something', result='SUCCESS',
+ changes='1,1 2,1')])
class TestBranchVariants(ZuulTestCase):
@@ -178,6 +288,165 @@
self.executor_server.release()
self.waitUntilSettled()
+ def test_branch_variants_reconfigure(self):
+ # Test branch variants of jobs with inheritance
+ self.executor_server.hold_jobs_in_build = True
+ # This creates a new branch with a copy of the config in master
+ self.create_branch('puppet-integration', 'stable')
+ self.fake_gerrit.addEvent(
+ self.fake_gerrit.getFakeBranchCreatedEvent(
+ 'puppet-integration', 'stable'))
+ self.waitUntilSettled()
+
+ with open(os.path.join(FIXTURE_DIR,
+ 'config/branch-variants/git/',
+ 'puppet-integration/.zuul.yaml')) as f:
+ config = f.read()
+
+ # Push a change that triggers a dynamic reconfiguration
+ file_dict = {'.zuul.yaml': config}
+ A = self.fake_gerrit.addFakeChange('puppet-integration', 'master', 'A',
+ files=file_dict)
+ self.fake_gerrit.addEvent(A.getPatchsetCreatedEvent(1))
+ self.waitUntilSettled()
+
+ ipath = self.builds[0].parameters['zuul']['_inheritance_path']
+ for i in ipath:
+ self.log.debug("inheritance path %s", i)
+ self.assertEqual(len(ipath), 5)
+ self.executor_server.hold_jobs_in_build = False
+ self.executor_server.release()
+ self.waitUntilSettled()
+
+ def test_branch_variants_divergent(self):
+ # Test branches can diverge and become independent
+ self.executor_server.hold_jobs_in_build = True
+ # This creates a new branch with a copy of the config in master
+ self.create_branch('puppet-integration', 'stable')
+ self.fake_gerrit.addEvent(
+ self.fake_gerrit.getFakeBranchCreatedEvent(
+ 'puppet-integration', 'stable'))
+ self.waitUntilSettled()
+
+ with open(os.path.join(FIXTURE_DIR,
+ 'config/branch-variants/git/',
+ 'puppet-integration/stable.zuul.yaml')) as f:
+ config = f.read()
+
+ file_dict = {'.zuul.yaml': config}
+ C = self.fake_gerrit.addFakeChange('puppet-integration', 'stable', 'C',
+ files=file_dict)
+ C.addApproval('Code-Review', 2)
+ self.fake_gerrit.addEvent(C.addApproval('Approved', 1))
+ self.waitUntilSettled()
+ self.fake_gerrit.addEvent(C.getChangeMergedEvent())
+ self.waitUntilSettled()
+
+ A = self.fake_gerrit.addFakeChange('puppet-integration', 'master', 'A')
+ self.fake_gerrit.addEvent(A.getPatchsetCreatedEvent(1))
+ self.waitUntilSettled()
+ B = self.fake_gerrit.addFakeChange('puppet-integration', 'stable', 'B')
+ self.fake_gerrit.addEvent(B.getPatchsetCreatedEvent(1))
+ self.waitUntilSettled()
+
+ self.assertEqual(self.builds[0].parameters['zuul']['jobtags'],
+ ['master'])
+
+ self.assertEqual(self.builds[1].parameters['zuul']['jobtags'],
+ ['stable'])
+
+ self.executor_server.hold_jobs_in_build = False
+ self.executor_server.release()
+ self.waitUntilSettled()
+
+
+class TestCentralJobs(ZuulTestCase):
+ tenant_config_file = 'config/central-jobs/main.yaml'
+
+ def setUp(self):
+ super(TestCentralJobs, self).setUp()
+ self.create_branch('org/project', 'stable')
+ self.fake_gerrit.addEvent(
+ self.fake_gerrit.getFakeBranchCreatedEvent(
+ 'org/project', 'stable'))
+ self.waitUntilSettled()
+
+ def _updateConfig(self, config, branch):
+ file_dict = {'.zuul.yaml': config}
+ C = self.fake_gerrit.addFakeChange('org/project', branch, 'C',
+ files=file_dict)
+ C.addApproval('Code-Review', 2)
+ self.fake_gerrit.addEvent(C.addApproval('Approved', 1))
+ self.waitUntilSettled()
+ self.fake_gerrit.addEvent(C.getChangeMergedEvent())
+ self.waitUntilSettled()
+
+ def _test_central_job_on_branch(self, branch, other_branch):
+ # Test that a job defined on a branchless repo only runs on
+ # the branch applied
+ config = textwrap.dedent(
+ """
+ - project:
+ name: org/project
+ check:
+ jobs:
+ - central-job
+ """)
+ self._updateConfig(config, branch)
+
+ A = self.fake_gerrit.addFakeChange('org/project', branch, 'A')
+ self.fake_gerrit.addEvent(A.getPatchsetCreatedEvent(1))
+ self.waitUntilSettled()
+
+ self.assertHistory([
+ dict(name='central-job', result='SUCCESS', changes='2,1')])
+
+ # No jobs should run for this change.
+ B = self.fake_gerrit.addFakeChange('org/project', other_branch, 'B')
+ self.fake_gerrit.addEvent(B.getPatchsetCreatedEvent(1))
+ self.waitUntilSettled()
+
+ self.assertHistory([
+ dict(name='central-job', result='SUCCESS', changes='2,1')])
+
+ def test_central_job_on_stable(self):
+ self._test_central_job_on_branch('master', 'stable')
+
+ def test_central_job_on_master(self):
+ self._test_central_job_on_branch('stable', 'master')
+
+ def _test_central_template_on_branch(self, branch, other_branch):
+ # Test that a project-template defined on a branchless repo
+ # only runs on the branch applied
+ config = textwrap.dedent(
+ """
+ - project:
+ name: org/project
+ templates: ['central-jobs']
+ """)
+ self._updateConfig(config, branch)
+
+ A = self.fake_gerrit.addFakeChange('org/project', branch, 'A')
+ self.fake_gerrit.addEvent(A.getPatchsetCreatedEvent(1))
+ self.waitUntilSettled()
+
+ self.assertHistory([
+ dict(name='central-job', result='SUCCESS', changes='2,1')])
+
+ # No jobs should run for this change.
+ B = self.fake_gerrit.addFakeChange('org/project', other_branch, 'B')
+ self.fake_gerrit.addEvent(B.getPatchsetCreatedEvent(1))
+ self.waitUntilSettled()
+
+ self.assertHistory([
+ dict(name='central-job', result='SUCCESS', changes='2,1')])
+
+ def test_central_template_on_stable(self):
+ self._test_central_template_on_branch('master', 'stable')
+
+ def test_central_template_on_master(self):
+ self._test_central_template_on_branch('stable', 'master')
+
class TestInRepoConfig(ZuulTestCase):
# A temporary class to hold new tests while others are disabled
@@ -239,6 +508,7 @@
- job:
name: project-test2
+ run: playbooks/project-test2.yaml
- project:
name: org/project
@@ -283,6 +553,37 @@
dict(name='project-test2', result='SUCCESS', changes='1,1'),
dict(name='project-test2', result='SUCCESS', changes='2,1')])
+ def test_dynamic_template(self):
+ # Tests that a project can't update a template in another
+ # project.
+ in_repo_conf = textwrap.dedent(
+ """
+ - job:
+ name: project-test1
+
+ - project-template:
+ name: common-config-template
+ check:
+ jobs:
+ - project-test1
+
+ - project:
+ name: org/project
+ templates: [common-config-template]
+ """)
+
+ file_dict = {'.zuul.yaml': in_repo_conf}
+ A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A',
+ files=file_dict)
+ self.fake_gerrit.addEvent(A.getPatchsetCreatedEvent(1))
+ self.waitUntilSettled()
+
+ self.assertEqual(A.patchsets[0]['approvals'][0]['value'], "-1")
+ self.assertIn('Project template common-config-template '
+ 'is already defined',
+ A.messages[0],
+ "A should have failed the check pipeline")
+
def test_dynamic_config_non_existing_job(self):
"""Test that requesting a non existent job fails"""
in_repo_conf = textwrap.dedent(
@@ -364,9 +665,11 @@
"""
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
- job:
name: project-test2
+ run: playbooks/project-test2.yaml
- project:
name: org/project
@@ -397,9 +700,11 @@
"""
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
- job:
name: project-test2
+ run: playbooks/project-test2.yaml
- project:
name: org/project
@@ -440,6 +745,7 @@
- job:
name: project-test2
+ run: playbooks/project-test2.yaml
- project:
name: org/project
@@ -512,6 +818,7 @@
- job:
name: project-test2
+ run: playbooks/project-test2.yaml
- project:
name: org/project
@@ -979,6 +1286,7 @@
"""
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
- project-template:
name: some-jobs
tenant-one-gate:
@@ -1037,6 +1345,7 @@
"""
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
- project:
name: org/project1
@@ -1091,6 +1400,7 @@
"""
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
- job:
name: project-test2
@@ -1207,9 +1517,11 @@
"""
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
- job:
name: project-test2
+ run: playbooks/project-test2.yaml
- project:
name: org/project
@@ -1253,6 +1565,7 @@
"""
- job:
name: project-test1
+ run: playbooks/project-test1.yaml
- project:
name: org/project
@@ -1386,13 +1699,14 @@
"""
- job:
name: %s
+ run: playbooks/%s.yaml
- project:
name: org/plugin-project
check:
jobs:
- %s
- """ % (job_name, job_name))
+ """ % (job_name, job_name, job_name))
file_dict = {'.zuul.yaml': conf}
A = self.fake_gerrit.addFakeChange('org/plugin-project', 'master', 'A',
@@ -1594,6 +1908,7 @@
- job:
name: project-test
parent: parent
+ run: playbooks/project-test.yaml
roles:
- zuul: org/project
@@ -1630,6 +1945,7 @@
"""
- job:
name: project-test
+ run: playbooks/project-test.yaml
roles:
- zuul: common-config
@@ -1719,14 +2035,13 @@
tenant_config_file = 'config/data-return/main.yaml'
def test_data_return(self):
- # This exercises a proposed change to a role being checked out
- # and used.
A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A')
self.fake_gerrit.addEvent(A.getPatchsetCreatedEvent(1))
self.waitUntilSettled()
self.assertHistory([
dict(name='data-return', result='SUCCESS', changes='1,1'),
dict(name='data-return-relative', result='SUCCESS', changes='1,1'),
+ dict(name='child', result='SUCCESS', changes='1,1'),
], ordered=False)
self.assertIn('- data-return http://example.com/test/log/url/',
A.messages[-1])
@@ -1812,6 +2127,57 @@
"B should not fail because of timeout limit")
+class TestPragma(ZuulTestCase):
+ tenant_config_file = 'config/pragma/main.yaml'
+
+ def test_no_pragma(self):
+ self.create_branch('org/project', 'stable')
+ with open(os.path.join(FIXTURE_DIR,
+ 'config/pragma/git/',
+ 'org_project/nopragma.yaml')) as f:
+ config = f.read()
+ file_dict = {'.zuul.yaml': config}
+ A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A',
+ files=file_dict)
+ A.addApproval('Code-Review', 2)
+ self.fake_gerrit.addEvent(A.addApproval('Approved', 1))
+ self.waitUntilSettled()
+ self.fake_gerrit.addEvent(A.getChangeMergedEvent())
+ self.waitUntilSettled()
+
+ # This is an untrusted repo with 2 branches, so it should have
+ # an implied branch matcher for the job.
+ tenant = self.sched.abide.tenants.get('tenant-one')
+ jobs = tenant.layout.getJobs('test-job')
+ self.assertEqual(len(jobs), 1)
+ for job in tenant.layout.getJobs('test-job'):
+ self.assertIsNotNone(job.branch_matcher)
+
+ def test_pragma(self):
+ self.create_branch('org/project', 'stable')
+ with open(os.path.join(FIXTURE_DIR,
+ 'config/pragma/git/',
+ 'org_project/pragma.yaml')) as f:
+ config = f.read()
+ file_dict = {'.zuul.yaml': config}
+ A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A',
+ files=file_dict)
+ A.addApproval('Code-Review', 2)
+ self.fake_gerrit.addEvent(A.addApproval('Approved', 1))
+ self.waitUntilSettled()
+ self.fake_gerrit.addEvent(A.getChangeMergedEvent())
+ self.waitUntilSettled()
+
+ # This is an untrusted repo with 2 branches, so it would
+ # normally have an implied branch matcher, but our pragma
+ # overrides it.
+ tenant = self.sched.abide.tenants.get('tenant-one')
+ jobs = tenant.layout.getJobs('test-job')
+ self.assertEqual(len(jobs), 1)
+ for job in tenant.layout.getJobs('test-job'):
+ self.assertIsNone(job.branch_matcher)
+
+
class TestBaseJobs(ZuulTestCase):
tenant_config_file = 'config/base-jobs/main.yaml'
@@ -1852,6 +2218,118 @@
self.assertHistory([])
+class TestSecretInheritance(ZuulTestCase):
+ tenant_config_file = 'config/secret-inheritance/main.yaml'
+
+ def _getSecrets(self, job, pbtype):
+ secrets = []
+ build = self.getJobFromHistory(job)
+ for pb in build.parameters[pbtype]:
+ secrets.append(pb['secrets'])
+ return secrets
+
+ def _checkTrustedSecrets(self):
+ secret = {'longpassword': 'test-passwordtest-password',
+ 'password': 'test-password',
+ 'username': 'test-username'}
+ self.assertEqual(
+ self._getSecrets('trusted-secrets', 'playbooks'),
+ [{'trusted-secret': secret}])
+ self.assertEqual(
+ self._getSecrets('trusted-secrets', 'pre_playbooks'), [])
+ self.assertEqual(
+ self._getSecrets('trusted-secrets', 'post_playbooks'), [])
+
+ self.assertEqual(
+ self._getSecrets('trusted-secrets-trusted-child',
+ 'playbooks'), [{}])
+ self.assertEqual(
+ self._getSecrets('trusted-secrets-trusted-child',
+ 'pre_playbooks'), [])
+ self.assertEqual(
+ self._getSecrets('trusted-secrets-trusted-child',
+ 'post_playbooks'), [])
+
+ self.assertEqual(
+ self._getSecrets('trusted-secrets-untrusted-child',
+ 'playbooks'), [{}])
+ self.assertEqual(
+ self._getSecrets('trusted-secrets-untrusted-child',
+ 'pre_playbooks'), [])
+ self.assertEqual(
+ self._getSecrets('trusted-secrets-untrusted-child',
+ 'post_playbooks'), [])
+
+ def _checkUntrustedSecrets(self):
+ secret = {'longpassword': 'test-passwordtest-password',
+ 'password': 'test-password',
+ 'username': 'test-username'}
+ self.assertEqual(
+ self._getSecrets('untrusted-secrets', 'playbooks'),
+ [{'untrusted-secret': secret}])
+ self.assertEqual(
+ self._getSecrets('untrusted-secrets', 'pre_playbooks'), [])
+ self.assertEqual(
+ self._getSecrets('untrusted-secrets', 'post_playbooks'), [])
+
+ self.assertEqual(
+ self._getSecrets('untrusted-secrets-trusted-child',
+ 'playbooks'), [{}])
+ self.assertEqual(
+ self._getSecrets('untrusted-secrets-trusted-child',
+ 'pre_playbooks'), [])
+ self.assertEqual(
+ self._getSecrets('untrusted-secrets-trusted-child',
+ 'post_playbooks'), [])
+
+ self.assertEqual(
+ self._getSecrets('untrusted-secrets-untrusted-child',
+ 'playbooks'), [{}])
+ self.assertEqual(
+ self._getSecrets('untrusted-secrets-untrusted-child',
+ 'pre_playbooks'), [])
+ self.assertEqual(
+ self._getSecrets('untrusted-secrets-untrusted-child',
+ 'post_playbooks'), [])
+
+ def test_trusted_secret_inheritance_check(self):
+ A = self.fake_gerrit.addFakeChange('common-config', 'master', 'A')
+ self.fake_gerrit.addEvent(A.getPatchsetCreatedEvent(1))
+ self.waitUntilSettled()
+ self.assertHistory([
+ dict(name='trusted-secrets', result='SUCCESS', changes='1,1'),
+ dict(name='trusted-secrets-trusted-child',
+ result='SUCCESS', changes='1,1'),
+ dict(name='trusted-secrets-untrusted-child',
+ result='SUCCESS', changes='1,1'),
+ ], ordered=False)
+
+ self._checkTrustedSecrets()
+
+ def test_untrusted_secret_inheritance_gate(self):
+ A = self.fake_gerrit.addFakeChange('common-config', 'master', 'A')
+ A.addApproval('Code-Review', 2)
+ self.fake_gerrit.addEvent(A.addApproval('Approved', 1))
+ self.waitUntilSettled()
+ self.assertHistory([
+ dict(name='untrusted-secrets', result='SUCCESS', changes='1,1'),
+ dict(name='untrusted-secrets-trusted-child',
+ result='SUCCESS', changes='1,1'),
+ dict(name='untrusted-secrets-untrusted-child',
+ result='SUCCESS', changes='1,1'),
+ ], ordered=False)
+
+ self._checkUntrustedSecrets()
+
+ def test_untrusted_secret_inheritance_check(self):
+ A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A')
+ self.fake_gerrit.addEvent(A.getPatchsetCreatedEvent(1))
+ self.waitUntilSettled()
+ # This configuration tries to run untrusted secrets in an
+ # non-post-review pipeline and should therefore run no jobs.
+ self.assertHistory([])
+
+
class TestSecretLeaks(AnsibleZuulTestCase):
tenant_config_file = 'config/secret-leaks/main.yaml'
diff --git a/tools/encrypt_secret.py b/tools/encrypt_secret.py
index df4f449..9b52846 100755
--- a/tools/encrypt_secret.py
+++ b/tools/encrypt_secret.py
@@ -86,7 +86,12 @@
if p.returncode != 0:
raise Exception("Return code %s from openssl" % p.returncode)
output = stdout.decode('utf-8')
- m = re.match(r'^Public-Key: \((\d+) bit\)$', output, re.MULTILINE)
+ openssl_version = subprocess.check_output(
+ ['openssl', 'version']).split()[1]
+ if openssl_version.startswith(b'0.'):
+ m = re.match(r'^Modulus \((\d+) bit\):$', output, re.MULTILINE)
+ else:
+ m = re.match(r'^Public-Key: \((\d+) bit\)$', output, re.MULTILINE)
nbits = int(m.group(1))
nbytes = int(nbits / 8)
max_bytes = nbytes - 42 # PKCS1-OAEP overhead
diff --git a/tools/run-migration.sh b/tools/run-migration.sh
deleted file mode 100755
index 618fc56..0000000
--- a/tools/run-migration.sh
+++ /dev/null
@@ -1,60 +0,0 @@
-#!/bin/bash
-# Copyright (c) 2017 Red Hat, Inc.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-# implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Stupid script I'm using to test migration script locally
-# Assumes project-config is adjacent to zuul and has the mapping file
-
-OPTS=$(getopt -o v --long final -n $0 -- "$@")
-if [ $? != 0 ] ; then
- echo "Failed parsing options." >&2
- exit 1
-fi
-eval set -- "$OPTS"
-set -ex
-
-FINAL=0
-VERBOSE=""
-
-while true; do
- case "$1" in
- --final)
- FINAL=1
- shift
- ;;
- -v)
- VERBOSE=-v
- shift
- ;;
- --)
- shift
- break
- ;;
- esac
-done
-
-BASE_DIR=$(cd $(dirname $0)/../..; pwd)
-cd $BASE_DIR/project-config
-if [[ $FINAL = 1 ]] ; then
- git reset --hard
-fi
-python3 $BASE_DIR/zuul/zuul/cmd/migrate.py --mapping=zuul/mapping.yaml \
- zuul/layout.yaml jenkins/jobs nodepool/nodepool.yaml . $VERBOSE
-if [[ $FINAL = 1 ]] ; then
- find ../openstack-zuul-jobs/playbooks/legacy -maxdepth 1 -mindepth 1 \
- -type d | xargs rm -rf
- mv zuul.d/zuul-legacy-* ../openstack-zuul-jobs/zuul.d/
- mv playbooks/legacy/* ../openstack-zuul-jobs/playbooks/legacy/
-fi
diff --git a/tools/trigger-job.py b/tools/trigger-job.py
deleted file mode 100755
index dd69f1b..0000000
--- a/tools/trigger-job.py
+++ /dev/null
@@ -1,78 +0,0 @@
-#!/usr/bin/env python
-# Copyright 2013 OpenStack Foundation
-#
-# Licensed under the Apache License, Version 2.0 (the "License"); you may
-# not use this file except in compliance with the License. You may obtain
-# a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
-# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
-# License for the specific language governing permissions and limitations
-# under the License.
-
-# This script can be used to manually trigger a job in the same way that
-# Zuul does. At the moment, it only supports the post set of Zuul
-# parameters.
-
-import argparse
-import time
-import json
-from uuid import uuid4
-
-import gear
-
-
-def main():
- c = gear.Client()
-
- parser = argparse.ArgumentParser(description='Trigger a Zuul job.')
- parser.add_argument('--job', dest='job', required=True,
- help='Job Name')
- parser.add_argument('--project', dest='project', required=True,
- help='Project name')
- parser.add_argument('--pipeline', dest='pipeline', default='release',
- help='Zuul pipeline')
- parser.add_argument('--refname', dest='refname',
- help='Ref name')
- parser.add_argument('--oldrev', dest='oldrev',
- default='0000000000000000000000000000000000000000',
- help='Old revision (SHA)')
- parser.add_argument('--newrev', dest='newrev',
- help='New revision (SHA)')
- parser.add_argument('--url', dest='url',
- default='http://zuul.openstack.org/p', help='Zuul URL')
- parser.add_argument('--logpath', dest='logpath', required=True,
- help='Path for log files.')
- args = parser.parse_args()
-
- data = {'ZUUL_PIPELINE': args.pipeline,
- 'ZUUL_PROJECT': args.project,
- 'ZUUL_UUID': str(uuid4().hex),
- 'ZUUL_REF': args.refname,
- 'ZUUL_REFNAME': args.refname,
- 'ZUUL_OLDREV': args.oldrev,
- 'ZUUL_NEWREV': args.newrev,
- 'ZUUL_SHORT_OLDREV': args.oldrev[:7],
- 'ZUUL_SHORT_NEWREV': args.newrev[:7],
- 'ZUUL_COMMIT': args.newrev,
- 'ZUUL_URL': args.url,
- 'LOG_PATH': args.logpath,
- }
-
- c.addServer('127.0.0.1', 4730)
- c.waitForServer()
-
- job = gear.Job("build:%s" % args.job,
- json.dumps(data),
- unique=data['ZUUL_UUID'])
- c.submitJob(job, precedence=gear.PRECEDENCE_HIGH)
-
- while not job.complete:
- time.sleep(1)
-
-
-if __name__ == '__main__':
- main()
diff --git a/tox.ini b/tox.ini
index 7e84677..28d6000 100644
--- a/tox.ini
+++ b/tox.ini
@@ -52,6 +52,6 @@
[flake8]
# These are ignored intentionally in openstack-infra projects;
# please don't submit patches that solely correct them or enable them.
-ignore = E125,E129,E402,H,W503
+ignore = E125,E129,E402,E741,H,W503
show-source = True
exclude = .venv,.tox,dist,doc,build,*.egg
diff --git a/zuul/ansible/callback/zuul_stream.py b/zuul/ansible/callback/zuul_stream.py
index 8ba3b86..8845e9b 100644
--- a/zuul/ansible/callback/zuul_stream.py
+++ b/zuul/ansible/callback/zuul_stream.py
@@ -128,22 +128,29 @@
continue
msg = "%s\n" % log_id
s.send(msg.encode("utf-8"))
- buff = s.recv(4096).decode("utf-8")
+ buff = s.recv(4096)
buffering = True
while buffering:
- if "\n" in buff:
- (line, buff) = buff.split("\n", 1)
- done = self._log_streamline(host, line)
+ if b'\n' in buff:
+ (line, buff) = buff.split(b'\n', 1)
+ # We can potentially get binary data here. In order to
+ # being able to handle that use the backslashreplace
+ # error handling method. This decodes unknown utf-8
+ # code points to escape sequences which exactly represent
+ # the correct data without throwing a decoding exception.
+ done = self._log_streamline(
+ host, line.decode("utf-8", "backslashreplace"))
if done:
return
else:
- more = s.recv(4096).decode("utf-8")
+ more = s.recv(4096)
if not more:
buffering = False
else:
buff += more
if buff:
- self._log_streamline(host, line)
+ self._log_streamline(
+ host, line.decode("utf-8", "backslashreplace"))
def _log_streamline(self, host, line):
if "[Zuul] Task exit code" in line:
diff --git a/zuul/ansible/filter/zuul_filters.py b/zuul/ansible/filter/zuul_filters.py
index 4092c06..fa21f6b 100644
--- a/zuul/ansible/filter/zuul_filters.py
+++ b/zuul/ansible/filter/zuul_filters.py
@@ -14,10 +14,19 @@
def zuul_legacy_vars(zuul):
- # omitted:
- # ZUUL_URL
- # ZUUL_REF
+ # intentionally omitted:
+ # BASE_LOG_PATH
+ # JOB_TAGS
+ # LOG_PATH
# ZUUL_COMMIT
+ # ZUUL_REF
+ # ZUUL_URL
+ #
+ # newly added to all builds:
+ # ZUUL_SHORT_PROJECT_NAME
+ #
+ # existing in most builds but newly added for periodic:
+ # ZUUL_BRANCH
short_name = zuul['project']['name'].split('/')[-1]
params = dict(ZUUL_UUID=zuul['build'],
diff --git a/zuul/ansible/library/zuul_console.py b/zuul/ansible/library/zuul_console.py
index ddada3f..f84766d 100644
--- a/zuul/ansible/library/zuul_console.py
+++ b/zuul/ansible/library/zuul_console.py
@@ -179,7 +179,7 @@
if console is not None:
try:
console.file.close()
- except:
+ except Exception:
pass
while True:
console = self.chunkConsole(conn, log_uuid)
diff --git a/zuul/cmd/client.py b/zuul/cmd/client.py
index c9e399a..7a26a62 100755
--- a/zuul/cmd/client.py
+++ b/zuul/cmd/client.py
@@ -21,6 +21,7 @@
import prettytable
import sys
import time
+import textwrap
import zuul.rpcclient
@@ -78,8 +79,15 @@
required=True)
cmd_enqueue.set_defaults(func=self.enqueue)
- cmd_enqueue = subparsers.add_parser('enqueue-ref',
- help='enqueue a ref')
+ cmd_enqueue = subparsers.add_parser(
+ 'enqueue-ref', help='enqueue a ref',
+ formatter_class=argparse.RawDescriptionHelpFormatter,
+ description=textwrap.dedent('''\
+ Submit a trigger event
+
+ Directly enqueue a trigger event. This is usually used
+ to manually "replay" a trigger received from an external
+ source such as gerrit.'''))
cmd_enqueue.add_argument('--tenant', help='tenant name',
required=True)
cmd_enqueue.add_argument('--trigger', help='trigger name',
@@ -91,11 +99,9 @@
cmd_enqueue.add_argument('--ref', help='ref name',
required=True)
cmd_enqueue.add_argument(
- '--oldrev', help='old revision',
- default='0000000000000000000000000000000000000000')
+ '--oldrev', help='old revision', default=None)
cmd_enqueue.add_argument(
- '--newrev', help='new revision',
- default='0000000000000000000000000000000000000000')
+ '--newrev', help='new revision', default=None)
cmd_enqueue.set_defaults(func=self.enqueue_ref)
cmd_promote = subparsers.add_parser('promote',
@@ -132,8 +138,17 @@
parser.print_help()
sys.exit(1)
if self.args.func == self.enqueue_ref:
- if self.args.oldrev == self.args.newrev:
- parser.error("The old and new revisions must not be the same.")
+ # if oldrev or newrev is set, ensure they're not the same
+ if (self.args.oldrev is not None) or \
+ (self.args.newrev is not None):
+ if self.args.oldrev == self.args.newrev:
+ parser.error(
+ "The old and new revisions must not be the same.")
+ # if they're not set, we pad them out to zero
+ if self.args.oldrev is None:
+ self.args.oldrev = '0000000000000000000000000000000000000000'
+ if self.args.newrev is None:
+ self.args.newrev = '0000000000000000000000000000000000000000'
def setup_logging(self):
"""Client logging does not rely on conf file"""
diff --git a/zuul/cmd/executor.py b/zuul/cmd/executor.py
index 63c621d..979989d 100755
--- a/zuul/cmd/executor.py
+++ b/zuul/cmd/executor.py
@@ -22,6 +22,7 @@
# instead it depends on lockfile-0.9.1 which uses pidfile.
pid_file_module = extras.try_imports(['daemon.pidlockfile', 'daemon.pidfile'])
+import grp
import logging
import os
import pwd
@@ -101,9 +102,13 @@
if os.getuid() != 0:
return
pw = pwd.getpwnam(self.user)
- os.setgroups([])
+ # get a list of supplementary groups for the target user, and make sure
+ # we set them when dropping privileges.
+ groups = [g.gr_gid for g in grp.getgrall() if self.user in g.gr_mem]
+ os.setgroups(groups)
os.setgid(pw.pw_gid)
os.setuid(pw.pw_uid)
+ os.chdir(pw.pw_dir)
os.umask(0o022)
def main(self, daemon=True):
diff --git a/zuul/cmd/scheduler.py b/zuul/cmd/scheduler.py
index d920d8e..2d71f4d 100755
--- a/zuul/cmd/scheduler.py
+++ b/zuul/cmd/scheduler.py
@@ -141,7 +141,6 @@
import zuul.merger.client
import zuul.nodepool
import zuul.webapp
- import zuul.rpclistener
import zuul.zk
signal.signal(signal.SIGUSR2, zuul.cmd.stack_dump_handler)
@@ -174,7 +173,6 @@
webapp = zuul.webapp.WebApp(
self.sched, port=port, cache_expiry=cache_expiry,
listen_address=listen_address)
- rpc = zuul.rpclistener.RPCListener(self.config, self.sched)
self.configure_connections()
self.sched.setExecutor(gearman)
@@ -195,8 +193,6 @@
sys.exit(1)
self.log.info('Starting Webapp')
webapp.start()
- self.log.info('Starting RPC')
- rpc.start()
signal.signal(signal.SIGHUP, self.reconfigure_handler)
signal.signal(signal.SIGUSR1, self.exit_handler)
diff --git a/zuul/cmd/web.py b/zuul/cmd/web.py
index 9869a2c..9432656 100755
--- a/zuul/cmd/web.py
+++ b/zuul/cmd/web.py
@@ -55,6 +55,9 @@
'web', 'listen_address',
'127.0.0.1')
params['listen_port'] = get_default(self.config, 'web', 'port', 9000)
+ params['static_cache_expiry'] = get_default(self.config, 'web',
+ 'static_cache_expiry',
+ 3600)
params['gear_server'] = get_default(self.config, 'gearman', 'server')
params['gear_port'] = get_default(self.config, 'gearman', 'port', 4730)
params['ssl_key'] = get_default(self.config, 'gearman', 'ssl_key')
diff --git a/zuul/configloader.py b/zuul/configloader.py
index 426842b..99f10f6 100644
--- a/zuul/configloader.py
+++ b/zuul/configloader.py
@@ -238,7 +238,7 @@
class ZuulSafeLoader(yaml.SafeLoader):
zuul_node_types = frozenset(('job', 'nodeset', 'secret', 'pipeline',
'project', 'project-template',
- 'semaphore'))
+ 'semaphore', 'pragma'))
def __init__(self, stream, context):
wrapped_stream = io.StringIO(stream)
@@ -313,6 +313,30 @@
private_key).decode('utf8')
+class PragmaParser(object):
+ pragma = {
+ 'implied-branch-matchers': bool,
+ '_source_context': model.SourceContext,
+ '_start_mark': ZuulMark,
+ }
+
+ schema = vs.Schema(pragma)
+
+ def __init__(self):
+ self.log = logging.getLogger("zuul.PragmaParser")
+
+ def fromYaml(self, conf):
+ with configuration_exceptions('project-template', conf):
+ self.schema(conf)
+
+ bm = conf.get('implied-branch-matchers')
+ if bm is None:
+ return
+
+ source_context = conf['_source_context']
+ source_context.implied_branch_matchers = bm
+
+
class NodeSetParser(object):
@staticmethod
def getSchema(anonymous=False):
@@ -393,7 +417,8 @@
role = vs.Any(zuul_role, galaxy_role)
job_project = {vs.Required('name'): str,
- 'override-branch': str}
+ 'override-branch': str,
+ 'override-checkout': str}
secret = {vs.Required('name'): str,
vs.Required('secret'): str}
@@ -428,6 +453,7 @@
'dependencies': to_list(str),
'allowed-projects': to_list(str),
'override-branch': str,
+ 'override-checkout': str,
'description': str,
'post-review': bool}
@@ -450,32 +476,30 @@
'failure-url',
'success-url',
'override-branch',
+ 'override-checkout',
]
@staticmethod
- def _getImpliedBranches(reference, job, project_pipeline):
- # If the current job definition is not in the same branch as
- # the reference definition of this job, and this is a project
- # repo, add an implicit branch matcher for this branch
- # (assuming there are no explicit branch matchers). But only
- # for top-level job definitions and variants. Never for
- # project-templates. They, and in-project project-pipeline
- # job variants, should more closely attach to their branch if
- # they appear in a project-repo. That's handled in the
- # ProjectParser.
- if (reference and
- reference.source_context and
- reference.source_context.branch != job.source_context.branch):
- same_branch = False
- else:
- same_branch = True
-
- if (job.source_context and
- (not job.source_context.trusted) and
- (not project_pipeline) and
- (not same_branch)):
+ def _getImpliedBranches(tenant, job):
+ # If the user has set a pragma directive for this, use the
+ # value (if unset, the value is None).
+ if job.source_context.implied_branch_matchers is True:
return [job.source_context.branch]
- return None
+ elif job.source_context.implied_branch_matchers is False:
+ return None
+
+ # If this is a trusted project, don't create implied branch
+ # matchers.
+ if job.source_context.trusted:
+ return None
+
+ # If this project only has one branch, don't create implied
+ # branch matchers. This way central job repos can work.
+ branches = tenant.getProjectBranches(job.source_context.project)
+ if len(branches) == 1:
+ return None
+
+ return [job.source_context.branch]
@staticmethod
def fromYaml(tenant, layout, conf, project_pipeline=False,
@@ -492,34 +516,22 @@
# them (e.g., "job.run = ..." rather than
# "job.run.append(...)").
- reference = layout.jobs.get(name, [None])[0]
-
job = model.Job(name)
job.source_context = conf.get('_source_context')
job.source_line = conf.get('_start_mark').line + 1
- is_variant = layout.hasJob(name)
- if not is_variant:
- if 'parent' in conf:
- if conf['parent'] is not None:
- # Parent job is explicitly specified, so inherit from it.
- parent = layout.getJob(conf['parent'])
- job.inheritFrom(parent)
- else:
- # Parent is explicitly set as None, so user intends
- # this to be a base job. That's only okay if we're in
- # a config project.
- if not conf['_source_context'].trusted:
- raise Exception(
- "Base jobs must be defined in config projects")
+ if 'parent' in conf:
+ if conf['parent'] is not None:
+ # Parent job is explicitly specified, so inherit from it.
+ job.parent = conf['parent']
else:
- parent = layout.getJob(tenant.default_base_job)
- job.inheritFrom(parent)
- else:
- if 'parent' in conf:
- # TODO(jeblair): warn the user that we're ignoring the
- # parent setting on this variant job definition.
- pass
+ # Parent is explicitly set as None, so user intends
+ # this to be a base job. That's only okay if we're in
+ # a config project.
+ if not conf['_source_context'].trusted:
+ raise Exception(
+ "Base jobs must be defined in config projects")
+ job.parent = job.BASE_JOB_MARKER
# Secrets are part of the playbook context so we must establish
# them earlier than playbooks.
@@ -598,12 +610,6 @@
run = model.PlaybookContext(job.source_context, conf['run'],
job.roles, secrets)
job.run = (run,)
- else:
- if not project_pipeline:
- run_name = os.path.join('playbooks', job.name)
- run = model.PlaybookContext(job.source_context, run_name,
- job.roles, secrets)
- job.implied_run = (run,) + job.implied_run
for k in JobParser.simple_attributes:
a = k.replace('-', '_')
@@ -630,23 +636,24 @@
if isinstance(project, dict):
project_name = project['name']
project_override_branch = project.get('override-branch')
+ project_override_checkout = project.get(
+ 'override-checkout')
else:
project_name = project
project_override_branch = None
+ project_override_checkout = None
(trusted, project) = tenant.getProject(project_name)
if project is None:
raise Exception("Unknown project %s" % (project_name,))
job_project = model.JobProject(project_name,
- project_override_branch)
+ project_override_branch,
+ project_override_checkout)
new_projects[project_name] = job_project
- job.updateProjects(new_projects)
+ job.required_projects = new_projects
tags = conf.get('tags')
if tags:
- # Tags are merged via a union rather than a
- # destructive copy because they are intended to
- # accumulate onto any previously applied tags.
- job.tags = job.tags.union(set(tags))
+ job.tags = set(tags)
job.dependencies = frozenset(as_list(conf.get('dependencies')))
@@ -654,7 +661,7 @@
if variables:
if 'zuul' in variables:
raise Exception("Variables named 'zuul' are not allowed.")
- job.updateVariables(variables)
+ job.variables = variables
allowed_projects = conf.get('allowed-projects', None)
if allowed_projects:
@@ -666,18 +673,9 @@
allowed.append(project.name)
job.allowed_projects = frozenset(allowed)
- # If the current job definition is not in the same branch as
- # the reference definition of this job, and this is a project
- # repo, add an implicit branch matcher for this branch
- # (assuming there are no explicit branch matchers). But only
- # for top-level job definitions and variants.
- # Project-pipeline job variants should more closely attach to
- # their branch if they appear in a project-repo.
-
branches = None
- if (project_pipeline or 'branches' not in conf):
- branches = JobParser._getImpliedBranches(
- reference, job, project_pipeline)
+ if ('branches' not in conf):
+ branches = JobParser._getImpliedBranches(tenant, job)
if (not branches) and ('branches' in conf):
branches = as_list(conf['branches'])
if branches:
@@ -748,8 +746,8 @@
if validate:
with configuration_exceptions('project-template', conf):
self.schema(conf)
- project_template = model.ProjectConfig(conf['name'])
source_context = conf['_source_context']
+ project_template = model.ProjectConfig(conf['name'], source_context)
start_mark = conf['_start_mark']
for pipeline in self.layout.pipelines.values():
conf_pipeline = conf.get(pipeline.name)
@@ -1127,7 +1125,7 @@
@staticmethod
def fromYaml(base, project_key_dir, connections, scheduler, merger, conf,
- cached):
+ old_tenant):
TenantParser.getSchema(connections)(conf)
tenant = model.Tenant(conf['name'])
if conf.get('max-nodes-per-job') is not None:
@@ -1151,8 +1149,13 @@
tenant.addUntrustedProject(tpc)
for tpc in config_tpcs + untrusted_tpcs:
+ TenantParser._getProjectBranches(tenant, tpc, old_tenant)
TenantParser._resolveShadowProjects(tenant, tpc)
+ if old_tenant:
+ cached = True
+ else:
+ cached = False
tenant.config_projects_config, tenant.untrusted_projects_config = \
TenantParser._loadTenantInRepoLayouts(merger, connections,
tenant.config_projects,
@@ -1174,6 +1177,22 @@
tpc.shadow_projects = frozenset(shadow_projects)
@staticmethod
+ def _getProjectBranches(tenant, tpc, old_tenant):
+ # If we're performing a tenant reconfiguration, we will have
+ # an old_tenant object, however, we may be doing so because of
+ # a branch creation event, so if we don't have any cached
+ # data, query the branches again as well.
+ if old_tenant and tpc.project.unparsed_config:
+ branches = old_tenant.getProjectBranches(tpc.project)[:]
+ else:
+ branches = sorted(tpc.project.source.getProjectBranches(
+ tpc.project, tenant))
+ if 'master' in branches:
+ branches.remove('master')
+ branches = ['master'] + branches
+ tpc.branches = branches
+
+ @staticmethod
def _loadProjectKeys(project_key_dir, connection_name, project):
project.private_key_file = (
os.path.join(project_key_dir, connection_name,
@@ -1272,8 +1291,8 @@
exclude = set(as_list(conf['exclude']))
current_include = current_include - exclude
for project in conf['projects']:
- sub_projects = TenantParser._getProjects(source, project,
- current_include)
+ sub_projects = TenantParser._getProjects(
+ source, project, current_include)
projects.extend(sub_projects)
elif len(conf.keys()) == 1:
# A project with overrides
@@ -1374,11 +1393,7 @@
# branch. Remember the branch and then implicitly add a
# branch selector to each job there. This makes the
# in-repo configuration apply only to that branch.
- branches = sorted(project.source.getProjectBranches(
- project, tenant))
- if 'master' in branches:
- branches.remove('master')
- branches = ['master'] + branches
+ branches = tenant.getProjectBranches(project)
for branch in branches:
new_project_unparsed_branch_config[project][branch] = \
model.UnparsedTenantConfig()
@@ -1484,6 +1499,12 @@
@staticmethod
def _parseLayoutItems(layout, tenant, data, scheduler, connections,
skip_pipelines=False, skip_semaphores=False):
+ # Handle pragma items first since they modify the source context
+ # used by other classes.
+ pragma_parser = PragmaParser()
+ for config_pragma in data.pragmas:
+ pragma_parser.fromYaml(config_pragma)
+
if not skip_pipelines:
for config_pipeline in data.pipelines:
classes = TenantParser._getLoadClasses(
@@ -1520,6 +1541,16 @@
"Skipped adding job %s which shadows an existing job" %
(job,))
+ # Now that all the jobs are loaded, verify their parents exist
+ for config_job in data.jobs:
+ classes = TenantParser._getLoadClasses(tenant, config_job)
+ if 'job' not in classes:
+ continue
+ with configuration_exceptions('job', config_job):
+ parent = config_job.get('parent')
+ if parent:
+ layout.getJob(parent)
+
if not skip_semaphores:
for config_semaphore in data.semaphores:
classes = TenantParser._getLoadClasses(
@@ -1543,8 +1574,9 @@
classes = TenantParser._getLoadClasses(tenant, config_template)
if 'project-template' not in classes:
continue
- layout.addProjectTemplate(project_template_parser.fromYaml(
- config_template))
+ with configuration_exceptions('project-template', config_template):
+ layout.addProjectTemplate(project_template_parser.fromYaml(
+ config_template))
project_parser = ProjectParser(tenant, layout, project_template_parser)
for config_projects in data.projects.values():
@@ -1610,7 +1642,7 @@
# When performing a full reload, do not use cached data.
tenant = TenantParser.fromYaml(
base, project_key_dir, connections, scheduler, merger,
- conf_tenant, cached=False)
+ conf_tenant, old_tenant=None)
abide.tenants[tenant.name] = tenant
return abide
@@ -1625,7 +1657,7 @@
# When reloading a tenant only, use cached data if available.
new_tenant = TenantParser.fromYaml(
base, project_key_dir, connections, scheduler, merger,
- tenant.unparsed_config, cached=True)
+ tenant.unparsed_config, old_tenant=tenant)
new_abide.tenants[tenant.name] = new_tenant
return new_abide
@@ -1635,7 +1667,10 @@
else:
# Use the cached branch list; since this is a dynamic
# reconfiguration there should not be any branch changes.
- branches = project.unparsed_branch_config.keys()
+ branches = sorted(project.unparsed_branch_config.keys())
+ if 'master' in branches:
+ branches.remove('master')
+ branches = ['master'] + branches
for branch in branches:
fns1 = []
diff --git a/zuul/connection/__init__.py b/zuul/connection/__init__.py
index b44fa46..483495d 100644
--- a/zuul/connection/__init__.py
+++ b/zuul/connection/__init__.py
@@ -56,7 +56,7 @@
driver=self.driver.name,
connection=self.connection_name,
event=event.type))
- except:
+ except Exception:
self.log.exception("Exception reporting event stats")
def onLoad(self):
diff --git a/zuul/driver/gerrit/gerritconnection.py b/zuul/driver/gerrit/gerritconnection.py
index 83871e3..f4b090d 100644
--- a/zuul/driver/gerrit/gerritconnection.py
+++ b/zuul/driver/gerrit/gerritconnection.py
@@ -125,7 +125,9 @@
# This checks whether the event created or deleted a branch so
# that Zuul may know to perform a reconfiguration on the
# project.
- if event.type == 'ref-updated' and not event.ref.startswith('refs/'):
+ if (event.type == 'ref-updated' and
+ ((not event.ref.startswith('refs/')) or
+ event.ref.startswith('refs/heads'))):
if event.oldrev == '0' * 40:
event.branch_created = True
event.branch = event.ref
@@ -175,7 +177,7 @@
return
try:
self._handleEvent()
- except:
+ except Exception:
self.log.exception("Exception moving Gerrit event:")
finally:
self.connection.eventDone()
@@ -250,7 +252,7 @@
if ret and ret not in [-1, 130]:
raise Exception("Gerrit error executing stream-events")
- except:
+ except Exception:
self.log.exception("Exception on ssh event stream:")
time.sleep(5)
finally:
@@ -299,6 +301,12 @@
self.baseurl = self.connection_config.get('baseurl',
'https://%s' % self.server)
+ default_gitweb_url_template = '{baseurl}/gitweb?' \
+ 'p={project.name}.git;' \
+ 'a=commitdiff;h={sha}'
+ url_template = self.connection_config.get('gitweb_url_template',
+ default_gitweb_url_template)
+ self.gitweb_url_template = url_template
self._change_cache = {}
self.projects = {}
@@ -338,7 +346,7 @@
change.ref = event.ref
change.oldrev = event.oldrev
change.newrev = event.newrev
- change.url = self._getGitwebUrl(project, sha=event.newrev)
+ change.url = self._getWebUrl(project, sha=event.newrev)
elif event.ref and not event.ref.startswith('refs/'):
# Pre 2.13 Gerrit ref-updated events don't have branch prefixes.
project = self.source.getProject(event.project_name)
@@ -347,7 +355,7 @@
change.ref = 'refs/heads/' + event.ref
change.oldrev = event.oldrev
change.newrev = event.newrev
- change.url = self._getGitwebUrl(project, sha=event.newrev)
+ change.url = self._getWebUrl(project, sha=event.newrev)
elif event.ref and event.ref.startswith('refs/heads/'):
# From the timer trigger or Post 2.13 Gerrit
project = self.source.getProject(event.project_name)
@@ -356,7 +364,7 @@
change.branch = event.ref[len('refs/heads/'):]
change.oldrev = event.oldrev
change.newrev = event.newrev
- change.url = self._getGitwebUrl(project, sha=event.newrev)
+ change.url = self._getWebUrl(project, sha=event.newrev)
elif event.ref:
# catch-all ref (ie, not a branch or head)
project = self.source.getProject(event.project_name)
@@ -364,7 +372,7 @@
change.ref = event.ref
change.oldrev = event.oldrev
change.newrev = event.newrev
- change.url = self._getGitwebUrl(project, sha=event.newrev)
+ change.url = self._getWebUrl(project, sha=event.newrev)
else:
self.log.warning("Unable to get change for %s" % (event,))
change = None
@@ -597,7 +605,7 @@
refs = {} # type: Dict[str, str]
try:
refs = self.getInfoRefs(project)
- except:
+ except Exception:
self.log.exception("Exception looking for ref %s" %
ref)
sha = refs.get(ref, '')
@@ -633,7 +641,7 @@
else:
# CLOSED, RULE_ERROR
return False
- except:
+ except Exception:
self.log.exception("Exception determining whether change"
"%s can merge:" % change)
return False
@@ -787,7 +795,7 @@
try:
self.log.debug("SSH command:\n%s" % command)
stdin, stdout, stderr = self.client.exec_command(command)
- except:
+ except Exception:
self._open()
stdin, stdout, stderr = self.client.exec_command(command)
@@ -812,7 +820,7 @@
def getInfoRefs(self, project: Project) -> Dict[str, str]:
try:
data = self._uploadPack(project)
- except:
+ except Exception:
self.log.error("Cannot get references from %s" % project)
raise # keeps error information
ret = {}
@@ -848,11 +856,11 @@
project.name)
return url
- def _getGitwebUrl(self, project: Project, sha: str=None) -> str:
- url = '%s/gitweb?p=%s.git' % (self.baseurl, project.name)
- if sha:
- url += ';a=commitdiff;h=' + sha
- return url
+ def _getWebUrl(self, project: Project, sha: str=None) -> str:
+ return self.gitweb_url_template.format(
+ baseurl=self.baseurl,
+ project=project.getSafeAttributes(),
+ sha=sha)
def onLoad(self):
self.log.debug("Starting Gerrit Connection/Watchers")
diff --git a/zuul/driver/git/gitconnection.py b/zuul/driver/git/gitconnection.py
index 0624088..f93824d 100644
--- a/zuul/driver/git/gitconnection.py
+++ b/zuul/driver/git/gitconnection.py
@@ -51,7 +51,7 @@
def getProjectBranches(self, project, tenant):
# TODO(jeblair): implement; this will need to handle local or
# remote git urls.
- raise NotImplemented()
+ return ['master']
def getGitUrl(self, project):
url = '%s/%s' % (self.baseurl, project.name)
diff --git a/zuul/driver/github/githubconnection.py b/zuul/driver/github/githubconnection.py
index 3d0eb37..f987f47 100644
--- a/zuul/driver/github/githubconnection.py
+++ b/zuul/driver/github/githubconnection.py
@@ -40,6 +40,8 @@
ACCESS_TOKEN_URL = 'https://api.github.com/installations/%s/access_tokens'
PREVIEW_JSON_ACCEPT = 'application/vnd.github.machine-man-preview+json'
+INSTALLATIONS_URL = 'https://api.github.com/app/installations'
+REPOS_URL = 'https://api.github.com/installation/repositories'
def _sign_request(body, secret):
@@ -86,7 +88,7 @@
try:
self.__dispatch_event(request)
- except:
+ except Exception:
self.log.exception("Exception handling Github event:")
def __dispatch_event(self, request):
@@ -101,7 +103,7 @@
try:
json_body = request.json_body
self.connection.addEvent(json_body, event)
- except:
+ except Exception:
message = 'Exception deserializing JSON body'
self.log.exception(message)
raise webob.exc.HTTPBadRequest(message)
@@ -135,6 +137,7 @@
"""Move events from GitHub into the scheduler"""
log = logging.getLogger("zuul.GithubEventConnector")
+ delay = 10.0
def __init__(self, connection):
super(GithubEventConnector, self).__init__()
@@ -147,9 +150,17 @@
self.connection.addEvent(None)
def _handleEvent(self):
- json_body, event_type = self.connection.getEvent()
+ ts, json_body, event_type = self.connection.getEvent()
if self._stopped:
return
+ # Github can produce inconsistent data immediately after an
+ # event, So ensure that we do not deliver the event to Zuul
+ # until at least a certain amount of time has passed. Note
+ # that if we receive several events in succession, we will
+ # only need to delay for the first event. In essence, Zuul
+ # should always be a constant number of seconds behind Github.
+ now = time.time()
+ time.sleep(max((ts + self.delay) - now, 0.0))
# If there's any installation mapping information in the body then
# update the project mapping before any requests are made.
@@ -177,7 +188,7 @@
try:
event = method(json_body)
- except:
+ except Exception:
self.log.exception('Exception when handling event:')
event = None
@@ -348,7 +359,7 @@
return
try:
self._handleEvent()
- except:
+ except Exception:
self.log.exception("Exception moving GitHub event:")
finally:
self.connection.eventDone()
@@ -435,6 +446,7 @@
self.registerHttpHandler(self.payload_path,
webhook_listener.handle_request)
self._authenticateGithubAPI()
+ self._prime_installation_map()
self._start_event_connector()
def onStop(self):
@@ -504,8 +516,24 @@
if app_key:
self.app_key = app_key
- def _get_installation_key(self, project, user_id=None):
- installation_id = self.installation_map.get(project)
+ def _get_app_auth_headers(self):
+ now = datetime.datetime.now(utc)
+ expiry = now + datetime.timedelta(minutes=5)
+
+ data = {'iat': now, 'exp': expiry, 'iss': self.app_id}
+ app_token = jwt.encode(data,
+ self.app_key,
+ algorithm='RS256').decode('utf-8')
+
+ headers = {'Accept': PREVIEW_JSON_ACCEPT,
+ 'Authorization': 'Bearer %s' % app_token}
+
+ return headers
+
+ def _get_installation_key(self, project, user_id=None, inst_id=None):
+ installation_id = inst_id
+ if project is not None:
+ installation_id = self.installation_map.get(project)
if not installation_id:
self.log.error("No installation ID available for project %s",
@@ -517,16 +545,8 @@
(None, None))
if ((not expiry) or (not token) or (now >= expiry)):
- expiry = now + datetime.timedelta(minutes=5)
-
- data = {'iat': now, 'exp': expiry, 'iss': self.app_id}
- app_token = jwt.encode(data,
- self.app_key,
- algorithm='RS256').decode('utf-8')
-
+ headers = self._get_app_auth_headers()
url = ACCESS_TOKEN_URL % installation_id
- headers = {'Accept': PREVIEW_JSON_ACCEPT,
- 'Authorization': 'Bearer %s' % app_token}
json_data = {'user_id': user_id} if user_id else None
response = requests.post(url, headers=headers, json=json_data)
@@ -542,8 +562,37 @@
return token
+ def _prime_installation_map(self):
+ """Walks each app install for the repos to prime install IDs"""
+
+ if not self.app_id:
+ return
+
+ url = INSTALLATIONS_URL
+ headers = self._get_app_auth_headers()
+ self.log.debug("Fetching installations for GitHub app")
+ response = requests.get(url, headers=headers)
+ response.raise_for_status()
+
+ data = response.json()
+
+ for install in data:
+ inst_id = install.get('id')
+ token = self._get_installation_key(project=None, inst_id=inst_id)
+ headers = {'Accept': PREVIEW_JSON_ACCEPT,
+ 'Authorization': 'token %s' % token}
+ url = REPOS_URL
+ self.log.debug("Fetching repos for install %s" % inst_id)
+ response = requests.get(url, headers=headers)
+ response.raise_for_status()
+ repos = response.json()
+
+ for repo in repos.get('repositories'):
+ project_name = repo.get('full_name')
+ self.installation_map[project_name] = inst_id
+
def addEvent(self, data, event=None):
- return self.event_queue.put((data, event))
+ return self.event_queue.put((time.time(), data, event))
def getEvent(self):
return self.event_queue.get()
@@ -1052,7 +1101,7 @@
rate_limit = github.rate_limit()
remaining = rate_limit['resources']['core']['remaining']
reset = rate_limit['resources']['core']['reset']
- except:
+ except Exception:
return
if github._zuul_user_id:
log.debug('GitHub API rate limit (%s, %s) remaining: %s reset: %s',
diff --git a/zuul/driver/smtp/smtpconnection.py b/zuul/driver/smtp/smtpconnection.py
index 56ca240..c456c3e 100644
--- a/zuul/driver/smtp/smtpconnection.py
+++ b/zuul/driver/smtp/smtpconnection.py
@@ -52,7 +52,7 @@
s = smtplib.SMTP(self.smtp_server, self.smtp_port)
s.sendmail(from_email, to_email.split(','), msg.as_string())
s.quit()
- except:
+ except Exception:
return "Could not send email via SMTP"
return
diff --git a/zuul/driver/sql/alembic.ini b/zuul/driver/sql/alembic.ini
deleted file mode 100644
index 0c59505..0000000
--- a/zuul/driver/sql/alembic.ini
+++ /dev/null
@@ -1,69 +0,0 @@
-# A generic, single database configuration.
-
-[alembic]
-# path to migration scripts
-# NOTE(jhesketh): We may use alembic for other db components of zuul in the
-# future. Use a sub-folder for the reporters own versions.
-script_location = alembic_reporter
-
-# template used to generate migration files
-# file_template = %%(rev)s_%%(slug)s
-
-# max length of characters to apply to the
-# "slug" field
-#truncate_slug_length = 40
-
-# set to 'true' to run the environment during
-# the 'revision' command, regardless of autogenerate
-# revision_environment = false
-
-# set to 'true' to allow .pyc and .pyo files without
-# a source .py file to be detected as revisions in the
-# versions/ directory
-# sourceless = false
-
-# version location specification; this defaults
-# to alembic/versions. When using multiple version
-# directories, initial revisions must be specified with --version-path
-# version_locations = %(here)s/bar %(here)s/bat alembic/versions
-
-# the output encoding used when revision files
-# are written from script.py.mako
-# output_encoding = utf-8
-
-sqlalchemy.url = mysql+pymysql://user@localhost/database
-
-# Logging configuration
-[loggers]
-keys = root,sqlalchemy,alembic
-
-[handlers]
-keys = console
-
-[formatters]
-keys = generic
-
-[logger_root]
-level = WARN
-handlers = console
-qualname =
-
-[logger_sqlalchemy]
-level = WARN
-handlers =
-qualname = sqlalchemy.engine
-
-[logger_alembic]
-level = INFO
-handlers =
-qualname = alembic
-
-[handler_console]
-class = StreamHandler
-args = (sys.stderr,)
-level = NOTSET
-formatter = generic
-
-[formatter_generic]
-format = %(levelname)-5.5s [%(name)s] %(message)s
-datefmt = %H:%M:%S
diff --git a/zuul/driver/sql/alembic_reporter/README b/zuul/driver/sql/alembic/README
similarity index 100%
rename from zuul/driver/sql/alembic_reporter/README
rename to zuul/driver/sql/alembic/README
diff --git a/zuul/driver/sql/alembic_reporter/env.py b/zuul/driver/sql/alembic/env.py
similarity index 100%
rename from zuul/driver/sql/alembic_reporter/env.py
rename to zuul/driver/sql/alembic/env.py
diff --git a/zuul/driver/sql/alembic_reporter/script.py.mako b/zuul/driver/sql/alembic/script.py.mako
similarity index 100%
rename from zuul/driver/sql/alembic_reporter/script.py.mako
rename to zuul/driver/sql/alembic/script.py.mako
diff --git a/zuul/driver/sql/alembic_reporter/versions/1dd914d4a482_allow_score_to_be_null.py b/zuul/driver/sql/alembic/versions/1dd914d4a482_allow_score_to_be_null.py
similarity index 100%
rename from zuul/driver/sql/alembic_reporter/versions/1dd914d4a482_allow_score_to_be_null.py
rename to zuul/driver/sql/alembic/versions/1dd914d4a482_allow_score_to_be_null.py
diff --git a/zuul/driver/sql/alembic_reporter/versions/20126015a87d_add_indexes.py b/zuul/driver/sql/alembic/versions/20126015a87d_add_indexes.py
similarity index 96%
rename from zuul/driver/sql/alembic_reporter/versions/20126015a87d_add_indexes.py
rename to zuul/driver/sql/alembic/versions/20126015a87d_add_indexes.py
index 3ac680d..12e7c09 100644
--- a/zuul/driver/sql/alembic_reporter/versions/20126015a87d_add_indexes.py
+++ b/zuul/driver/sql/alembic/versions/20126015a87d_add_indexes.py
@@ -53,4 +53,4 @@
def downgrade():
- pass
+ raise Exception("Downgrades not supported")
diff --git a/zuul/driver/sql/alembic_reporter/versions/4d3ebd7f06b9_set_up_initial_reporter_tables.py b/zuul/driver/sql/alembic/versions/4d3ebd7f06b9_set_up_initial_reporter_tables.py
similarity index 100%
rename from zuul/driver/sql/alembic_reporter/versions/4d3ebd7f06b9_set_up_initial_reporter_tables.py
rename to zuul/driver/sql/alembic/versions/4d3ebd7f06b9_set_up_initial_reporter_tables.py
diff --git a/zuul/driver/sql/alembic_reporter/versions/f86c9871ee67_add_tenant_column.py b/zuul/driver/sql/alembic/versions/5efb477fa963_add_ref_url_column.py
similarity index 71%
copy from zuul/driver/sql/alembic_reporter/versions/f86c9871ee67_add_tenant_column.py
copy to zuul/driver/sql/alembic/versions/5efb477fa963_add_ref_url_column.py
index 7728bd4..f9c3535 100644
--- a/zuul/driver/sql/alembic_reporter/versions/f86c9871ee67_add_tenant_column.py
+++ b/zuul/driver/sql/alembic/versions/5efb477fa963_add_ref_url_column.py
@@ -12,17 +12,17 @@
# License for the specific language governing permissions and limitations
# under the License.
-"""Add tenant column
+"""Add ref_url column
-Revision ID: f86c9871ee67
-Revises: 20126015a87d
-Create Date: 2017-07-17 05:47:48.189767
+Revision ID: 5efb477fa963
+Revises: 60c119eb1e3f
+Create Date: 2017-09-12 22:50:29.307695
"""
# revision identifiers, used by Alembic.
-revision = 'f86c9871ee67'
-down_revision = '20126015a87d'
+revision = '5efb477fa963'
+down_revision = '60c119eb1e3f'
branch_labels = None
depends_on = None
@@ -31,8 +31,8 @@
def upgrade():
- op.add_column('zuul_buildset', sa.Column('tenant', sa.String(255)))
+ op.add_column('zuul_buildset', sa.Column('ref_url', sa.String(255)))
def downgrade():
- op.drop_column('zuul_buildset', 'tenant')
+ raise Exception("Downgrades not supported")
diff --git a/zuul/driver/sql/alembic_reporter/versions/60c119eb1e3f_use_build_set_results.py b/zuul/driver/sql/alembic/versions/60c119eb1e3f_use_build_set_results.py
similarity index 66%
rename from zuul/driver/sql/alembic_reporter/versions/60c119eb1e3f_use_build_set_results.py
rename to zuul/driver/sql/alembic/versions/60c119eb1e3f_use_build_set_results.py
index 8e8142f..985eb0c 100644
--- a/zuul/driver/sql/alembic_reporter/versions/60c119eb1e3f_use_build_set_results.py
+++ b/zuul/driver/sql/alembic/versions/60c119eb1e3f_use_build_set_results.py
@@ -35,15 +35,4 @@
def downgrade():
- op.add_column(BUILDSET_TABLE, sa.Column('score', sa.Integer))
-
- connection = op.get_bind()
- connection.execute(
- """
- UPDATE {buildset_table}
- SET score=(
- SELECT CASE result
- WHEN 'SUCCESS' THEN 1
- ELSE -1 END)
- """.format(buildset_table=BUILDSET_TABLE))
- op.drop_column(BUILDSET_TABLE, 'result')
+ raise Exception("Downgrades not supported")
diff --git a/zuul/driver/sql/alembic/versions/ba4cdce9b18c_add_rev_columns.py b/zuul/driver/sql/alembic/versions/ba4cdce9b18c_add_rev_columns.py
new file mode 100644
index 0000000..dc75983
--- /dev/null
+++ b/zuul/driver/sql/alembic/versions/ba4cdce9b18c_add_rev_columns.py
@@ -0,0 +1,25 @@
+"""Add oldrev/newrev columns
+
+Revision ID: ba4cdce9b18c
+Revises: 5efb477fa963
+Create Date: 2017-09-27 19:33:21.800198
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = 'ba4cdce9b18c'
+down_revision = '5efb477fa963'
+branch_labels = None
+depends_on = None
+
+from alembic import op
+import sqlalchemy as sa
+
+
+def upgrade():
+ op.add_column('zuul_buildset', sa.Column('oldrev', sa.String(255)))
+ op.add_column('zuul_buildset', sa.Column('newrev', sa.String(255)))
+
+
+def downgrade():
+ raise Exception("Downgrades not supported")
diff --git a/zuul/driver/sql/alembic_reporter/versions/f86c9871ee67_add_tenant_column.py b/zuul/driver/sql/alembic/versions/f86c9871ee67_add_tenant_column.py
similarity index 95%
rename from zuul/driver/sql/alembic_reporter/versions/f86c9871ee67_add_tenant_column.py
rename to zuul/driver/sql/alembic/versions/f86c9871ee67_add_tenant_column.py
index 7728bd4..4087af3 100644
--- a/zuul/driver/sql/alembic_reporter/versions/f86c9871ee67_add_tenant_column.py
+++ b/zuul/driver/sql/alembic/versions/f86c9871ee67_add_tenant_column.py
@@ -35,4 +35,4 @@
def downgrade():
- op.drop_column('zuul_buildset', 'tenant')
+ raise Exception("Downgrades not supported")
diff --git a/zuul/driver/sql/sqlconnection.py b/zuul/driver/sql/sqlconnection.py
index afdc747..b964c0b 100644
--- a/zuul/driver/sql/sqlconnection.py
+++ b/zuul/driver/sql/sqlconnection.py
@@ -44,11 +44,10 @@
# Recycle connections if they've been idle for more than 1 second.
# MySQL connections are lightweight and thus keeping long-lived
# connections around is not valuable.
- # TODO(mordred) Add a config paramter
self.engine = sa.create_engine(
self.dburi,
poolclass=sqlalchemy.pool.QueuePool,
- pool_recycle=1)
+ pool_recycle=self.connection_config.get('pool_recycle', 1))
self._migrate()
self._setup_tables()
self.zuul_buildset_table, self.zuul_build_table \
@@ -72,7 +71,7 @@
config = alembic.config.Config()
config.set_main_option("script_location",
- "zuul:driver/sql/alembic_reporter")
+ "zuul:driver/sql/alembic")
config.set_main_option("sqlalchemy.url",
self.connection_config.get('dburi'))
@@ -91,6 +90,9 @@
sa.Column('change', sa.Integer, nullable=True),
sa.Column('patchset', sa.Integer, nullable=True),
sa.Column('ref', sa.String(255)),
+ sa.Column('oldrev', sa.String(255)),
+ sa.Column('newrev', sa.String(255)),
+ sa.Column('ref_url', sa.String(255)),
sa.Column('result', sa.String(255)),
sa.Column('message', sa.TEXT()),
sa.Column('tenant', sa.String(255)),
diff --git a/zuul/driver/sql/sqlreporter.py b/zuul/driver/sql/sqlreporter.py
index fdc280a..f9537ac 100644
--- a/zuul/driver/sql/sqlreporter.py
+++ b/zuul/driver/sql/sqlreporter.py
@@ -33,9 +33,11 @@
return
with self.connection.engine.begin() as conn:
- change = getattr(item.change, 'number', '')
- patchset = getattr(item.change, 'patchset', '')
+ change = getattr(item.change, 'number', None)
+ patchset = getattr(item.change, 'patchset', None)
ref = getattr(item.change, 'ref', '')
+ oldrev = getattr(item.change, 'oldrev', '')
+ newrev = getattr(item.change, 'newrev', '')
buildset_ins = self.connection.zuul_buildset_table.insert().values(
zuul_ref=item.current_build_set.ref,
pipeline=item.pipeline.name,
@@ -43,6 +45,9 @@
change=change,
patchset=patchset,
ref=ref,
+ oldrev=oldrev,
+ newrev=newrev,
+ ref_url=item.change.url,
result=item.current_build_set.result,
message=self._formatItemReport(
item, with_jobs=False),
diff --git a/zuul/executor/client.py b/zuul/executor/client.py
index 041f754..fba472f 100644
--- a/zuul/executor/client.py
+++ b/zuul/executor/client.py
@@ -49,7 +49,7 @@
return
try:
self.gearman.lookForLostBuilds()
- except:
+ except Exception:
self.log.exception("Exception checking builds:")
@@ -133,7 +133,7 @@
self.gearman.shutdown()
self.log.debug("Stopped")
- def execute(self, job, item, pipeline, dependent_items=[],
+ def execute(self, job, item, pipeline, dependent_changes=[],
merger_items=[]):
tenant = pipeline.layout.tenant
uuid = str(uuid4().hex)
@@ -143,11 +143,7 @@
job, uuid,
item.current_build_set.getJobNodeSet(job.name),
item.change,
- [x.change for x in dependent_items]))
-
- dependent_items = dependent_items[:]
- dependent_items.reverse()
- all_items = dependent_items + [item]
+ dependent_changes))
# TODOv3(jeblair): This ansible vars data structure will
# replace the environment variables below.
@@ -187,25 +183,8 @@
and item.change.newrev != '0' * 40):
zuul_params['newrev'] = item.change.newrev
zuul_params['projects'] = [] # Set below
- zuul_params['items'] = []
- for i in all_items:
- d = dict()
- d['project'] = dict(
- name=i.change.project.name,
- short_name=i.change.project.name.split('/')[-1],
- canonical_hostname=i.change.project.canonical_hostname,
- canonical_name=i.change.project.canonical_name,
- src_dir=os.path.join('src', i.change.project.canonical_name),
- )
- if hasattr(i.change, 'number'):
- d['change'] = str(i.change.number)
- if hasattr(i.change, 'url'):
- d['change_url'] = i.change.url
- if hasattr(i.change, 'patchset'):
- d['patchset'] = str(i.change.patchset)
- if hasattr(i.change, 'branch'):
- d['branch'] = i.change.branch
- zuul_params['items'].append(d)
+ zuul_params['_projects'] = {} # transitional to convert to dict
+ zuul_params['items'] = dependent_changes
params = dict()
params['job'] = job.name
@@ -217,6 +196,7 @@
else:
params['branch'] = None
params['override_branch'] = job.override_branch
+ params['override_checkout'] = job.override_checkout
params['repo_state'] = item.current_build_set.repo_state
if job.name != 'noop':
@@ -237,7 +217,8 @@
projects = set()
required_projects = set()
- def make_project_dict(project, override_branch=None):
+ def make_project_dict(project, override_branch=None,
+ override_checkout=None):
project_config = item.layout.project_configs.get(
project.canonical_name, None)
if project_config:
@@ -249,6 +230,7 @@
name=project.name,
canonical_name=project.canonical_name,
override_branch=override_branch,
+ override_checkout=override_checkout,
default_branch=project_default_branch)
if job.required_projects:
@@ -260,24 +242,38 @@
(job_project.project_name,))
params['projects'].append(
make_project_dict(project,
- job_project.override_branch))
+ job_project.override_branch,
+ job_project.override_checkout))
projects.add(project)
required_projects.add(project)
- for i in all_items:
- if i.change.project not in projects:
- project = i.change.project
+ for change in dependent_changes:
+ # We have to find the project this way because it may not
+ # be registered in the tenant (ie, a foreign project).
+ source = self.sched.connections.getSourceByHostname(
+ change['project']['canonical_hostname'])
+ project = source.getProject(change['project']['name'])
+ if project not in projects:
params['projects'].append(make_project_dict(project))
projects.add(project)
-
for p in projects:
- zuul_params['projects'].append(dict(
+ zuul_params['_projects'][p.canonical_name] = (dict(
name=p.name,
short_name=p.name.split('/')[-1],
- canonical_hostname=p.canonical_hostname,
+ # Duplicate this into the dict too, so that iterating
+ # project.values() is easier for callers
canonical_name=p.canonical_name,
+ canonical_hostname=p.canonical_hostname,
src_dir=os.path.join('src', p.canonical_name),
required=(p in required_projects),
))
+ # We are transitioning "projects" from a list to a dict
+ # indexed by canonical name, as it is much easier to access
+ # values in ansible. Existing callers are converted to
+ # "_projects", then once "projects" is unused we switch it,
+ # then convert callers back. Finally when "_projects" is
+ # unused it will be removed.
+ for cn, p in zuul_params['_projects'].items():
+ zuul_params['projects'].append(p)
build = Build(job, uuid)
build.parameters = params
@@ -428,7 +424,7 @@
if req.response.startswith(b"OK"):
try:
del self.builds[job.unique]
- except:
+ except Exception:
pass
# Since this isn't otherwise going to get a build complete
# event, send one to the scheduler so that it can unlock
diff --git a/zuul/executor/server.py b/zuul/executor/server.py
index e3f8a24..79fa91e 100644
--- a/zuul/executor/server.py
+++ b/zuul/executor/server.py
@@ -353,6 +353,13 @@
self.pre_playbooks = []
self.post_playbooks = []
self.job_output_file = os.path.join(self.log_root, 'job-output.txt')
+ # We need to create the job-output.txt upfront in order to close the
+ # gap between url reporting and ansible creating the file. Otherwise
+ # there is a period of time where the user can click on the live log
+ # link on the status page but the log streaming fails because the file
+ # is not there yet.
+ with open(self.job_output_file, 'w'):
+ pass
self.trusted_projects = []
self.trusted_project_index = {}
@@ -674,7 +681,9 @@
ref,
args['branch'],
args['override_branch'],
+ args['override_checkout'],
project['override_branch'],
+ project['override_checkout'],
project['default_branch'])
# Delete the origin remote from each repo we set up since
@@ -750,22 +759,32 @@
return True
def checkoutBranch(self, repo, project_name, ref, zuul_branch,
- job_branch, project_override_branch,
+ job_override_branch, job_override_checkout,
+ project_override_branch, project_override_checkout,
project_default_branch):
branches = repo.getBranches()
+ refs = [r.name for r in repo.getRefs()]
if project_override_branch in branches:
self.log.info("Checking out %s project override branch %s",
project_name, project_override_branch)
- repo.checkoutLocalBranch(project_override_branch)
- elif job_branch in branches:
- self.log.info("Checking out %s job branch %s",
- project_name, job_branch)
- repo.checkoutLocalBranch(job_branch)
+ repo.checkout(project_override_branch)
+ if project_override_checkout in refs:
+ self.log.info("Checking out %s project override ref %s",
+ project_name, project_override_checkout)
+ repo.checkout(project_override_checkout)
+ elif job_override_branch in branches:
+ self.log.info("Checking out %s job override branch %s",
+ project_name, job_override_branch)
+ repo.checkout(job_override_branch)
+ elif job_override_checkout in refs:
+ self.log.info("Checking out %s job override ref %s",
+ project_name, job_override_checkout)
+ repo.checkout(job_override_checkout)
elif ref and ref.startswith('refs/heads/'):
b = ref[len('refs/heads/'):]
self.log.info("Checking out %s branch ref %s",
project_name, b)
- repo.checkoutLocalBranch(b)
+ repo.checkout(b)
elif ref and ref.startswith('refs/tags/'):
t = ref[len('refs/tags/'):]
self.log.info("Checking out %s tag ref %s",
@@ -774,11 +793,11 @@
elif zuul_branch and zuul_branch in branches:
self.log.info("Checking out %s zuul branch %s",
project_name, zuul_branch)
- repo.checkoutLocalBranch(zuul_branch)
+ repo.checkout(zuul_branch)
elif project_default_branch in branches:
self.log.info("Checking out %s project default branch %s",
project_name, project_default_branch)
- repo.checkoutLocalBranch(project_default_branch)
+ repo.checkout(project_default_branch)
else:
raise ExecutorError("Project %s does not have the "
"default branch %s" %
@@ -901,6 +920,10 @@
private_ipv4=node.get('private_ipv4'),
public_ipv6=node.get('public_ipv6')))
+ username = node.get('username')
+ if username:
+ host_vars['ansible_user'] = username
+
host_keys = []
for key in node.get('host_keys'):
if port != 22:
@@ -928,43 +951,38 @@
"Ansible plugin dir %s found adjacent to playbook %s in "
"non-trusted repo." % (entry, path))
- def findPlaybook(self, path, required=False, trusted=False):
- for ext in ['.yaml', '.yml']:
+ def findPlaybook(self, path, trusted=False):
+ for ext in ['', '.yaml', '.yml']:
fn = path + ext
if os.path.exists(fn):
if not trusted:
playbook_dir = os.path.dirname(os.path.abspath(fn))
self._blockPluginDirs(playbook_dir)
return fn
- if required:
- raise ExecutorError("Unable to find playbook %s" % path)
- return None
+ raise ExecutorError("Unable to find playbook %s" % path)
def preparePlaybooks(self, args):
self.writeAnsibleConfig(self.jobdir.setup_playbook)
for playbook in args['pre_playbooks']:
jobdir_playbook = self.jobdir.addPrePlaybook()
- self.preparePlaybook(jobdir_playbook, playbook,
- args, required=True)
+ self.preparePlaybook(jobdir_playbook, playbook, args)
for playbook in args['playbooks']:
jobdir_playbook = self.jobdir.addPlaybook()
- self.preparePlaybook(jobdir_playbook, playbook,
- args, required=False)
+ self.preparePlaybook(jobdir_playbook, playbook, args)
if jobdir_playbook.path is not None:
self.jobdir.playbook = jobdir_playbook
break
if self.jobdir.playbook is None:
- raise ExecutorError("No valid playbook found")
+ raise ExecutorError("No playbook specified")
for playbook in args['post_playbooks']:
jobdir_playbook = self.jobdir.addPostPlaybook()
- self.preparePlaybook(jobdir_playbook, playbook,
- args, required=True)
+ self.preparePlaybook(jobdir_playbook, playbook, args)
- def preparePlaybook(self, jobdir_playbook, playbook, args, required):
+ def preparePlaybook(self, jobdir_playbook, playbook, args):
self.log.debug("Prepare playbook repo for %s" %
(playbook['project'],))
# Check out the playbook repo if needed and set the path to
@@ -999,7 +1017,6 @@
jobdir_playbook.path = self.findPlaybook(
path,
- required=required,
trusted=playbook['trusted'])
# If this playbook doesn't exist, don't bother preparing
@@ -1538,7 +1555,8 @@
self.jobdir_root = jobdir_root
# TODOv3(mordred): make the executor name more unique --
# perhaps hostname+pid.
- self.hostname = socket.gethostname()
+ self.hostname = get_default(self.config, 'executor', 'hostname',
+ socket.gethostname())
self.log_streaming_port = log_streaming_port
self.merger_lock = threading.Lock()
self.run_lock = threading.Lock()
@@ -1688,7 +1706,7 @@
def unregister_work(self):
self.accepting_work = False
- self.executor_worker.unregisterFunction("executor:execute")
+ self.executor_worker.unRegisterFunction("executor:execute")
def stop(self):
self.log.debug("Stopping")
diff --git a/zuul/lib/connections.py b/zuul/lib/connections.py
index 79d78f4..262490a 100644
--- a/zuul/lib/connections.py
+++ b/zuul/lib/connections.py
@@ -161,3 +161,10 @@
def getTrigger(self, connection_name, config=None):
connection = self.connections[connection_name]
return connection.driver.getTrigger(connection, config)
+
+ def getSourceByHostname(self, canonical_hostname):
+ for connection in self.connections.values():
+ if hasattr(connection, 'canonical_hostname'):
+ if connection.canonical_hostname == canonical_hostname:
+ return self.getSource(connection.connection_name)
+ return None
diff --git a/zuul/lib/log_streamer.py b/zuul/lib/log_streamer.py
index 3ecaf4d..1906be7 100644
--- a/zuul/lib/log_streamer.py
+++ b/zuul/lib/log_streamer.py
@@ -49,6 +49,11 @@
MAX_REQUEST_LEN = 1024
REQUEST_TIMEOUT = 10
+ # NOTE(Shrews): We only use this to log exceptions since a new process
+ # is used per-request (and having multiple processes write to the same
+ # log file constantly is bad).
+ log = logging.getLogger("zuul.log_streamer.RequestHandler")
+
def get_command(self):
poll = select.poll()
bitmask = (select.POLLIN | select.POLLERR |
@@ -78,7 +83,14 @@
pass
def handle(self):
- build_uuid = self.get_command()
+ try:
+ build_uuid = self.get_command()
+ except Exception:
+ self.log.exception("Failure during get_command:")
+ msg = 'Internal streaming error'
+ self.request.sendall(msg.encode("utf-8"))
+ return
+
build_uuid = build_uuid.rstrip()
# validate build ID
@@ -100,7 +112,13 @@
self.request.sendall(msg.encode("utf-8"))
return
- self.stream_log(log_file)
+ try:
+ self.stream_log(log_file)
+ except Exception:
+ self.log.exception("Streaming failure for build UUID %s:",
+ build_uuid)
+ msg = 'Internal streaming error'
+ self.request.sendall(msg.encode("utf-8"))
def stream_log(self, log_file):
log = None
@@ -108,7 +126,7 @@
if log is not None:
try:
log.file.close()
- except:
+ except Exception:
pass
while True:
log = self.chunk_log(log_file)
@@ -164,7 +182,7 @@
return False
-class CustomForkingTCPServer(socketserver.ForkingTCPServer):
+class CustomThreadingTCPServer(socketserver.ThreadingTCPServer):
'''
Custom version that allows us to drop privileges after port binding.
'''
@@ -175,7 +193,7 @@
self.jobdir_root = kwargs.pop('jobdir_root')
# For some reason, setting custom attributes does not work if we
# call the base class __init__ first. Wha??
- socketserver.ForkingTCPServer.__init__(self, *args, **kwargs)
+ socketserver.ThreadingTCPServer.__init__(self, *args, **kwargs)
def change_privs(self):
'''
@@ -191,7 +209,7 @@
def server_bind(self):
self.allow_reuse_address = True
- socketserver.ForkingTCPServer.server_bind(self)
+ socketserver.ThreadingTCPServer.server_bind(self)
if self.user:
self.change_privs()
@@ -208,6 +226,16 @@
return
raise
+ def process_request(self, request, client_address):
+ '''
+ Overridden from the base class to name the thread.
+ '''
+ t = threading.Thread(target=self.process_request_thread,
+ name='FingerStreamer',
+ args=(request, client_address))
+ t.daemon = self.daemon_threads
+ t.start()
+
class LogStreamer(object):
'''
@@ -217,19 +245,27 @@
def __init__(self, user, host, port, jobdir_root):
self.log = logging.getLogger('zuul.lib.LogStreamer')
self.log.debug("LogStreamer starting on port %s", port)
- self.server = CustomForkingTCPServer((host, port),
- RequestHandler,
- user=user,
- jobdir_root=jobdir_root)
+ self.server = CustomThreadingTCPServer((host, port),
+ RequestHandler,
+ user=user,
+ jobdir_root=jobdir_root)
# We start the actual serving within a thread so we can return to
# the owner.
- self.thd = threading.Thread(target=self.server.serve_forever)
+ self.thd = threading.Thread(target=self._run)
self.thd.daemon = True
self.thd.start()
+ def _run(self):
+ try:
+ self.server.serve_forever()
+ except Exception:
+ self.log.exception("Abnormal termination:")
+ raise
+
def stop(self):
if self.thd.isAlive():
self.server.shutdown()
self.server.server_close()
+ self.thd.join()
self.log.debug("LogStreamer stopped")
diff --git a/zuul/lib/queue.py b/zuul/lib/queue.py
new file mode 100644
index 0000000..db8af47
--- /dev/null
+++ b/zuul/lib/queue.py
@@ -0,0 +1,78 @@
+# Copyright 2014 OpenStack Foundation
+# Copyright 2017 Red Hat, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+
+import collections
+import threading
+
+
+class MergedQueue(object):
+ def __init__(self):
+ self.queue = collections.deque()
+ self.lock = threading.RLock()
+ self.condition = threading.Condition(self.lock)
+ self.join_condition = threading.Condition(self.lock)
+ self.tasks = 0
+
+ def qsize(self):
+ return len(self.queue)
+
+ def empty(self):
+ return self.qsize() == 0
+
+ def put(self, item):
+ # Returns the original item if added, or an updated equivalent
+ # item if already enqueued.
+ self.condition.acquire()
+ ret = None
+ try:
+ for x in self.queue:
+ if item == x:
+ ret = x
+ if hasattr(ret, 'merge'):
+ ret.merge(item)
+ if ret is None:
+ ret = item
+ self.queue.append(item)
+ self.condition.notify()
+ finally:
+ self.condition.release()
+ return ret
+
+ def get(self):
+ self.condition.acquire()
+ try:
+ while True:
+ try:
+ ret = self.queue.popleft()
+ self.join_condition.acquire()
+ self.tasks += 1
+ self.join_condition.release()
+ return ret
+ except IndexError:
+ self.condition.wait()
+ finally:
+ self.condition.release()
+
+ def task_done(self):
+ self.join_condition.acquire()
+ self.tasks -= 1
+ self.join_condition.notify()
+ self.join_condition.release()
+
+ def join(self):
+ self.join_condition.acquire()
+ while self.tasks:
+ self.join_condition.wait()
+ self.join_condition.release()
diff --git a/zuul/manager/__init__.py b/zuul/manager/__init__.py
index edea69c..6c72c2d 100644
--- a/zuul/manager/__init__.py
+++ b/zuul/manager/__init__.py
@@ -156,7 +156,7 @@
if ret:
self.log.error("Reporting item start %s received: %s" %
(item, ret))
- except:
+ except Exception:
self.log.exception("Exception while reporting start:")
def sendReport(self, action_reporters, item, message=None):
@@ -223,13 +223,19 @@
if item.change.equals(change):
self.removeItem(item)
- def reEnqueueItem(self, item, last_head):
+ def reEnqueueItem(self, item, last_head, old_item_ahead, item_ahead_valid):
with self.getChangeQueue(item.change, last_head.queue) as change_queue:
if change_queue:
self.log.debug("Re-enqueing change %s in queue %s" %
(item.change, change_queue))
change_queue.enqueueItem(item)
+ # If the old item ahead was re-enqued, this value will
+ # be true, so we should attempt to move the item back
+ # to where it was in case an item ahead is already
+ # failing.
+ if item_ahead_valid:
+ change_queue.moveItem(item, old_item_ahead)
# Get an updated copy of the layout and update the job
# graph if necessary. This resumes the buildset merge
# state machine. If we have an up-to-date layout, it
@@ -358,14 +364,14 @@
try:
nodeset = item.current_build_set.getJobNodeSet(job.name)
self.sched.nodepool.useNodeSet(nodeset)
- build = self.sched.executor.execute(job, item,
- self.pipeline,
- build_set.dependent_items,
- build_set.merger_items)
+ build = self.sched.executor.execute(
+ job, item, self.pipeline,
+ build_set.dependent_changes,
+ build_set.merger_items)
self.log.debug("Adding build %s of job %s to item %s" %
(build, job, item))
item.addBuild(build)
- except:
+ except Exception:
self.log.exception("Exception while executing job %s "
"for change %s:" % (job, item.change))
@@ -397,7 +403,7 @@
was_running = False
try:
was_running = self.sched.executor.cancel(build)
- except:
+ except Exception:
self.log.exception("Exception while canceling build %s "
"for change %s" % (build, item.change))
finally:
@@ -820,7 +826,7 @@
if ret:
self.log.error("Reporting item %s received: %s" %
(item, ret))
- except:
+ except Exception:
self.log.exception("Exception while reporting:")
item.setReportedResult('ERROR')
return ret
@@ -862,5 +868,5 @@
if dt:
self.sched.statsd.timing(key + '.resident_time', dt)
self.sched.statsd.incr(key + '.total_changes')
- except:
+ except Exception:
self.log.exception("Exception reporting pipeline stats")
diff --git a/zuul/merger/merger.py b/zuul/merger/merger.py
index 035d1d0..06ec4b2 100644
--- a/zuul/merger/merger.py
+++ b/zuul/merger/merger.py
@@ -176,6 +176,8 @@
return branch in origin.refs
def getBranches(self):
+ # TODO(jeblair): deprecate with override-branch; replaced by
+ # getRefs().
repo = self.createRepoObject()
return [x.name for x in repo.heads]
@@ -386,7 +388,7 @@
self.log.info("Checking out %s/%s branch %s",
connection_name, project_name, branch)
repo = self.getRepo(connection_name, project_name)
- repo.checkoutLocalBranch(branch)
+ repo.checkout(branch)
def _saveRepoState(self, connection_name, project_name, repo,
repo_state, recent):
diff --git a/zuul/model.py b/zuul/model.py
index 464ee16..081d165 100644
--- a/zuul/model.py
+++ b/zuul/model.py
@@ -79,6 +79,12 @@
STATE_DELETING])
+class NoMatchingParentError(Exception):
+ """A job referenced a parent, but that parent had no variants which
+ matched the current change."""
+ pass
+
+
class Attributes(object):
"""A class to hold attributes for string formatting."""
@@ -352,6 +358,9 @@
def __repr__(self):
return '<Project %s>' % (self.name)
+ def getSafeAttributes(self):
+ return Attributes(name=self.name)
+
class Node(object):
"""A single node for use by a job.
@@ -379,6 +388,7 @@
self.az = None
self.provider = None
self.region = None
+ self.username = None
@property
def state(self):
@@ -629,6 +639,7 @@
self.branch = branch
self.path = path
self.trusted = trusted
+ self.implied_branch_matchers = None
def __str__(self):
return '%s/%s@%s' % (self.project, self.path, self.branch)
@@ -697,6 +708,13 @@
self.roles == other.roles and
self.secrets == other.secrets)
+ def copy(self):
+ r = PlaybookContext(self.source_context,
+ self.path,
+ self.roles,
+ self.secrets)
+ return r
+
def toDict(self):
# Render to a dict to use in passing json to the executor
secrets = {}
@@ -785,6 +803,8 @@
(e.g., "job.run = ..." rather than "job.run.append(...)").
"""
+ BASE_JOB_MARKER = object()
+
def __init__(self, name):
# These attributes may override even the final form of a job
# in the context of a project-pipeline. They can not affect
@@ -811,6 +831,7 @@
# declared "final", these may not be overriden in a
# project-pipeline.
self.execution_attributes = dict(
+ parent=None,
timeout=None,
variables={},
nodeset=NodeSet(),
@@ -818,7 +839,6 @@
pre_run=(),
post_run=(),
run=(),
- implied_run=(),
semaphore=None,
attempts=3,
final=False,
@@ -826,6 +846,7 @@
required_projects={},
allowed_projects=None,
override_branch=None,
+ override_checkout=None,
post_review=None,
)
@@ -836,6 +857,7 @@
source_context=None,
source_line=None,
inheritance_path=(),
+ parent_data=None,
)
self.inheritable_attributes = {}
@@ -887,11 +909,11 @@
def getSafeAttributes(self):
return Attributes(name=self.name)
- def setRun(self):
- msg = 'self %s' % (repr(self),)
- self.inheritance_path = self.inheritance_path + (msg,)
- if not self.run:
- self.run = self.implied_run
+ def isBase(self):
+ return self.parent is self.BASE_JOB_MARKER
+
+ def setBase(self):
+ self.inheritance_path = self.inheritance_path + (repr(self),)
def addRoles(self, roles):
newroles = []
@@ -927,11 +949,48 @@
matchers.append(change_matcher.BranchMatcher(branch))
self.branch_matcher = change_matcher.MatchAny(matchers)
+ def getSimpleBranchMatcher(self):
+ # If the job has a simple branch matcher, return it; otherwise None.
+ if not self.branch_matcher:
+ return None
+ m = self.branch_matcher
+ if not isinstance(m, change_matcher.AbstractMatcherCollection):
+ return None
+ if len(m.matchers) != 1:
+ return None
+ m = m.matchers[0]
+ if not isinstance(m, change_matcher.BranchMatcher):
+ return None
+ return m._regex
+
+ def addBranchMatcher(self, branch):
+ # Add a branch matcher that combines as a boolean *and* with
+ # existing branch matchers, if any.
+ matchers = [change_matcher.BranchMatcher(branch)]
+ if self.branch_matcher:
+ matchers.append(self.branch_matcher)
+ self.branch_matcher = change_matcher.MatchAll(matchers)
+
def updateVariables(self, other_vars):
v = copy.deepcopy(self.variables)
Job._deepUpdate(v, other_vars)
self.variables = v
+ def updateParentData(self, other_vars):
+ # Update variables, but give the current values priority (used
+ # for job return data which is lower precedence than defined
+ # job vars).
+ v = self.parent_data or {}
+ Job._deepUpdate(v, other_vars)
+ # To avoid running afoul of checks that jobs don't set zuul
+ # variables, remove them from parent data here.
+ if 'zuul' in v:
+ del v['zuul']
+ self.parent_data = v
+ v = copy.deepcopy(self.parent_data)
+ Job._deepUpdate(v, self.variables)
+ self.variables = v
+
def updateProjects(self, other_projects):
required_projects = self.required_projects.copy()
required_projects.update(other_projects)
@@ -948,24 +1007,6 @@
else:
a[k] = bv
- def inheritFrom(self, other):
- """Copy the inheritable attributes which have been set on the other
- job to this job."""
- if not isinstance(other, Job):
- raise Exception("Job unable to inherit from %s" % (other,))
-
- if other.final:
- raise Exception("Unable to inherit from final job %s" %
- (repr(other),))
-
- # copy all attributes
- for k in self.inheritable_attributes:
- if (other._get(k) is not None):
- setattr(self, k, getattr(other, k))
-
- msg = 'inherit from %s' % (repr(other),)
- self.inheritance_path = other.inheritance_path + (msg,)
-
def copy(self):
job = Job(self.name)
for k in self.attributes:
@@ -973,10 +1014,22 @@
setattr(job, k, copy.deepcopy(self._get(k)))
return job
+ def freezePlaybooks(self, pblist):
+ """Take a list of playbooks, and return a copy of it updated with this
+ job's roles.
+
+ """
+
+ ret = []
+ for old_pb in pblist:
+ pb = old_pb.copy()
+ pb.roles = self.roles
+ ret.append(pb)
+ return tuple(ret)
+
def applyVariant(self, other):
"""Copy the attributes which have been set on the other job to this
job."""
-
if not isinstance(other, Job):
raise Exception("Job unable to inherit from %s" % (other,))
@@ -988,8 +1041,9 @@
"%s=%s with variant %s" % (
repr(self), k, other._get(k),
repr(other)))
- if k not in set(['pre_run', 'post_run', 'roles', 'variables',
- 'required_projects']):
+ if k not in set(['pre_run', 'run', 'post_run', 'roles',
+ 'variables', 'required_projects']):
+ # TODO(jeblair): determine if deepcopy is required
setattr(self, k, copy.deepcopy(other._get(k)))
# Don't set final above so that we don't trip an error halfway
@@ -997,12 +1051,19 @@
if other.final != self.attributes['final']:
self.final = other.final
- if other._get('pre_run') is not None:
- self.pre_run = self.pre_run + other.pre_run
- if other._get('post_run') is not None:
- self.post_run = other.post_run + self.post_run
+ # We must update roles before any playbook contexts
if other._get('roles') is not None:
self.addRoles(other.roles)
+
+ if other._get('run') is not None:
+ other_run = self.freezePlaybooks(other.run)
+ self.run = other_run
+ if other._get('pre_run') is not None:
+ other_pre_run = self.freezePlaybooks(other.pre_run)
+ self.pre_run = self.pre_run + other_pre_run
+ if other._get('post_run') is not None:
+ other_post_run = self.freezePlaybooks(other.post_run)
+ self.post_run = other_post_run + self.post_run
if other._get('variables') is not None:
self.updateVariables(other.variables)
if other._get('required_projects') is not None:
@@ -1016,8 +1077,7 @@
if other._get('tags') is not None:
self.tags = self.tags.union(other.tags)
- msg = 'apply variant %s' % (repr(other),)
- self.inheritance_path = self.inheritance_path + (msg,)
+ self.inheritance_path = self.inheritance_path + (repr(other),)
def changeMatches(self, change):
if self.branch_matcher and not self.branch_matcher.matches(change):
@@ -1037,9 +1097,11 @@
class JobProject(object):
""" A reference to a project from a job. """
- def __init__(self, project_name, override_branch=None):
+ def __init__(self, project_name, override_branch=None,
+ override_checkout=None):
self.project_name = project_name
self.override_branch = override_branch
+ self.override_checkout = override_checkout
class JobList(object):
@@ -1058,10 +1120,28 @@
for jobname, jobs in other.jobs.items():
joblist = self.jobs.setdefault(jobname, [])
for job in jobs:
- if not job.branch_matcher and implied_branch:
- job = job.copy()
- job.setBranchMatcher([implied_branch])
- joblist.append(job)
+ if implied_branch:
+ # If setting an implied branch and the current
+ # branch matcher is a simple match for a different
+ # branch, then simply do not add this job. If it
+ # is absent, set it to the implied branch.
+ # Otherwise, combine it with the implied branch to
+ # ensure that it still only affects this branch
+ # (whatever else it may do).
+ simple_branch = job.getSimpleBranchMatcher()
+ if simple_branch and simple_branch != implied_branch:
+ # Job is for a different branch, don't add it.
+ continue
+ if not simple_branch:
+ # The branch matcher could be complex, or
+ # missing. Add our implied matcher.
+ job = job.copy()
+ job.addBranchMatcher(implied_branch)
+ # Otherwise we have a simple branch matcher which
+ # is the same as our implied branch, the job can
+ # be added as-is.
+ if job not in joblist:
+ joblist.append(job)
class JobGraph(object):
@@ -1263,11 +1343,9 @@
self.item = item
self.builds = {}
self.result = None
- self.next_build_set = None
- self.previous_build_set = None
self.uuid = None
self.commit = None
- self.dependent_items = None
+ self.dependent_changes = None
self.merger_items = None
self.unable_to_merge = False
self.config_error = None # None or an error message string.
@@ -1296,18 +1374,16 @@
# The change isn't enqueued until after it's created
# so we don't know what the other changes ahead will be
# until jobs start.
- if self.dependent_items is None:
- items = []
+ if not self.uuid:
+ self.uuid = uuid4().hex
+ if self.dependent_changes is None:
+ items = [self.item]
next_item = self.item.item_ahead
while next_item:
items.append(next_item)
next_item = next_item.item_ahead
- self.dependent_items = items
- if not self.uuid:
- self.uuid = uuid4().hex
- if self.merger_items is None:
- items = [self.item] + self.dependent_items
items.reverse()
+ self.dependent_changes = [i.change.toDict() for i in items]
self.merger_items = [i.makeMergerItem() for i in items]
def getStateName(self, state_num):
@@ -1402,10 +1478,8 @@
self.pipeline = queue.pipeline
self.queue = queue
self.change = change # a ref
- self.build_sets = []
self.dequeued_needing_change = False
self.current_build_set = BuildSet(self)
- self.build_sets.append(self.current_build_set)
self.item_ahead = None
self.items_behind = []
self.enqueue_time = None
@@ -1427,12 +1501,7 @@
id(self), self.change, pipeline)
def resetAllBuilds(self):
- old = self.current_build_set
- self.current_build_set.result = 'CANCELED'
self.current_build_set = BuildSet(self)
- old.next_build_set = self.current_build_set
- self.current_build_set.previous_build_set = old
- self.build_sets.append(self.current_build_set)
self.layout = None
self.job_graph = None
@@ -1582,18 +1651,33 @@
else:
jobs_not_started.add(job)
- # Attempt to request nodes for jobs in the order jobs appear
- # in configuration.
+ # Attempt to run jobs in the order they appear in
+ # configuration.
for job in self.job_graph.getJobs():
if job not in jobs_not_started:
continue
all_parent_jobs_successful = True
+ parent_builds_with_data = {}
for parent_job in self.job_graph.getParentJobsRecursively(
job.name):
if parent_job.name not in successful_job_names:
all_parent_jobs_successful = False
break
+ parent_build = self.current_build_set.getBuild(parent_job.name)
+ if parent_build.result_data:
+ parent_builds_with_data[parent_job.name] = parent_build
+
if all_parent_jobs_successful:
+ # Iterate in reverse order over all jobs of the graph (which is
+ # in sorted config order) and apply parent data of the jobs we
+ # already found.
+ if len(parent_builds_with_data) > 0:
+ for parent_job in reversed(self.job_graph.getJobs()):
+ parent_build = parent_builds_with_data.get(
+ parent_job.name)
+ if parent_build:
+ job.updateParentData(parent_build.result_data)
+
nodeset = self.current_build_set.getJobNodeSet(job.name)
if nodeset is None:
# The nodes for this job are not ready, skip
@@ -1979,6 +2063,18 @@
oldrev=self.oldrev,
newrev=self.newrev)
+ def toDict(self):
+ # Render to a dict to use in passing json to the executor
+ d = dict()
+ d['project'] = dict(
+ name=self.project.name,
+ short_name=self.project.name.split('/')[-1],
+ canonical_hostname=self.project.canonical_hostname,
+ canonical_name=self.project.canonical_name,
+ src_dir=os.path.join('src', self.project.canonical_name),
+ )
+ return d
+
class Branch(Ref):
"""An existing branch state for a Project."""
@@ -1986,6 +2082,12 @@
super(Branch, self).__init__(project)
self.branch = None
+ def toDict(self):
+ # Render to a dict to use in passing json to the executor
+ d = super(Branch, self).toDict()
+ d['branch'] = self.branch
+ return d
+
class Tag(Ref):
"""An existing tag state for a Project."""
@@ -2048,6 +2150,14 @@
number=self.number,
patchset=self.patchset)
+ def toDict(self):
+ # Render to a dict to use in passing json to the executor
+ d = super(Change, self).toDict()
+ d['change'] = str(self.number)
+ d['change_url'] = self.url
+ d['patchset'] = str(self.patchset)
+ return d
+
class TriggerEvent(object):
"""Incoming event from an external system."""
@@ -2138,7 +2248,7 @@
self.project = project
self.load_classes = set()
self.shadow_projects = set()
-
+ self.branches = []
# The tenant's default setting of exclude_unprotected_branches will
# be overridden by this one if not None.
self.exclude_unprotected_branches = None
@@ -2146,8 +2256,11 @@
class ProjectConfig(object):
# Represents a project cofiguration
- def __init__(self, name):
+ def __init__(self, name, source_context=None):
self.name = name
+ # If this is a template, it will have a source_context, but
+ # not if it is a project definition.
+ self.source_context = source_context
self.merge_mode = None
# The default branch for the project (usually master).
self.default_branch = None
@@ -2266,6 +2379,7 @@
"""A collection of yaml lists that has not yet been parsed into objects."""
def __init__(self):
+ self.pragmas = []
self.pipelines = []
self.jobs = []
self.project_templates = []
@@ -2276,6 +2390,7 @@
def copy(self):
r = UnparsedTenantConfig()
+ r.pragmas = copy.deepcopy(self.pragmas)
r.pipelines = copy.deepcopy(self.pipelines)
r.jobs = copy.deepcopy(self.jobs)
r.project_templates = copy.deepcopy(self.project_templates)
@@ -2287,6 +2402,7 @@
def extend(self, conf):
if isinstance(conf, UnparsedTenantConfig):
+ self.pragmas.extend(conf.pragmas)
self.pipelines.extend(conf.pipelines)
self.jobs.extend(conf.jobs)
self.project_templates.extend(conf.project_templates)
@@ -2321,6 +2437,8 @@
self.secrets.append(value)
elif key == 'semaphore':
self.semaphores.append(value)
+ elif key == 'pragma':
+ self.pragmas.append(value)
else:
raise ConfigItemUnknownError()
@@ -2340,7 +2458,10 @@
# elements are aspects of that job with different matchers
# that override some attribute of the job. These aspects all
# inherit from the reference definition.
- self.jobs = {'noop': [Job('noop')]}
+ noop = Job('noop')
+ noop.parent = noop.BASE_JOB_MARKER
+ noop.run = 'noop.yaml'
+ self.jobs = {'noop': [noop]}
self.nodesets = {}
self.secrets = {}
self.semaphores = {}
@@ -2408,32 +2529,73 @@
self.pipelines[pipeline.name] = pipeline
def addProjectTemplate(self, project_template):
- self.project_templates[project_template.name] = project_template
+ template = self.project_templates.get(project_template.name)
+ if template:
+ if (project_template.source_context.project !=
+ template.source_context.project):
+ raise Exception("Project template %s is already defined" %
+ (project_template.name,))
+ for pipeline in project_template.pipelines:
+ template.pipelines[pipeline].job_list.\
+ inheritFrom(project_template.pipelines[pipeline].job_list,
+ None)
+ else:
+ self.project_templates[project_template.name] = project_template
def addProjectConfig(self, project_config):
self.project_configs[project_config.name] = project_config
+ def collectJobs(self, jobname, change, path=None, jobs=None, stack=None):
+ if stack is None:
+ stack = []
+ if jobs is None:
+ jobs = []
+ if path is None:
+ path = []
+ path.append(jobname)
+ matched = False
+ for variant in self.getJobs(jobname):
+ if not variant.changeMatches(change):
+ continue
+ if not variant.isBase():
+ parent = variant.parent
+ if not jobs and parent is None:
+ parent = self.tenant.default_base_job
+ else:
+ parent = None
+ if parent and parent not in path:
+ if parent in stack:
+ raise Exception("Dependency cycle in jobs: %s" % stack)
+ self.collectJobs(parent, change, path, jobs, stack + [jobname])
+ matched = True
+ jobs.append(variant)
+ if not matched:
+ raise NoMatchingParentError()
+ return jobs
+
def _createJobGraph(self, item, job_list, job_graph):
change = item.change
pipeline = item.pipeline
for jobname in job_list.jobs:
# This is the final job we are constructing
frozen_job = None
- # Whether the change matches any globally defined variant
- matched = False
- for variant in self.getJobs(jobname):
- if variant.changeMatches(change):
- if frozen_job is None:
- frozen_job = variant.copy()
- frozen_job.setRun()
- else:
- frozen_job.applyVariant(variant)
- matched = True
- if not matched:
+ try:
+ variants = self.collectJobs(jobname, change)
+ except NoMatchingParentError:
+ variants = None
+ if not variants:
# A change must match at least one defined job variant
# (that is to say that it must match more than just
# the job that is defined in the tree).
continue
+ for variant in variants:
+ if frozen_job is None:
+ frozen_job = variant.copy()
+ frozen_job.setBase()
+ else:
+ frozen_job.applyVariant(variant)
+ frozen_job.name = variant.name
+ frozen_job.name = jobname
# Whether the change matches any of the project pipeline
# variants
matched = False
@@ -2453,6 +2615,9 @@
raise Exception("Pre-review pipeline %s does not allow "
"post-review job %s" % (
pipeline.name, frozen_job.name))
+ if not frozen_job.run:
+ raise Exception("Job %s does not specify a run playbook" % (
+ frozen_job.name,))
job_graph.addJob(frozen_job)
def createJobGraph(self, item):
@@ -2655,6 +2820,18 @@
raise Exception("Project %s is neither trusted nor untrusted" %
(project,))
+ def getProjectBranches(self, project):
+ """Return a project's branches (filtered by this tenant config)
+
+ :arg Project project: The project object.
+
+ :returns: A list of branch names.
+ :rtype: [str]
+
+ """
+ tpc = self.project_configs[project.canonical_name]
+ return tpc.branches
+
def addConfigProject(self, tpc):
self.config_projects.append(tpc.project)
self._addProject(tpc)
diff --git a/zuul/nodepool.py b/zuul/nodepool.py
index 7dafca0..b96d1ca 100644
--- a/zuul/nodepool.py
+++ b/zuul/nodepool.py
@@ -87,9 +87,6 @@
:param set autohold_key: A set with the tenant/project/job names
associated with the given NodeSet.
'''
- if autohold_key not in self.sched.autohold_requests:
- return
-
(hold_iterations, reason) = self.sched.autohold_requests[autohold_key]
nodes = nodeset.getNodes()
diff --git a/zuul/rpclistener.py b/zuul/rpclistener.py
index 11d6684..8c8c783 100644
--- a/zuul/rpclistener.py
+++ b/zuul/rpclistener.py
@@ -56,6 +56,31 @@
self.worker.registerFunction("zuul:promote")
self.worker.registerFunction("zuul:get_running_jobs")
self.worker.registerFunction("zuul:get_job_log_stream_address")
+ self.worker.registerFunction("zuul:tenant_list")
+ self.worker.registerFunction("zuul:status_get")
+
+ def getFunctions(self):
+ functions = {}
+ for connection in self.worker.active_connections:
+ try:
+ req = gear.StatusAdminRequest()
+ connection.sendAdminRequest(req, timeout=300)
+ except Exception:
+ self.log.exception("Exception while listing functions")
+ self.worker._lostConnection(connection)
+ continue
+ for line in req.response.decode('utf8').split('\n'):
+ parts = [x.strip() for x in line.split('\t')]
+ if len(parts) < 4:
+ continue
+ # parts[0] - function name
+ # parts[1] - total jobs queued (including building)
+ # parts[2] - jobs building
+ # parts[3] - workers registered
+ data = functions.setdefault(parts[0], [0, 0, 0])
+ for i in range(3):
+ data[i] += int(parts[i + 1])
+ return functions
def stop(self):
self.log.debug("Stopping")
@@ -246,3 +271,15 @@
job_log_stream_address['server'] = build.worker.hostname
job_log_stream_address['port'] = build.worker.log_port
job.sendWorkComplete(json.dumps(job_log_stream_address))
+
+ def handle_tenant_list(self, job):
+ output = []
+ for tenant_name, tenant in self.sched.abide.tenants.items():
+ output.append({'name': tenant_name,
+ 'projects': len(tenant.untrusted_projects)})
+ job.sendWorkComplete(json.dumps(output))
+
+ def handle_status_get(self, job):
+ args = json.loads(job.arguments)
+ output = self.sched.formatStatusJSON(args.get("tenant"))
+ job.sendWorkComplete(output)
diff --git a/zuul/scheduler.py b/zuul/scheduler.py
index e5924f8..a725fcd 100644
--- a/zuul/scheduler.py
+++ b/zuul/scheduler.py
@@ -29,8 +29,10 @@
from zuul import model
from zuul import exceptions
from zuul import version as zuul_version
+from zuul import rpclistener
from zuul.lib.config import get_default
from zuul.lib.statsd import get_statsd
+import zuul.lib.queue
class ManagementEvent(object):
@@ -76,8 +78,23 @@
"""
def __init__(self, tenant, project):
super(TenantReconfigureEvent, self).__init__()
- self.tenant = tenant
- self.project = project
+ self.tenant_name = tenant.name
+ self.projects = set([project])
+
+ def __ne__(self, other):
+ return not self.__eq__(other)
+
+ def __eq__(self, other):
+ if not isinstance(other, TenantReconfigureEvent):
+ return False
+ # We don't check projects because they will get combined when
+ # merged.
+ return (self.tenant_name == other.tenant_name)
+
+ def merge(self, other):
+ if self.tenant_name != other.tenant_name:
+ raise Exception("Can not merge events from different tenants")
+ self.projects |= other.projects
class PromoteEvent(ManagementEvent):
@@ -197,6 +214,7 @@
"""
log = logging.getLogger("zuul.Scheduler")
+ _stats_interval = 30
def __init__(self, config, testonly=False):
threading.Thread.__init__(self)
@@ -212,6 +230,9 @@
self.merger = None
self.connections = None
self.statsd = get_statsd(config)
+ self.rpc = rpclistener.RPCListener(config, self)
+ self.stats_thread = threading.Thread(target=self.runStats)
+ self.stats_stop = threading.Event()
# TODO(jeblair): fix this
# Despite triggers being part of the pipeline, there is one trigger set
# per scheduler. The pipeline handles the trigger filters but since
@@ -223,7 +244,7 @@
self.trigger_event_queue = queue.Queue()
self.result_event_queue = queue.Queue()
- self.management_event_queue = queue.Queue()
+ self.management_event_queue = zuul.lib.queue.MergedQueue()
self.abide = model.Abide()
if not testonly:
@@ -235,10 +256,19 @@
self.tenant_last_reconfigured = {}
self.autohold_requests = {}
+ def start(self):
+ super(Scheduler, self).start()
+ self.rpc.start()
+ self.stats_thread.start()
+
def stop(self):
self._stopped = True
+ self.stats_stop.set()
self.stopConnections()
self.wake_event.set()
+ self.stats_thread.join()
+ self.rpc.stop()
+ self.rpc.join()
def registerConnections(self, connections, webapp, load=True):
# load: whether or not to trigger the onLoad for the connection. This
@@ -262,6 +292,44 @@
def setZooKeeper(self, zk):
self.zk = zk
+ def runStats(self):
+ while not self.stats_stop.wait(self._stats_interval):
+ try:
+ self._runStats()
+ except Exception:
+ self.log.exception("Error in periodic stats:")
+
+ def _runStats(self):
+ if not self.statsd:
+ return
+ functions = self.rpc.getFunctions()
+ executors_accepting = 0
+ executors_online = 0
+ execute_queue = 0
+ execute_running = 0
+ mergers_online = 0
+ merge_queue = 0
+ merge_running = 0
+ for (name, (queued, running, registered)) in functions.items():
+ if name == 'executor:execute':
+ executors_accepting = registered
+ execute_queue = queued - running
+ execute_running = running
+ if name.startswith('executor:stop'):
+ executors_online += registered
+ if name == 'merger:merge':
+ mergers_online = registered
+ if name.startswith('merger:'):
+ merge_queue += queued - running
+ merge_running += running
+ self.statsd.gauge('zuul.mergers.online', mergers_online)
+ self.statsd.gauge('zuul.mergers.jobs_running', merge_running)
+ self.statsd.gauge('zuul.mergers.jobs_queued', merge_queue)
+ self.statsd.gauge('zuul.executors.online', executors_online)
+ self.statsd.gauge('zuul.executors.accepting', executors_accepting)
+ self.statsd.gauge('zuul.executors.jobs_running', execute_running)
+ self.statsd.gauge('zuul.executors.jobs_queued', execute_queue)
+
def addEvent(self, event):
self.trigger_event_queue.put(event)
self.wake_event.set()
@@ -306,9 +374,10 @@
self.statsd.incr(key)
# zuul.tenant.<tenant>.pipeline.<pipeline>.project.
# <host>.<project>.<branch>.job.<job>.wait_time
- key = '%s.wait_time' % jobkey
- dt = int((build.start_time - build.execute_time) * 1000)
- self.statsd.timing(key, dt)
+ if build.start_time:
+ key = '%s.wait_time' % jobkey
+ dt = int((build.start_time - build.execute_time) * 1000)
+ self.statsd.timing(key, dt)
except Exception:
self.log.exception("Exception reporting runtime stats")
event = BuildCompletedEvent(build)
@@ -433,11 +502,11 @@
def resume(self):
try:
self._load_queue()
- except:
+ except Exception:
self.log.exception("Unable to load queue")
try:
self._delete_queue()
- except:
+ except Exception:
self.log.exception("Unable to delete saved queue")
self.log.debug("Resuming queue processing")
self.wake_event.set()
@@ -475,15 +544,16 @@
self.log.debug("Tenant reconfiguration beginning")
# If a change landed to a project, clear out the cached
# config before reconfiguring.
- if event.project:
- event.project.unparsed_config = None
+ for project in event.projects:
+ project.unparsed_config = None
+ old_tenant = self.abide.tenants[event.tenant_name]
loader = configloader.ConfigLoader()
abide = loader.reloadTenant(
self.config.get('scheduler', 'tenant_config'),
self._get_project_key_dir(),
self, self.merger, self.connections,
- self.abide, event.tenant)
- tenant = abide.tenants[event.tenant.name]
+ self.abide, old_tenant)
+ tenant = abide.tenants[event.tenant_name]
self._reconfigureTenant(tenant)
self.abide = abide
finally:
@@ -544,13 +614,22 @@
item.queue = None
item.change.project = self._reenqueueGetProject(
tenant, item)
+ # If the old item ahead made it in, re-enqueue
+ # this one behind it.
+ if item.item_ahead in items_to_remove:
+ old_item_ahead = None
+ item_ahead_valid = False
+ else:
+ old_item_ahead = item.item_ahead
+ item_ahead_valid = True
item.item_ahead = None
item.items_behind = []
reenqueued = False
if item.change.project:
try:
reenqueued = new_pipeline.manager.reEnqueueItem(
- item, last_head)
+ item, last_head, old_item_ahead,
+ item_ahead_valid=item_ahead_valid)
except Exception:
self.log.exception(
"Exception while re-enqueing item %s",
@@ -610,8 +689,10 @@
try:
for pipeline in tenant.layout.pipelines.values():
items = len(pipeline.getAllItems())
- # stats.gauges.zuul.pipeline.NAME.current_changes
- key = 'zuul.pipeline.%s' % pipeline.name
+ # stats.gauges.zuul.tenant.<tenant>.pipeline.
+ # <pipeline>.current_changes
+ key = 'zuul.tenant.%s.pipeline.%s' % (
+ tenant.name, pipeline.name)
self.statsd.gauge(key + '.current_changes', items)
except Exception:
self.log.exception("Exception reporting initial "
@@ -851,16 +932,19 @@
autohold_key = (build.pipeline.layout.tenant.name,
build.build_set.item.change.project.canonical_name,
build.job.name)
-
- try:
- self.nodepool.holdNodeSet(nodeset, autohold_key)
- except Exception:
- self.log.exception("Unable to process autohold for %s:",
- autohold_key)
- if autohold_key in self.autohold_requests:
- self.log.debug("Removing autohold %s due to exception",
- autohold_key)
- del self.autohold_requests[autohold_key]
+ if (build.result == "FAILURE" and
+ autohold_key in self.autohold_requests):
+ # We explicitly only want to hold nodes for jobs if they have
+ # failed and have an autohold request.
+ try:
+ self.nodepool.holdNodeSet(nodeset, autohold_key)
+ except Exception:
+ self.log.exception("Unable to process autohold for %s:",
+ autohold_key)
+ if autohold_key in self.autohold_requests:
+ self.log.debug("Removing autohold %s due to exception",
+ autohold_key)
+ del self.autohold_requests[autohold_key]
self.nodepool.returnNodeSet(nodeset)
except Exception:
@@ -939,6 +1023,9 @@
data['result_event_queue'] = {}
data['result_event_queue']['length'] = \
self.result_event_queue.qsize()
+ data['management_event_queue'] = {}
+ data['management_event_queue']['length'] = \
+ self.management_event_queue.qsize()
if self.last_reconfigured:
data['last_reconfigured'] = self.last_reconfigured * 1000
diff --git a/zuul/web/__init__.py b/zuul/web/__init__.py
index 89f5efe..766a21d 100755
--- a/zuul/web/__init__.py
+++ b/zuul/web/__init__.py
@@ -19,6 +19,7 @@
import json
import logging
import os
+import time
import uvloop
import aiohttp
@@ -148,26 +149,93 @@
return ws
+class GearmanHandler(object):
+ log = logging.getLogger("zuul.web.GearmanHandler")
+
+ # Tenant status cache expiry
+ cache_expiry = 1
+
+ def __init__(self, rpc):
+ self.rpc = rpc
+ self.cache = {}
+ self.cache_time = {}
+ self.controllers = {
+ 'tenant_list': self.tenant_list,
+ 'status_get': self.status_get,
+ }
+
+ def tenant_list(self, request):
+ job = self.rpc.submitJob('zuul:tenant_list', {})
+ return web.json_response(json.loads(job.data[0]))
+
+ def status_get(self, request):
+ tenant = request.match_info["tenant"]
+ if tenant not in self.cache or \
+ (time.time() - self.cache_time[tenant]) > self.cache_expiry:
+ job = self.rpc.submitJob('zuul:status_get', {'tenant': tenant})
+ self.cache[tenant] = json.loads(job.data[0])
+ self.cache_time[tenant] = time.time()
+ resp = web.json_response(self.cache[tenant])
+ resp.headers['Access-Control-Allow-Origin'] = '*'
+ resp.headers["Cache-Control"] = "public, max-age=%d" % \
+ self.cache_expiry
+ resp.last_modified = self.cache_time[tenant]
+ return resp
+
+ async def processRequest(self, request, action):
+ try:
+ resp = self.controllers[action](request)
+ except asyncio.CancelledError:
+ self.log.debug("request handling cancelled")
+ except Exception as e:
+ self.log.exception("exception:")
+ resp = web.json_response({'error_description': 'Internal error'},
+ status=500)
+ return resp
+
+
class ZuulWeb(object):
log = logging.getLogger("zuul.web.ZuulWeb")
def __init__(self, listen_address, listen_port,
gear_server, gear_port,
- ssl_key=None, ssl_cert=None, ssl_ca=None):
+ ssl_key=None, ssl_cert=None, ssl_ca=None,
+ static_cache_expiry=3600):
self.listen_address = listen_address
self.listen_port = listen_port
self.event_loop = None
self.term = None
+ self.static_cache_expiry = static_cache_expiry
# instanciate handlers
self.rpc = zuul.rpcclient.RPCClient(gear_server, gear_port,
ssl_key, ssl_cert, ssl_ca)
self.log_streaming_handler = LogStreamingHandler(self.rpc)
+ self.gearman_handler = GearmanHandler(self.rpc)
async def _handleWebsocket(self, request):
return await self.log_streaming_handler.processRequest(
request)
+ async def _handleTenantsRequest(self, request):
+ return await self.gearman_handler.processRequest(request,
+ 'tenant_list')
+
+ async def _handleStatusRequest(self, request):
+ return await self.gearman_handler.processRequest(request, 'status_get')
+
+ async def _handleStaticRequest(self, request):
+ fp = None
+ if request.path.endswith("tenants.html") or request.path.endswith("/"):
+ fp = os.path.join(STATIC_DIR, "index.html")
+ elif request.path.endswith("status.html"):
+ fp = os.path.join(STATIC_DIR, "status.html")
+ headers = {}
+ if self.static_cache_expiry:
+ headers['Cache-Control'] = "public, max-age=%d" % \
+ self.static_cache_expiry
+ return web.FileResponse(fp, headers=headers)
+
def run(self, loop=None):
"""
Run the websocket daemon.
@@ -181,6 +249,11 @@
"""
routes = [
('GET', '/console-stream', self._handleWebsocket),
+ ('GET', '/tenants.json', self._handleTenantsRequest),
+ ('GET', '/{tenant}/status.json', self._handleStatusRequest),
+ ('GET', '/{tenant}/status.html', self._handleStaticRequest),
+ ('GET', '/tenants.html', self._handleStaticRequest),
+ ('GET', '/', self._handleStaticRequest),
]
self.log.debug("ZuulWeb starting")
diff --git a/zuul/web/static/README b/zuul/web/static/README
new file mode 100644
index 0000000..f17ea5f
--- /dev/null
+++ b/zuul/web/static/README
@@ -0,0 +1,62 @@
+External requirements needs to be installed in these locations
+* /static/js/angular.min.js
+* /static/js/jquery.min.js
+* /static/js/jquery-visibility.min.js
+* /static/js/jquery.graphite.min.js
+* /static/bootstrap/css/bootstrap.min.css
+
+
+Use python2-rjsmin or another js minifier:
+```
+DEST_DIR=/var/www/html/static/
+mkdir -p $DEST_DIR/js
+echo "Fetching angular..."
+curl -L --silent https://ajax.googleapis.com/ajax/libs/angularjs/1.5.6/angular.min.js > $DEST_DIR/js/angular.min.js
+
+echo "Fetching jquery..."
+curl -L --silent http://code.jquery.com/jquery.min.js > $DEST_DIR/js/jquery.min.js
+
+echo "Fetching jquery-visibility..."
+curl -L --silent https://raw.githubusercontent.com/mathiasbynens/jquery-visibility/master/jquery-visibility.js > $DEST_DIR/js/jquery-visibility.js
+python2 -mrjsmin < $DEST_DIR/js/jquery-visibility.js > $DEST_DIR/js/jquery-visibility.min.js
+
+echo "Fetching bootstrap..."
+curl -L --silent https://github.com/twbs/bootstrap/releases/download/v3.1.1/bootstrap-3.1.1-dist.zip > bootstrap.zip
+unzip -q -o bootstrap.zip -d $DEST_DIR/
+mv $DEST_DIR/bootstrap-3.1.1-dist $DEST_DIR/bootstrap
+rm -f bootstrap.zip
+
+echo "Fetching jquery-graphite..."
+curl -L --silent https://github.com/prestontimmons/graphitejs/archive/master.zip > jquery-graphite.zip
+unzip -q -o jquery-graphite.zip -d $DEST_DIR/
+python2 -mrjsmin < $DEST_DIR/graphitejs-master/jquery.graphite.js > $DEST_DIR/js/jquery.graphite.min.js
+rm -Rf jquery-graphite.zip $DEST_DIR/graphitejs-master
+```
+
+
+Here is an example apache vhost configuration:
+<VirtualHost zuul-web.example.com:80>
+ DocumentRoot /var/www/zuul-web
+
+ LogLevel warn
+
+ Alias "/static" "/var/www/zuul-web"
+ AliasMatch "^/.*/(.*).html" "/var/www/zuul-web/$1.html"
+ AliasMatch "^/.*.html" "/var/www/zuul-web/index.html"
+ <Directory /var/www/zuul-web>
+ Require all granted
+ Order allow,deny
+ Allow from all
+ </Directory>
+
+ # Console-stream needs a special proxy-pass for websocket
+ ProxyPass /console-stream ws://localhost:9000/console-stream nocanon retry=0
+ ProxyPassReverse /console-stream ws://localhost:9000/console-stream
+
+ # Then only the json calls are sent to the zuul-web endpoints
+ ProxyPassMatch ^/(.*.json)$ http://localhost:9000/$1 nocanon retry=0
+ ProxyPassReverse / http://localhost:9000/
+</VirtualHost>
+
+Then copy the zuul/web/static/ files and external requirements to
+/var/www/zuul-web
diff --git a/zuul/web/static/images/black.png b/zuul/web/static/images/black.png
new file mode 100644
index 0000000..252d874
--- /dev/null
+++ b/zuul/web/static/images/black.png
Binary files differ
diff --git a/zuul/web/static/images/green.png b/zuul/web/static/images/green.png
new file mode 100644
index 0000000..a8765f1
--- /dev/null
+++ b/zuul/web/static/images/green.png
Binary files differ
diff --git a/zuul/web/static/images/grey.png b/zuul/web/static/images/grey.png
new file mode 100644
index 0000000..eaee0d7
--- /dev/null
+++ b/zuul/web/static/images/grey.png
Binary files differ
diff --git a/zuul/web/static/images/line-angle.png b/zuul/web/static/images/line-angle.png
new file mode 100644
index 0000000..fa74868
--- /dev/null
+++ b/zuul/web/static/images/line-angle.png
Binary files differ
diff --git a/zuul/web/static/images/line-t.png b/zuul/web/static/images/line-t.png
new file mode 100644
index 0000000..cfd3111
--- /dev/null
+++ b/zuul/web/static/images/line-t.png
Binary files differ
diff --git a/zuul/web/static/images/line.png b/zuul/web/static/images/line.png
new file mode 100644
index 0000000..ace6bab
--- /dev/null
+++ b/zuul/web/static/images/line.png
Binary files differ
diff --git a/zuul/web/static/images/red.png b/zuul/web/static/images/red.png
new file mode 100644
index 0000000..e9956e8
--- /dev/null
+++ b/zuul/web/static/images/red.png
Binary files differ
diff --git a/zuul/web/static/index.html b/zuul/web/static/index.html
new file mode 100644
index 0000000..6747e66
--- /dev/null
+++ b/zuul/web/static/index.html
@@ -0,0 +1,57 @@
+<!--
+Copyright 2017 Red Hat
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may
+not use this file except in compliance with the License. You may obtain
+a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+License for the specific language governing permissions and limitations
+under the License.
+-->
+<!DOCTYPE html>
+<html>
+<head>
+ <title>Zuul Tenants</title>
+ <link rel="stylesheet" href="/static/bootstrap/css/bootstrap.min.css">
+ <link rel="stylesheet" href="static/styles/zuul.css" />
+ <script src="/static/js/jquery.min.js"></script>
+ <script src="/static/js/angular.min.js"></script>
+ <script src="static/javascripts/zuul.angular.js"></script>
+</head>
+<body ng-app="zuulTenants" ng-controller="mainController"><div class="container-fluid">
+ <nav class="navbar navbar-default">
+ <div class="container-fluid">
+ <div class="navbar-header">
+ <a class="navbar-brand">Zuul Dashboard</a>
+ </div>
+ <ul class="nav navbar-nav">
+ <li class="active"><a href="tenants.html">Tenants</a></li>
+ </ul>
+ </div>
+ </nav>
+ <table class="table table-hover table-condensed">
+ <thead>
+ <tr>
+ <th>Name</th>
+ <th>Status</th>
+ <th>Jobs</th>
+ <th>Builds</th>
+ <th>Projects count</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr ng-repeat="tenant in tenants">
+ <td>{{ tenant.name }}</td>
+ <td><a href="{{ tenant.name }}/status.html">status</a></td>
+ <td><a href="{{ tenant.name }}/jobs.html">jobs</a></td>
+ <td><a href="{{ tenant.name }}/builds.html">builds</a></td>
+ <td>{{ tenant.projects }}</td>
+ </tr>
+ </tbody>
+ </table>
+</div></body></html>
diff --git a/zuul/web/static/javascripts/jquery.zuul.js b/zuul/web/static/javascripts/jquery.zuul.js
new file mode 100644
index 0000000..7e6788b
--- /dev/null
+++ b/zuul/web/static/javascripts/jquery.zuul.js
@@ -0,0 +1,945 @@
+// jquery plugin for Zuul status page
+//
+// @licstart The following is the entire license notice for the
+// JavaScript code in this page.
+//
+// Copyright 2012 OpenStack Foundation
+// Copyright 2013 Timo Tijhof
+// Copyright 2013 Wikimedia Foundation
+// Copyright 2014 Rackspace Australia
+//
+// Licensed under the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License. You may obtain
+// a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+// License for the specific language governing permissions and limitations
+// under the License.
+//
+// @licend The above is the entire license notice
+// for the JavaScript code in this page.
+
+(function ($) {
+ 'use strict';
+
+ function set_cookie(name, value) {
+ document.cookie = name + '=' + value + '; path=/';
+ }
+
+ function read_cookie(name, default_value) {
+ var nameEQ = name + '=';
+ var ca = document.cookie.split(';');
+ for(var i=0;i < ca.length;i++) {
+ var c = ca[i];
+ while (c.charAt(0) === ' ') {
+ c = c.substring(1, c.length);
+ }
+ if (c.indexOf(nameEQ) === 0) {
+ return c.substring(nameEQ.length, c.length);
+ }
+ }
+ return default_value;
+ }
+
+ $.zuul = function(options) {
+ options = $.extend({
+ 'enabled': true,
+ 'graphite_url': '',
+ 'source': 'status.json',
+ 'msg_id': '#zuul_msg',
+ 'pipelines_id': '#zuul_pipelines',
+ 'queue_events_num': '#zuul_queue_events_num',
+ 'queue_management_events_num': '#zuul_queue_management_events_num',
+ 'queue_results_num': '#zuul_queue_results_num',
+ }, options);
+
+ var collapsed_exceptions = [];
+ var current_filter = read_cookie('zuul_filter_string', '');
+ var change_set_in_url = window.location.href.split('#')[1];
+ if (change_set_in_url) {
+ current_filter = change_set_in_url;
+ }
+ var $jq;
+
+ var xhr,
+ zuul_graph_update_count = 0,
+ zuul_sparkline_urls = {};
+
+ function get_sparkline_url(pipeline_name) {
+ if (options.graphite_url !== '') {
+ if (!(pipeline_name in zuul_sparkline_urls)) {
+ zuul_sparkline_urls[pipeline_name] = $.fn.graphite
+ .geturl({
+ url: options.graphite_url,
+ from: "-8hours",
+ width: 100,
+ height: 26,
+ margin: 0,
+ hideLegend: true,
+ hideAxes: true,
+ hideGrid: true,
+ target: [
+ "color(stats.gauges.zuul.pipeline." + pipeline_name
+ + ".current_changes, '6b8182')"
+ ]
+ });
+ }
+ return zuul_sparkline_urls[pipeline_name];
+ }
+ return false;
+ }
+
+ var format = {
+ job: function(job) {
+ var $job_line = $('<span />');
+
+ if (job.result !== null) {
+ $job_line.append(
+ $('<a />')
+ .addClass('zuul-job-name')
+ .attr('href', job.report_url)
+ .text(job.name)
+ );
+ }
+ else if (job.url !== null) {
+ $job_line.append(
+ $('<a />')
+ .addClass('zuul-job-name')
+ .attr('href', job.url)
+ .text(job.name)
+ );
+ }
+ else {
+ $job_line.append(
+ $('<span />')
+ .addClass('zuul-job-name')
+ .text(job.name)
+ );
+ }
+
+ $job_line.append(this.job_status(job));
+
+ if (job.voting === false) {
+ $job_line.append(
+ $(' <small />')
+ .addClass('zuul-non-voting-desc')
+ .text(' (non-voting)')
+ );
+ }
+
+ $job_line.append($('<div style="clear: both"></div>'));
+ return $job_line;
+ },
+
+ job_status: function(job) {
+ var result = job.result ? job.result.toLowerCase() : null;
+ if (result === null) {
+ result = job.url ? 'in progress' : 'queued';
+ }
+
+ if (result === 'in progress') {
+ return this.job_progress_bar(job.elapsed_time,
+ job.remaining_time);
+ }
+ else {
+ return this.status_label(result);
+ }
+ },
+
+ status_label: function(result) {
+ var $status = $('<span />');
+ $status.addClass('zuul-job-result label');
+
+ switch (result) {
+ case 'success':
+ $status.addClass('label-success');
+ break;
+ case 'failure':
+ $status.addClass('label-danger');
+ break;
+ case 'unstable':
+ $status.addClass('label-warning');
+ break;
+ case 'skipped':
+ $status.addClass('label-info');
+ break;
+ // 'in progress' 'queued' 'lost' 'aborted' ...
+ default:
+ $status.addClass('label-default');
+ }
+ $status.text(result);
+ return $status;
+ },
+
+ job_progress_bar: function(elapsed_time, remaining_time) {
+ var progress_percent = 100 * (elapsed_time / (elapsed_time +
+ remaining_time));
+ var $bar_inner = $('<div />')
+ .addClass('progress-bar')
+ .attr('role', 'progressbar')
+ .attr('aria-valuenow', 'progressbar')
+ .attr('aria-valuemin', progress_percent)
+ .attr('aria-valuemin', '0')
+ .attr('aria-valuemax', '100')
+ .css('width', progress_percent + '%');
+
+ var $bar_outter = $('<div />')
+ .addClass('progress zuul-job-result')
+ .append($bar_inner);
+
+ return $bar_outter;
+ },
+
+ enqueue_time: function(ms) {
+ // Special format case for enqueue time to add style
+ var hours = 60 * 60 * 1000;
+ var now = Date.now();
+ var delta = now - ms;
+ var status = 'text-success';
+ var text = this.time(delta, true);
+ if (delta > (4 * hours)) {
+ status = 'text-danger';
+ } else if (delta > (2 * hours)) {
+ status = 'text-warning';
+ }
+ return '<span class="' + status + '">' + text + '</span>';
+ },
+
+ time: function(ms, words) {
+ if (typeof(words) === 'undefined') {
+ words = false;
+ }
+ var seconds = (+ms)/1000;
+ var minutes = Math.floor(seconds/60);
+ var hours = Math.floor(minutes/60);
+ seconds = Math.floor(seconds % 60);
+ minutes = Math.floor(minutes % 60);
+ var r = '';
+ if (words) {
+ if (hours) {
+ r += hours;
+ r += ' hr ';
+ }
+ r += minutes + ' min';
+ } else {
+ if (hours < 10) {
+ r += '0';
+ }
+ r += hours + ':';
+ if (minutes < 10) {
+ r += '0';
+ }
+ r += minutes + ':';
+ if (seconds < 10) {
+ r += '0';
+ }
+ r += seconds;
+ }
+ return r;
+ },
+
+ change_total_progress_bar: function(change) {
+ var job_percent = Math.floor(100 / change.jobs.length);
+ var $bar_outter = $('<div />')
+ .addClass('progress zuul-change-total-result');
+
+ $.each(change.jobs, function (i, job) {
+ var result = job.result ? job.result.toLowerCase() : null;
+ if (result === null) {
+ result = job.url ? 'in progress' : 'queued';
+ }
+
+ if (result !== 'queued') {
+ var $bar_inner = $('<div />')
+ .addClass('progress-bar');
+
+ switch (result) {
+ case 'success':
+ $bar_inner.addClass('progress-bar-success');
+ break;
+ case 'lost':
+ case 'failure':
+ $bar_inner.addClass('progress-bar-danger');
+ break;
+ case 'unstable':
+ $bar_inner.addClass('progress-bar-warning');
+ break;
+ case 'in progress':
+ case 'queued':
+ break;
+ }
+ $bar_inner.attr('title', job.name)
+ .css('width', job_percent + '%');
+ $bar_outter.append($bar_inner);
+ }
+ });
+ return $bar_outter;
+ },
+
+ change_header: function(change) {
+ var change_id = change.id || 'NA';
+
+ var $change_link = $('<small />');
+ if (change.url !== null) {
+ var github_id = change_id.match(/^([0-9]+),([0-9a-f]{40})$/);
+ if (github_id) {
+ $change_link.append(
+ $('<a />').attr('href', change.url).append(
+ $('<abbr />')
+ .attr('title', change_id)
+ .text('#' + github_id[1])
+ )
+ );
+ } else if (/^[0-9a-f]{40}$/.test(change_id)) {
+ var change_id_short = change_id.slice(0, 7);
+ $change_link.append(
+ $('<a />').attr('href', change.url).append(
+ $('<abbr />')
+ .attr('title', change_id)
+ .text(change_id_short)
+ )
+ );
+ }
+ else {
+ $change_link.append(
+ $('<a />').attr('href', change.url).text(change_id)
+ );
+ }
+ }
+ else {
+ if (change_id.length === 40) {
+ change_id = change_id.substr(0, 7);
+ }
+ $change_link.text(change_id);
+ }
+
+ var $change_progress_row_left = $('<div />')
+ .addClass('col-xs-4')
+ .append($change_link);
+ var $change_progress_row_right = $('<div />')
+ .addClass('col-xs-8')
+ .append(this.change_total_progress_bar(change));
+
+ var $change_progress_row = $('<div />')
+ .addClass('row')
+ .append($change_progress_row_left)
+ .append($change_progress_row_right);
+
+ var $project_span = $('<span />')
+ .addClass('change_project')
+ .text(change.project);
+
+ var $left = $('<div />')
+ .addClass('col-xs-8')
+ .append($project_span, $change_progress_row);
+
+ var remaining_time = this.time(
+ change.remaining_time, true);
+ var enqueue_time = this.enqueue_time(
+ change.enqueue_time);
+ var $remaining_time = $('<small />').addClass('time')
+ .attr('title', 'Remaining Time').html(remaining_time);
+ var $enqueue_time = $('<small />').addClass('time')
+ .attr('title', 'Elapsed Time').html(enqueue_time);
+
+ var $right = $('<div />');
+ if (change.live === true) {
+ $right.addClass('col-xs-4 text-right')
+ .append($remaining_time, $('<br />'), $enqueue_time);
+ }
+
+ var $header = $('<div />')
+ .addClass('row')
+ .append($left, $right);
+ return $header;
+ },
+
+ change_list: function(jobs) {
+ var format = this;
+ var $list = $('<ul />')
+ .addClass('list-group zuul-patchset-body');
+
+ $.each(jobs, function (i, job) {
+ var $item = $('<li />')
+ .addClass('list-group-item')
+ .addClass('zuul-change-job')
+ .append(format.job(job));
+ $list.append($item);
+ });
+
+ return $list;
+ },
+
+ change_panel: function (change) {
+ var $header = $('<div />')
+ .addClass('panel-heading zuul-patchset-header')
+ .append(this.change_header(change));
+
+ var panel_id = change.id ? change.id.replace(',', '_')
+ : change.project.replace('/', '_') +
+ '-' + change.enqueue_time;
+ var $panel = $('<div />')
+ .attr('id', panel_id)
+ .addClass('panel panel-default zuul-change')
+ .append($header)
+ .append(this.change_list(change.jobs));
+
+ $header.click(this.toggle_patchset);
+ return $panel;
+ },
+
+ change_status_icon: function(change) {
+ var icon_name = 'green.png';
+ var icon_title = 'Succeeding';
+
+ if (change.active !== true) {
+ // Grey icon
+ icon_name = 'grey.png';
+ icon_title = 'Waiting until closer to head of queue to' +
+ ' start jobs';
+ }
+ else if (change.live !== true) {
+ // Grey icon
+ icon_name = 'grey.png';
+ icon_title = 'Dependent change required for testing';
+ }
+ else if (change.failing_reasons &&
+ change.failing_reasons.length > 0) {
+ var reason = change.failing_reasons.join(', ');
+ icon_title = 'Failing because ' + reason;
+ if (reason.match(/merge conflict/)) {
+ // Black icon
+ icon_name = 'black.png';
+ }
+ else {
+ // Red icon
+ icon_name = 'red.png';
+ }
+ }
+
+ var $icon = $('<img />')
+ .attr('src', '../static/images/' + icon_name)
+ .attr('title', icon_title)
+ .css('margin-top', '-6px');
+
+ return $icon;
+ },
+
+ change_with_status_tree: function(change, change_queue) {
+ var $change_row = $('<tr />');
+
+ for (var i = 0; i < change_queue._tree_columns; i++) {
+ var $tree_cell = $('<td />')
+ .css('height', '100%')
+ .css('padding', '0 0 10px 0')
+ .css('margin', '0')
+ .css('width', '16px')
+ .css('min-width', '16px')
+ .css('overflow', 'hidden')
+ .css('vertical-align', 'top');
+
+ if (i < change._tree.length && change._tree[i] !== null) {
+ $tree_cell.css('background-image',
+ 'url(\'../static/images/line.png\')')
+ .css('background-repeat', 'repeat-y');
+ }
+
+ if (i === change._tree_index) {
+ $tree_cell.append(
+ this.change_status_icon(change));
+ }
+ if (change._tree_branches.indexOf(i) !== -1) {
+ var $image = $('<img />')
+ .css('vertical-align', 'baseline');
+ if (change._tree_branches.indexOf(i) ===
+ change._tree_branches.length - 1) {
+ // Angle line
+ $image.attr('src', '../static/images/line-angle.png');
+ }
+ else {
+ // T line
+ $image.attr('src', '../static/images/line-t.png');
+ }
+ $tree_cell.append($image);
+ }
+ $change_row.append($tree_cell);
+ }
+
+ var change_width = 360 - 16*change_queue._tree_columns;
+ var $change_column = $('<td />')
+ .css('width', change_width + 'px')
+ .addClass('zuul-change-cell')
+ .append(this.change_panel(change));
+
+ $change_row.append($change_column);
+
+ var $change_table = $('<table />')
+ .addClass('zuul-change-box')
+ .css('-moz-box-sizing', 'content-box')
+ .css('box-sizing', 'content-box')
+ .append($change_row);
+
+ return $change_table;
+ },
+
+ pipeline_sparkline: function(pipeline_name) {
+ if (options.graphite_url !== '') {
+ var $sparkline = $('<img />')
+ .addClass('pull-right')
+ .attr('src', get_sparkline_url(pipeline_name));
+ return $sparkline;
+ }
+ return false;
+ },
+
+ pipeline_header: function(pipeline, count) {
+ // Format the pipeline name, sparkline and description
+ var $header_div = $('<div />')
+ .addClass('zuul-pipeline-header');
+
+ var $heading = $('<h3 />')
+ .css('vertical-align', 'middle')
+ .text(pipeline.name)
+ .append(
+ $('<span />')
+ .addClass('badge pull-right')
+ .css('vertical-align', 'middle')
+ .css('margin-top', '0.5em')
+ .text(count)
+ )
+ .append(this.pipeline_sparkline(pipeline.name));
+
+ $header_div.append($heading);
+
+ if (typeof pipeline.description === 'string') {
+ var descr = $('<small />')
+ $.each( pipeline.description.split(/\r?\n\r?\n/), function(index, descr_part){
+ descr.append($('<p />').text(descr_part));
+ });
+ $header_div.append(
+ $('<p />').append(descr)
+ );
+ }
+ return $header_div;
+ },
+
+ pipeline: function (pipeline, count) {
+ var format = this;
+ var $html = $('<div />')
+ .addClass('zuul-pipeline col-md-4')
+ .append(this.pipeline_header(pipeline, count));
+
+ $.each(pipeline.change_queues,
+ function (queue_i, change_queue) {
+ $.each(change_queue.heads, function (head_i, changes) {
+ if (pipeline.change_queues.length > 1 &&
+ head_i === 0) {
+ var name = change_queue.name;
+ var short_name = name;
+ if (short_name.length > 32) {
+ short_name = short_name.substr(0, 32) + '...';
+ }
+ $html.append(
+ $('<p />')
+ .text('Queue: ')
+ .append(
+ $('<abbr />')
+ .attr('title', name)
+ .text(short_name)
+ )
+ );
+ }
+
+ $.each(changes, function (change_i, change) {
+ var $change_box =
+ format.change_with_status_tree(
+ change, change_queue);
+ $html.append($change_box);
+ format.display_patchset($change_box);
+ });
+ });
+ });
+ return $html;
+ },
+
+ toggle_patchset: function(e) {
+ // Toggle showing/hiding the patchset when the header is
+ // clicked.
+
+ if (e.target.nodeName.toLowerCase() === 'a') {
+ // Ignore clicks from gerrit patch set link
+ return;
+ }
+
+ // Grab the patchset panel
+ var $panel = $(e.target).parents('.zuul-change');
+ var $body = $panel.children('.zuul-patchset-body');
+ $body.toggle(200);
+ var collapsed_index = collapsed_exceptions.indexOf(
+ $panel.attr('id'));
+ if (collapsed_index === -1 ) {
+ // Currently not an exception, add it to list
+ collapsed_exceptions.push($panel.attr('id'));
+ }
+ else {
+ // Currently an except, remove from exceptions
+ collapsed_exceptions.splice(collapsed_index, 1);
+ }
+ },
+
+ display_patchset: function($change_box, animate) {
+ // Determine if to show or hide the patchset and/or the results
+ // when loaded
+
+ // See if we should hide the body/results
+ var $panel = $change_box.find('.zuul-change');
+ var panel_change = $panel.attr('id');
+ var $body = $panel.children('.zuul-patchset-body');
+ var expand_by_default = $('#expand_by_default')
+ .prop('checked');
+
+ var collapsed_index = collapsed_exceptions
+ .indexOf(panel_change);
+
+ if (expand_by_default && collapsed_index === -1 ||
+ !expand_by_default && collapsed_index !== -1) {
+ // Expand by default, or is an exception
+ $body.show(animate);
+ }
+ else {
+ $body.hide(animate);
+ }
+
+ // Check if we should hide the whole panel
+ var panel_project = $panel.find('.change_project').text()
+ .toLowerCase();
+
+
+ var panel_pipeline = $change_box
+ .parents('.zuul-pipeline')
+ .find('.zuul-pipeline-header > h3')
+ .html()
+ .toLowerCase();
+
+ if (current_filter !== '') {
+ var show_panel = false;
+ var filter = current_filter.trim().split(/[\s,]+/);
+ $.each(filter, function(index, f_val) {
+ if (f_val !== '') {
+ f_val = f_val.toLowerCase();
+ if (panel_project.indexOf(f_val) !== -1 ||
+ panel_pipeline.indexOf(f_val) !== -1 ||
+ panel_change.indexOf(f_val) !== -1) {
+ show_panel = true;
+ }
+ }
+ });
+ if (show_panel === true) {
+ $change_box.show(animate);
+ }
+ else {
+ $change_box.hide(animate);
+ }
+ }
+ else {
+ $change_box.show(animate);
+ }
+ },
+ };
+
+ var app = {
+ schedule: function (app) {
+ app = app || this;
+ if (!options.enabled) {
+ setTimeout(function() {app.schedule(app);}, 5000);
+ return;
+ }
+ app.update().always(function () {
+ setTimeout(function() {app.schedule(app);}, 5000);
+ });
+
+ /* Only update graphs every minute */
+ if (zuul_graph_update_count > 11) {
+ zuul_graph_update_count = 0;
+ zuul.update_sparklines();
+ }
+ },
+
+ /** @return {jQuery.Promise} */
+ update: function () {
+ // Cancel the previous update if it hasn't completed yet.
+ if (xhr) {
+ xhr.abort();
+ }
+
+ this.emit('update-start');
+ var app = this;
+
+ var $msg = $(options.msg_id);
+ xhr = $.getJSON(options.source)
+ .done(function (data) {
+ if ('message' in data) {
+ $msg.removeClass('alert-danger')
+ .addClass('alert-info')
+ .text(data.message)
+ .show();
+ } else {
+ $msg.empty()
+ .hide();
+ }
+
+ if ('zuul_version' in data) {
+ $('#zuul-version-span').text(data.zuul_version);
+ }
+ if ('last_reconfigured' in data) {
+ var last_reconfigured =
+ new Date(data.last_reconfigured);
+ $('#last-reconfigured-span').text(
+ last_reconfigured.toString());
+ }
+
+ var $pipelines = $(options.pipelines_id);
+ $pipelines.html('');
+ $.each(data.pipelines, function (i, pipeline) {
+ var count = app.create_tree(pipeline);
+ $pipelines.append(
+ format.pipeline(pipeline, count));
+ });
+
+ $(options.queue_events_num).text(
+ data.trigger_event_queue ?
+ data.trigger_event_queue.length : '0'
+ );
+ $(options.queue_management_events_num).text(
+ data.management_event_queue ?
+ data.management_event_queue.length : '0'
+ );
+ $(options.queue_results_num).text(
+ data.result_event_queue ?
+ data.result_event_queue.length : '0'
+ );
+ })
+ .fail(function (jqXHR, statusText, errMsg) {
+ if (statusText === 'abort') {
+ return;
+ }
+ $msg.text(options.source + ': ' + errMsg)
+ .addClass('alert-danger')
+ .removeClass('zuul-msg-wrap-off')
+ .show();
+ })
+ .always(function () {
+ xhr = undefined;
+ app.emit('update-end');
+ });
+
+ return xhr;
+ },
+
+ update_sparklines: function() {
+ $.each(zuul_sparkline_urls, function(name, url) {
+ var newimg = new Image();
+ var parts = url.split('#');
+ newimg.src = parts[0] + '#' + new Date().getTime();
+ $(newimg).load(function () {
+ zuul_sparkline_urls[name] = newimg.src;
+ });
+ });
+ },
+
+ emit: function () {
+ $jq.trigger.apply($jq, arguments);
+ return this;
+ },
+ on: function () {
+ $jq.on.apply($jq, arguments);
+ return this;
+ },
+ one: function () {
+ $jq.one.apply($jq, arguments);
+ return this;
+ },
+
+ control_form: function() {
+ // Build the filter form filling anything from cookies
+
+ var $control_form = $('<form />')
+ .attr('role', 'form')
+ .addClass('form-inline')
+ .submit(this.handle_filter_change);
+
+ $control_form
+ .append(this.filter_form_group())
+ .append(this.expand_form_group());
+
+ return $control_form;
+ },
+
+ filter_form_group: function() {
+ // Update the filter form with a clear button if required
+
+ var $label = $('<label />')
+ .addClass('control-label')
+ .attr('for', 'filter_string')
+ .text('Filters')
+ .css('padding-right', '0.5em');
+
+ var $input = $('<input />')
+ .attr('type', 'text')
+ .attr('id', 'filter_string')
+ .addClass('form-control')
+ .attr('title',
+ 'project(s), pipeline(s) or review(s) comma ' +
+ 'separated')
+ .attr('value', current_filter);
+
+ $input.change(this.handle_filter_change);
+
+ var $clear_icon = $('<span />')
+ .addClass('form-control-feedback')
+ .addClass('glyphicon glyphicon-remove-circle')
+ .attr('id', 'filter_form_clear_box')
+ .attr('title', 'clear filter')
+ .css('cursor', 'pointer');
+
+ $clear_icon.click(function() {
+ $('#filter_string').val('').change();
+ });
+
+ if (current_filter === '') {
+ $clear_icon.hide();
+ }
+
+ var $form_group = $('<div />')
+ .addClass('form-group has-feedback')
+ .append($label, $input, $clear_icon);
+ return $form_group;
+ },
+
+ expand_form_group: function() {
+ var expand_by_default = (
+ read_cookie('zuul_expand_by_default', false) === 'true');
+
+ var $checkbox = $('<input />')
+ .attr('type', 'checkbox')
+ .attr('id', 'expand_by_default')
+ .prop('checked', expand_by_default)
+ .change(this.handle_expand_by_default);
+
+ var $label = $('<label />')
+ .css('padding-left', '1em')
+ .html('Expand by default: ')
+ .append($checkbox);
+
+ var $form_group = $('<div />')
+ .addClass('checkbox')
+ .append($label);
+ return $form_group;
+ },
+
+ handle_filter_change: function() {
+ // Update the filter and save it to a cookie
+ current_filter = $('#filter_string').val();
+ set_cookie('zuul_filter_string', current_filter);
+ if (current_filter === '') {
+ $('#filter_form_clear_box').hide();
+ }
+ else {
+ $('#filter_form_clear_box').show();
+ }
+
+ $('.zuul-change-box').each(function(index, obj) {
+ var $change_box = $(obj);
+ format.display_patchset($change_box, 200);
+ });
+ return false;
+ },
+
+ handle_expand_by_default: function(e) {
+ // Handle toggling expand by default
+ set_cookie('zuul_expand_by_default', e.target.checked);
+ collapsed_exceptions = [];
+ $('.zuul-change-box').each(function(index, obj) {
+ var $change_box = $(obj);
+ format.display_patchset($change_box, 200);
+ });
+ },
+
+ create_tree: function(pipeline) {
+ var count = 0;
+ var pipeline_max_tree_columns = 1;
+ $.each(pipeline.change_queues, function(change_queue_i,
+ change_queue) {
+ var tree = [];
+ var max_tree_columns = 1;
+ var changes = [];
+ var last_tree_length = 0;
+ $.each(change_queue.heads, function(head_i, head) {
+ $.each(head, function(change_i, change) {
+ changes[change.id] = change;
+ change._tree_position = change_i;
+ });
+ });
+ $.each(change_queue.heads, function(head_i, head) {
+ $.each(head, function(change_i, change) {
+ if (change.live === true) {
+ count += 1;
+ }
+ var idx = tree.indexOf(change.id);
+ if (idx > -1) {
+ change._tree_index = idx;
+ // remove...
+ tree[idx] = null;
+ while (tree[tree.length - 1] === null) {
+ tree.pop();
+ }
+ } else {
+ change._tree_index = 0;
+ }
+ change._tree_branches = [];
+ change._tree = [];
+ if (typeof(change.items_behind) === 'undefined') {
+ change.items_behind = [];
+ }
+ change.items_behind.sort(function(a, b) {
+ return (changes[b]._tree_position -
+ changes[a]._tree_position);
+ });
+ $.each(change.items_behind, function(i, id) {
+ tree.push(id);
+ if (tree.length>last_tree_length &&
+ last_tree_length > 0) {
+ change._tree_branches.push(
+ tree.length - 1);
+ }
+ });
+ if (tree.length > max_tree_columns) {
+ max_tree_columns = tree.length;
+ }
+ if (tree.length > pipeline_max_tree_columns) {
+ pipeline_max_tree_columns = tree.length;
+ }
+ change._tree = tree.slice(0); // make a copy
+ last_tree_length = tree.length;
+ });
+ });
+ change_queue._tree_columns = max_tree_columns;
+ });
+ pipeline._tree_columns = pipeline_max_tree_columns;
+ return count;
+ },
+ };
+
+ $jq = $(app);
+ return {
+ options: options,
+ format: format,
+ app: app,
+ jq: $jq
+ };
+ };
+}(jQuery));
diff --git a/zuul/web/static/javascripts/zuul.angular.js b/zuul/web/static/javascripts/zuul.angular.js
new file mode 100644
index 0000000..3152fc0
--- /dev/null
+++ b/zuul/web/static/javascripts/zuul.angular.js
@@ -0,0 +1,32 @@
+// @licstart The following is the entire license notice for the
+// JavaScript code in this page.
+//
+// Copyright 2017 Red Hat
+//
+// Licensed under the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License. You may obtain
+// a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+// License for the specific language governing permissions and limitations
+// under the License.
+//
+// @licend The above is the entire license notice
+// for the JavaScript code in this page.
+
+angular.module('zuulTenants', []).controller(
+ 'mainController', function($scope, $http)
+{
+ $scope.tenants = undefined;
+ $scope.tenants_fetch = function() {
+ $http.get("tenants.json")
+ .then(function success(result) {
+ $scope.tenants = result.data;
+ });
+ }
+ $scope.tenants_fetch();
+});
diff --git a/zuul/web/static/javascripts/zuul.app.js b/zuul/web/static/javascripts/zuul.app.js
new file mode 100644
index 0000000..7ceb2dd
--- /dev/null
+++ b/zuul/web/static/javascripts/zuul.app.js
@@ -0,0 +1,110 @@
+// Client script for Zuul status page
+//
+// @licstart The following is the entire license notice for the
+// JavaScript code in this page.
+//
+// Copyright 2013 OpenStack Foundation
+// Copyright 2013 Timo Tijhof
+// Copyright 2013 Wikimedia Foundation
+// Copyright 2014 Rackspace Australia
+//
+// Licensed under the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License. You may obtain
+// a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+// License for the specific language governing permissions and limitations
+// under the License.
+//
+// @licend The above is the entire license notice
+// for the JavaScript code in this page.
+
+/*exported zuul_build_dom, zuul_start */
+
+function zuul_build_dom($, container) {
+ // Build a default-looking DOM
+ var default_layout = '<div class="container">'
+ + '<h1>Zuul Status</h1>'
+ + '<p>Real-time status monitor of Zuul, the pipeline manager between Gerrit and Workers.</p>'
+ + '<div class="zuul-container" id="zuul-container">'
+ + '<div style="display: none;" class="alert" id="zuul_msg"></div>'
+ + '<button class="btn pull-right zuul-spinner">updating <span class="glyphicon glyphicon-refresh"></span></button>'
+ + '<p>Queue lengths: <span id="zuul_queue_events_num">0</span> events, <span id="zuul_queue_management_events_num">0</span> management events, <span id="zuul_queue_results_num">0</span> results.</p>'
+ + '<div id="zuul_controls"></div>'
+ + '<div id="zuul_pipelines" class="row"></div>'
+ + '<p>Zuul version: <span id="zuul-version-span"></span></p>'
+ + '<p>Last reconfigured: <span id="last-reconfigured-span"></span></p>'
+ + '</div></div>';
+
+ $(function ($) {
+ // DOM ready
+ var $container = $(container);
+ $container.html(default_layout);
+ });
+}
+
+/**
+ * @return The $.zuul instance
+ */
+function zuul_start($) {
+ // Start the zuul app (expects default dom)
+
+ var $container, $indicator;
+ var demo = location.search.match(/[?&]demo=([^?&]*)/),
+ source_url = location.search.match(/[?&]source_url=([^?&]*)/),
+ source = demo ? './status-' + (demo[1] || 'basic') + '.json-sample' :
+ 'status.json';
+ source = source_url ? source_url[1] : source;
+
+ var zuul = $.zuul({
+ source: source,
+ //graphite_url: 'http://graphite.openstack.org/render/'
+ });
+
+ zuul.jq.on('update-start', function () {
+ $container.addClass('zuul-container-loading');
+ $indicator.addClass('zuul-spinner-on');
+ });
+
+ zuul.jq.on('update-end', function () {
+ $container.removeClass('zuul-container-loading');
+ setTimeout(function () {
+ $indicator.removeClass('zuul-spinner-on');
+ }, 500);
+ });
+
+ zuul.jq.one('update-end', function () {
+ // Do this asynchronous so that if the first update adds a
+ // message, it will not animate while we fade in the content.
+ // Instead it simply appears with the rest of the content.
+ setTimeout(function () {
+ // Fade in the content
+ $container.addClass('zuul-container-ready');
+ });
+ });
+
+ $(function ($) {
+ // DOM ready
+ $container = $('#zuul-container');
+ $indicator = $('#zuul-spinner');
+ $('#zuul_controls').append(zuul.app.control_form());
+
+ zuul.app.schedule();
+
+ $(document).on({
+ 'show.visibility': function () {
+ zuul.options.enabled = true;
+ zuul.app.update();
+ },
+ 'hide.visibility': function () {
+ zuul.options.enabled = false;
+ }
+ });
+ });
+
+ return zuul;
+}
diff --git a/zuul/web/static/status.html b/zuul/web/static/status.html
new file mode 100644
index 0000000..7cb9536
--- /dev/null
+++ b/zuul/web/static/status.html
@@ -0,0 +1,51 @@
+<!--
+Copyright 2013 OpenStack Foundation
+Copyright 2013 Timo Tijhof
+Copyright 2013 Wikimedia Foundation
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may
+not use this file except in compliance with the License. You may obtain
+a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+License for the specific language governing permissions and limitations
+under the License.
+-->
+<!DOCTYPE html>
+<html>
+<head>
+ <title>Zuul Status</title>
+ <link rel="stylesheet" href="/static/bootstrap/css/bootstrap.min.css">
+ <link rel="stylesheet" href="../static/styles/zuul.css" />
+ <script src="/static/js/jquery.min.js"></script>
+ <script src="/static/js/jquery-visibility.min.js"></script>
+ <script src="/static/js/jquery.graphite.min.js"></script>
+ <script src="../static/javascripts/jquery.zuul.js"></script>
+ <script src="../static/javascripts/zuul.app.js"></script>
+</head>
+<body>
+ <nav class="navbar navbar-default">
+ <div class="container-fluid">
+ <div class="navbar-header">
+ <a class="navbar-brand" href="../" target="_self">Zuul Dashboard</a>
+ </div>
+ <ul class="nav navbar-nav">
+ <li class="active"><a href="status.html" target="_self">Status</a></li>
+ <li><a href="jobs.html" target="_self">Jobs</a></li>
+ <li><a href="builds.html" target="_self">Builds</a></li>
+ </ul>
+ </div>
+ </nav>
+ <div id="zuul_container"></div>
+ <script>
+ // @license magnet:?xt=urn:btih:8e4f440f4c65981c5bf93c76d35135ba5064d8b7&dn=apache-2.0.txt Apache 2.0
+ zuul_build_dom(jQuery, '#zuul_container');
+ zuul_start(jQuery);
+ // @license-end
+ </script>
+</body>
+</html>
diff --git a/zuul/web/static/styles/zuul.css b/zuul/web/static/styles/zuul.css
new file mode 100644
index 0000000..44fd737
--- /dev/null
+++ b/zuul/web/static/styles/zuul.css
@@ -0,0 +1,58 @@
+.zuul-change {
+ margin-bottom: 10px;
+}
+
+.zuul-change-id {
+ float: right;
+}
+
+.zuul-job-result {
+ float: right;
+ width: 70px;
+ height: 15px;
+ margin: 2px 0 0 0;
+}
+
+.zuul-change-total-result {
+ height: 10px;
+ width: 100px;
+ margin: 0;
+ display: inline-block;
+ vertical-align: middle;
+}
+
+.zuul-spinner,
+.zuul-spinner:hover {
+ opacity: 0;
+ transition: opacity 0.5s ease-out;
+ cursor: default;
+ pointer-events: none;
+}
+
+.zuul-spinner-on,
+.zuul-spinner-on:hover {
+ opacity: 1;
+ transition-duration: 0.2s;
+ cursor: progress;
+}
+
+.zuul-change-cell {
+ padding-left: 5px;
+}
+
+.zuul-change-job {
+ padding: 2px 8px;
+}
+
+.zuul-job-name {
+ font-size: small;
+}
+
+.zuul-non-voting-desc {
+ font-size: smaller;
+}
+
+.zuul-patchset-header {
+ font-size: small;
+ padding: 8px 12px;
+}
\ No newline at end of file
diff --git a/zuul/webapp.py b/zuul/webapp.py
index 134fb3c..b5fdc0e 100644
--- a/zuul/webapp.py
+++ b/zuul/webapp.py
@@ -108,15 +108,17 @@
def _handle_keys(self, request, path):
m = re.match('/keys/(.*?)/(.*?).pub', path)
if not m:
- raise webob.exc.HTTPNotFound()
+ raise webob.exc.HTTPBadRequest()
source_name = m.group(1)
project_name = m.group(2)
source = self.scheduler.connections.getSource(source_name)
if not source:
- raise webob.exc.HTTPNotFound()
+ raise webob.exc.HTTPNotFound(
+ detail="Cannot locate a source named %s" % source_name)
project = source.getProject(project_name)
- if not project:
- raise webob.exc.HTTPNotFound()
+ if not project or not hasattr(project, 'public_key'):
+ raise webob.exc.HTTPNotFound(
+ detail="Cannot locate a project named %s" % project_name)
pem_public_key = encryption.serialize_rsa_public_key(
project.public_key)
@@ -179,7 +181,7 @@
# Call time.time() again because formatting above may take
# longer than the cache timeout.
self.cache_time = time.time()
- except:
+ except Exception:
self.log.exception("Exception formatting status:")
raise
