Merge "Ensure ref-updated jobs run with their ref" into feature/zuulv3
diff --git a/doc/source/admin/components.rst b/doc/source/admin/components.rst
index e30966b..dcaa7b4 100644
--- a/doc/source/admin/components.rst
+++ b/doc/source/admin/components.rst
@@ -229,6 +229,8 @@
To start the merger, run ``zuul-merger``. To stop it, kill the
PID which was saved in the pidfile specified in the configuration.
+.. _executor:
+
Executor
--------
diff --git a/doc/source/admin/tenants.rst b/doc/source/admin/tenants.rst
index 1f8f7db..b3b2d9c 100644
--- a/doc/source/admin/tenants.rst
+++ b/doc/source/admin/tenants.rst
@@ -29,6 +29,7 @@
- tenant:
name: my-tenant
max-nodes-per-job: 5
+ exclude-unprotected-branches: false
source:
gerrit:
config-projects:
@@ -39,7 +40,8 @@
- zuul-jobs:
shadow: common-config
- project1
- - project2
+ - project2:
+ exclude-unprotected-branches: true
The following attributes are supported:
@@ -53,6 +55,16 @@
The maximum number of nodes a job can request, default to 5.
A '-1' value removes the limit.
+**exclude-unprotected-branches** (optional)
+ When using a branch and pull model on a shared github repository there are
+ usually one or more protected branches which are gated and a dynamic number of
+ personal/feature branches which are the source for the pull requests. These
+ branches can potentially include broken zuul config and therefore break the
+ global tenant wide configuration. In order to deal with this zuul's operations
+ can be limited to the protected branches which are gated. This is a tenant
+ wide setting and can be overridden per project. If not specified, defaults
+ to ``false``.
+
**source** (required)
A dictionary of sources to consult for projects. A tenant may
contain projects from multiple sources; each of those sources must
@@ -104,6 +116,10 @@
"zuul-jobs" projects, the definition in "common-config" will be
used.
+ **exclude-unprotected-branches**
+ Define if unprotected github branches should be processed. Defaults to the
+ tenant wide setting of exclude-unprotected-branches.
+
The order of the projects listed in a tenant is important. A job
which is defined in one project may not be redefined in another
project; therefore, once a job appears in one project, a project
diff --git a/doc/source/user/concepts.rst b/doc/source/user/concepts.rst
index 6197396..318de09 100644
--- a/doc/source/user/concepts.rst
+++ b/doc/source/user/concepts.rst
@@ -40,7 +40,8 @@
configured with any number of reporters. See :ref:`drivers` for a
full list of available reporters.
-The items enqueued into a pipeline are each associated with a git ref.
+The items enqueued into a pipeline are each associated with a
+`git ref <https://git-scm.com/book/en/v2/Git-Internals-Git-References>`_.
That ref may point to a proposed change, or it may be the tip of a
branch or a tag. The triggering event determines the ref, and whether
it represents a proposed or merged commit. Zuul prepares the ref for
@@ -67,7 +68,7 @@
change appears.
Jobs specify the type and quantity of nodes which they require.
-Before executing each job, Zuul will contact it's companion program,
+Before executing each job, Zuul will contact its companion program,
Nodepool, to supply them. Nodepool may be configured to supply static
nodes or contact cloud providers to create or delete nodes as
necessary. The types of nodes available to Zuul are determined by the
@@ -80,6 +81,6 @@
script) or sophisticated deployment scenarios. When Zuul runs
Ansible, it attempts to do so in a manner most similar to the way that
Ansible might be used to orchestrate remote systems. Ansible itself
-is run on the executor and acts remotely upon the test nodes supplied
-to a job. This facilitates continuous delivery by making it possible
-to use the same Ansible playbooks in testing and production.
+is run on the :ref:`executor <executor>` and acts remotely upon the test
+nodes supplied to a job. This facilitates continuous delivery by making it
+possible to use the same Ansible playbooks in testing and production.
diff --git a/doc/source/user/gating.rst b/doc/source/user/gating.rst
index c1d04a7..795df72 100644
--- a/doc/source/user/gating.rst
+++ b/doc/source/user/gating.rst
@@ -227,7 +227,8 @@
A given dependent pipeline may have as many shared change queues as
necessary, so groups of related projects may share a change queue
-without interfering with unrelated projects. Independent pipelines do
+without interfering with unrelated projects.
+:value:`Independent pipelines <pipeline.manager.independent>` do
not use shared change queues, however, they may still be used to test
changes across projects using cross-project dependencies.
diff --git a/doc/source/user/index.rst b/doc/source/user/index.rst
index 3eca04b..8c7308b 100644
--- a/doc/source/user/index.rst
+++ b/doc/source/user/index.rst
@@ -4,7 +4,7 @@
This guide is for all users of Zuul. If you work on a project where
Zuul is used to drive automation (whether that's testing proposed
changes, building artifacts, or deploying builds), this guide will
-help you understand the concepts that underly Zuul, and how to
+help you understand the concepts that underlie Zuul, and how to
configure it to meet your needs.
diff --git a/doc/source/user/jobs.rst b/doc/source/user/jobs.rst
index 068da0b..577d147 100644
--- a/doc/source/user/jobs.rst
+++ b/doc/source/user/jobs.rst
@@ -91,23 +91,25 @@
Job Variables
~~~~~~~~~~~~~
-Any variables specified in the job definition are available as Ansible
-host variables. They are added to the `vars` section of the inventory
-file under the `all` hosts group, so they are available to all hosts.
-Simply refer to them by the name specified in the job's `vars`
-section.
+Any variables specified in the job definition (using the
+:attr:`job.vars` attribute) are available as Ansible host variables.
+They are added to the ``vars`` section of the inventory file under the
+``all`` hosts group, so they are available to all hosts. Simply refer
+to them by the name specified in the job's ``vars`` section.
Secrets
~~~~~~~
-Secrets also appear as variables available to Ansible. Unlike job
-variables, these are not added to the inventory file (so that the
-inventory file may be kept for debugging purposes without revealing
-secrets). But they are still available to Ansible as normal
+:ref:`Secrets <secret>` also appear as variables available to Ansible.
+Unlike job variables, these are not added to the inventory file (so
+that the inventory file may be kept for debugging purposes without
+revealing secrets). But they are still available to Ansible as normal
variables. Because secrets are groups of variables, they will appear
as a dictionary structure in templates, with the dictionary itself
being the name of the secret, and its members the individual items in
-the secret. For example, a secret defined as::
+the secret. For example, a secret defined as:
+
+.. code-block:: yaml
- secret:
name: credentials
@@ -119,13 +121,12 @@
{{ credentials.username }} {{ credentials.password }}
-.. TODO: xref job vars
Zuul Variables
~~~~~~~~~~~~~~
Zuul supplies not only the variables specified by the job definition
-to Ansible, but also some variables from the Zuul itself.
+to Ansible, but also some variables from Zuul itself.
When a pipeline is triggered by an action, it enqueues items which may
vary based on the pipeline's configuration. For example, when a new
@@ -137,93 +138,123 @@
attributes in common. But other attributes may vary based on the type
of item.
-All items provide the following information as Ansible variables:
+.. var:: zuul
-**zuul.build**
- The UUID of the build. A build is a single execution of a job.
- When an item is enqueued into a pipeline, this usually results in
- one build of each job configured for that item's project. However,
- items may be re-enqueued in which case another build may run. In
- dependent pipelines, the same job may run multiple times for the
- same item as circumstances change ahead in the queue. Each time a
- job is run, for whatever reason, it is acompanied with a new
- unique id.
+ All items provide the following information as Ansible variables
+ under the ``zuul`` key:
-**zuul.buildset**
- The build set UUID. When Zuul runs jobs for an item, the collection
- of those jobs is known as a buildset. If the configuration of items
- ahead in a dependent pipeline changes, Zuul creates a new buildset
- and restarts all of the jobs.
+ .. var:: build
-**zuul.ref**
- The git ref of the item. This will be the full path (e.g.,
- 'refs/heads/master' or 'refs/changes/...').
+ The UUID of the build. A build is a single execution of a job.
+ When an item is enqueued into a pipeline, this usually results
+ in one build of each job configured for that item's project.
+ However, items may be re-enqueued in which case another build
+ may run. In dependent pipelines, the same job may run multiple
+ times for the same item as circumstances change ahead in the
+ queue. Each time a job is run, for whatever reason, it is
+ acompanied with a new unique id.
-**zuul.pipeline**
- The name of the pipeline in which the job is being run.
+ .. var:: buildset
-**zuul.job**
- The name of the job being run.
+ The build set UUID. When Zuul runs jobs for an item, the
+ collection of those jobs is known as a buildset. If the
+ configuration of items ahead in a dependent pipeline changes,
+ Zuul creates a new buildset and restarts all of the jobs.
-**zuul.voting**
- A boolean indicating whether the job is voting.
+ .. var:: ref
-**zuul.project**
- The item's project. This is a data structure with the following
- fields:
+ The git ref of the item. This will be the full path (e.g.,
+ `refs/heads/master` or `refs/changes/...`).
-**zuul.project.name**
- The name of the project, excluding hostname. E.g., `org/project`.
+ .. var:: pipeline
-**zuul.project.short_name**
- The name of the project, excluding directories or organizations.
- E.g., `project`.
+ The name of the pipeline in which the job is being run.
-**zuul.project.canonical_hostname**
- The canonical hostname where the project lives. E.g.,
- `git.example.com`.
+ .. var:: job
-**zuul.project.canonical_name**
- The full canonical name of the project including hostname. E.g.,
- `git.example.com/org/project`.
+ The name of the job being run.
-**zuul.tenant**
- The name of the current Zuul tenant.
+ .. var:: voting
-**zuul.jobtags**
- A list of tags associated with the job. Not to be confused with git
- tags, these are simply free-form text fields that can be used by the
- job for reporting or classification purposes.
+ A boolean indicating whether the job is voting.
-**zuul.items**
+ .. var:: project
- A list of dictionaries, each representing an item being tested with
- this change with the format:
+ The item's project. This is a data structure with the following
+ fields:
- **project.name**
- The name of the project, excluding hostname. E.g., `org/project`.
+ .. var:: name
- **project.short_name**
- The name of the project, excluding directories or organizations.
- E.g., `project`.
+ The name of the project, excluding hostname. E.g., `org/project`.
- **project.canonical_hostname**
- The canonical hostname where the project lives. E.g.,
- `git.example.com`.
+ .. var:: short_name
- **project.canonical_name**
- The full canonical name of the project including hostname. E.g.,
- `git.example.com/org/project`.
+ The name of the project, excluding directories or
+ organizations. E.g., `project`.
- **branch**
- The target branch of the change (without the `refs/heads/` prefix).
+ .. var:: canonical_hostname
- **change**
- The identifier for the change.
+ The canonical hostname where the project lives. E.g.,
+ `git.example.com`.
- **patchset**
- The patchset identifier for the change. If a change is revised,
- this will have a different value.
+ .. var:: canonical_name
+
+ The full canonical name of the project including hostname.
+ E.g., `git.example.com/org/project`.
+
+ .. var:: tenant
+
+ The name of the current Zuul tenant.
+
+ .. var:: jobtags
+
+ A list of tags associated with the job. Not to be confused with
+ git tags, these are simply free-form text fields that can be
+ used by the job for reporting or classification purposes.
+
+ .. var:: items
+ :type: list
+
+ A list of dictionaries, each representing an item being tested
+ with this change with the format:
+
+ .. var:: project
+
+ The item's project. This is a data structure with the
+ following fields:
+
+ .. var:: name
+
+ The name of the project, excluding hostname. E.g.,
+ `org/project`.
+
+ .. var:: short_name
+
+ The name of the project, excluding directories or
+ organizations. E.g., `project`.
+
+ .. var:: canonical_hostname
+
+ The canonical hostname where the project lives. E.g.,
+ `git.example.com`.
+
+ .. var:: canonical_name
+
+ The full canonical name of the project including hostname.
+ E.g., `git.example.com/org/project`.
+
+ .. var:: branch
+
+ The target branch of the change (without the `refs/heads/` prefix).
+
+ .. var:: change
+
+ The identifier for the change.
+
+ .. var:: patchset
+
+ The patchset identifier for the change. If a change is
+ revised, this will have a different value.
Change Items
++++++++++++
@@ -233,15 +264,21 @@
change or a GitHub pull request). The following additional variables
are available:
-**zuul.branch**
- The target branch of the change (without the `refs/heads/` prefix).
+.. var:: zuul
+ :hidden:
-**zuul.change**
- The identifier for the change.
+ .. var:: branch
-**zuul.patchset**
- The patchset identifier for the change. If a change is revised,
- this will have a different value.
+ The target branch of the change (without the `refs/heads/` prefix).
+
+ .. var:: change
+
+ The identifier for the change.
+
+ .. var:: patchset
+
+ The patchset identifier for the change. If a change is revised,
+ this will have a different value.
Branch Items
++++++++++++
@@ -252,18 +289,25 @@
of verifying the current condition of the branch. The following
additional variables are available:
-**zuul.branch**
- The name of the item's branch (without the `refs/heads/` prefix).
+.. var:: zuul
+ :hidden:
-**zuul.oldrev**
- If the item was enqueued as the result of a change merging or being
- pushed to the branch, the git sha of the old revision will be
- included here. Otherwise, this variable will be undefined.
+ .. var:: branch
-**zuul.newrev**
- If the item was enqueued as the result of a change merging or being
- pushed to the branch, the git sha of the new revision will be
- included here. Otherwise, this variable will be undefined.
+ The name of the item's branch (without the `refs/heads/`
+ prefix).
+
+ .. var:: oldrev
+
+ If the item was enqueued as the result of a change merging or
+ being pushed to the branch, the git sha of the old revision will
+ be included here. Otherwise, this variable will be undefined.
+
+ .. var:: newrev
+
+ If the item was enqueued as the result of a change merging or
+ being pushed to the branch, the git sha of the new revision will
+ be included here. Otherwise, this variable will be undefined.
Tag Items
+++++++++
@@ -272,18 +316,24 @@
tag was created or deleted. The following additional variables are
available:
-**zuul.tag**
- The name of the item's tag (without the `refs/tags/` prefix).
+.. var:: zuul
+ :hidden:
-**zuul.oldrev**
- If the item was enqueued as the result of a tag being deleted, the
- previous git sha of the tag will be included here. If the tag was
- created, this variable will be undefined.
+ .. var:: tag
-**zuul.newrev**
- If the item was enqueued as the result of a tag being created, the
- new git sha of the tag will be included here. If the tag was
- deleted, this variable will be undefined.
+ The name of the item's tag (without the `refs/tags/` prefix).
+
+ .. var:: oldrev
+
+ If the item was enqueued as the result of a tag being deleted,
+ the previous git sha of the tag will be included here. If the
+ tag was created, this variable will be undefined.
+
+ .. var:: newrev
+
+ If the item was enqueued as the result of a tag being created,
+ the new git sha of the tag will be included here. If the tag
+ was deleted, this variable will be undefined.
Ref Items
+++++++++
@@ -293,15 +343,20 @@
to identify the ref. The following additional variables are
available:
-**zuul.oldrev**
- If the item was enqueued as the result of a ref being deleted, the
- previous git sha of the ref will be included here. If the ref was
- created, this variable will be undefined.
+.. var:: zuul
+ :hidden:
-**zuul.newrev**
- If the item was enqueued as the result of a ref being created, the
- new git sha of the ref will be included here. If the ref was
- deleted, this variable will be undefined.
+ .. var:: oldrev
+
+ If the item was enqueued as the result of a ref being deleted,
+ the previous git sha of the ref will be included here. If the
+ ref was created, this variable will be undefined.
+
+ .. var:: newrev
+
+ If the item was enqueued as the result of a ref being created,
+ the new git sha of the ref will be included here. If the ref
+ was deleted, this variable will be undefined.
Working Directory
+++++++++++++++++
@@ -309,17 +364,29 @@
Additionally, some information about the working directory and the
executor running the job is available:
-**zuul.executor.hostname**
- The hostname of the executor.
+.. var:: zuul
+ :hidden:
-**zuul.executor.src_root**
- The path to the source directory.
+ .. var:: executor
-**zuul.executor.log_root**
- The path to the logs directory.
+ A number of values related to the executor running the job are
+ available:
-**zuul.executor.work_root**
- The path to the working directory.
+ .. var:: hostname
+
+ The hostname of the executor.
+
+ .. var:: src_root
+
+ The path to the source directory.
+
+ .. var:: log_root
+
+ The path to the logs directory.
+
+ .. var:: work_root
+
+ The path to the working directory.
.. _user_sitewide_variables:
@@ -355,7 +422,9 @@
The job may return some values to Zuul to affect its behavior. To
return a value, use the *zuul_return* Ansible module in a job
-playbook. For example::
+playbook. For example:
+
+.. code-block:: yaml
tasks:
- zuul_return:
@@ -368,7 +437,9 @@
Several uses of these values are planned, but the only currently
implemented use is to set the log URL for a build. To do so, set the
-**zuul.log_url** value. For example::
+**zuul.log_url** value. For example:
+
+.. code-block:: yaml
tasks:
- zuul_return:
diff --git a/tests/base.py b/tests/base.py
index 5a06f85..4b06c28 100755
--- a/tests/base.py
+++ b/tests/base.py
@@ -554,6 +554,98 @@
_points_to_commits_only = True
+class FakeGithub(object):
+
+ class FakeUser(object):
+ def __init__(self, login):
+ self.login = login
+ self.name = "Github User"
+ self.email = "github.user@example.com"
+
+ class FakeBranch(object):
+ def __init__(self, branch='master'):
+ self.name = branch
+
+ class FakeStatus(object):
+ def __init__(self, state, url, description, context, user):
+ self._state = state
+ self._url = url
+ self._description = description
+ self._context = context
+ self._user = user
+
+ def as_dict(self):
+ return {
+ 'state': self._state,
+ 'url': self._url,
+ 'description': self._description,
+ 'context': self._context,
+ 'creator': {
+ 'login': self._user
+ }
+ }
+
+ class FakeCommit(object):
+ def __init__(self):
+ self._statuses = []
+
+ def set_status(self, state, url, description, context, user):
+ status = FakeGithub.FakeStatus(
+ state, url, description, context, user)
+ # always insert a status to the front of the list, to represent
+ # the last status provided for a commit.
+ self._statuses.insert(0, status)
+
+ def statuses(self):
+ return self._statuses
+
+ class FakeRepository(object):
+ def __init__(self):
+ self._branches = [FakeGithub.FakeBranch()]
+ self._commits = {}
+
+ def branches(self, protected=False):
+ if protected:
+ # simulate there is no protected branch
+ return []
+ return self._branches
+
+ def create_status(self, sha, state, url, description, context,
+ user='zuul'):
+ # Since we're bypassing github API, which would require a user, we
+ # default the user as 'zuul' here.
+ commit = self._commits.get(sha, None)
+ if commit is None:
+ commit = FakeGithub.FakeCommit()
+ self._commits[sha] = commit
+ commit.set_status(state, url, description, context, user)
+
+ def commit(self, sha):
+ commit = self._commits.get(sha, None)
+ if commit is None:
+ commit = FakeGithub.FakeCommit()
+ self._commits[sha] = commit
+ return commit
+
+ def __init__(self):
+ self._repos = {}
+
+ def user(self, login):
+ return self.FakeUser(login)
+
+ def repository(self, owner, proj):
+ return self._repos.get((owner, proj), None)
+
+ def repo_from_project(self, project):
+ # This is a convenience method for the tests.
+ owner, proj = project.split('/')
+ return self.repository(owner, proj)
+
+ def addProject(self, project):
+ owner, proj = project.name.split('/')
+ self._repos[(owner, proj)] = self.FakeRepository()
+
+
class FakeGithubPullRequest(object):
def __init__(self, github, number, project, branch,
@@ -889,6 +981,13 @@
self.merge_failure = False
self.merge_not_allowed_count = 0
self.reports = []
+ self.github_client = FakeGithub()
+
+ def getGithubClient(self,
+ project=None,
+ user_id=None,
+ use_app=True):
+ return self.github_client
def openFakePullRequest(self, project, branch, subject, files=[],
body=None):
@@ -937,6 +1036,12 @@
data=payload, headers=headers)
return urllib.request.urlopen(req)
+ def addProject(self, project):
+ # use the original method here and additionally register it in the
+ # fake github
+ super(FakeGithubConnection, self).addProject(project)
+ self.getGithubClient(project).addProject(project)
+
def getPull(self, project, number):
pr = self.pull_requests[number - 1]
data = {
@@ -975,14 +1080,6 @@
pr = self.pull_requests[number - 1]
return pr.reviews
- def getUser(self, login):
- data = {
- 'username': login,
- 'name': 'Github User',
- 'email': 'github.user@example.com'
- }
- return data
-
def getRepoPermission(self, project, login):
owner, proj = project.split('/')
for pr in self.pull_requests:
@@ -999,12 +1096,6 @@
def real_getGitUrl(self, project):
return super(FakeGithubConnection, self).getGitUrl(project)
- def getProjectBranches(self, project):
- """Masks getProjectBranches since we don't have a real github"""
-
- # just returns master for now
- return ['master']
-
def commentPull(self, project, pr_number, message):
# record that this got reported
self.reports.append((project, pr_number, 'comment'))
@@ -1023,27 +1114,13 @@
' conflict')
pull_request.setMerged(commit_message)
- def getCommitStatuses(self, project, sha):
- return self.statuses.get(project, {}).get(sha, [])
-
def setCommitStatus(self, project, sha, state, url='', description='',
context='default', user='zuul'):
- # record that this got reported
+ # record that this got reported and call original method
self.reports.append((project, sha, 'status', (user, context, state)))
- # always insert a status to the front of the list, to represent
- # the last status provided for a commit.
- # Since we're bypassing github API, which would require a user, we
- # default the user as 'zuul' here.
- self.statuses.setdefault(project, {}).setdefault(sha, [])
- self.statuses[project][sha].insert(0, {
- 'state': state,
- 'url': url,
- 'description': description,
- 'context': context,
- 'creator': {
- 'login': user
- }
- })
+ super(FakeGithubConnection, self).setCommitStatus(
+ project, sha, state,
+ url=url, description=description, context=context)
def labelPull(self, project, pr_number, label):
# record that this got reported
@@ -2131,15 +2208,15 @@
config = [{'tenant':
{'name': 'tenant-one',
'source': {driver:
- {'config-projects': ['common-config'],
+ {'config-projects': ['org/common-config'],
'untrusted-projects': untrusted_projects}}}}]
f.write(yaml.dump(config).encode('utf8'))
f.close()
self.config.set('scheduler', 'tenant_config',
os.path.join(FIXTURE_DIR, f.name))
- self.init_repo('common-config')
- self.addCommitToRepo('common-config', 'add content from fixture',
+ self.init_repo('org/common-config')
+ self.addCommitToRepo('org/common-config', 'add content from fixture',
files, branch='master', tag='init')
return True
diff --git a/tests/fixtures/config/multi-driver/git/common-config/playbooks/project-gerrit.yaml b/tests/fixtures/config/multi-driver/git/org_common-config/playbooks/project-gerrit.yaml
similarity index 100%
rename from tests/fixtures/config/multi-driver/git/common-config/playbooks/project-gerrit.yaml
rename to tests/fixtures/config/multi-driver/git/org_common-config/playbooks/project-gerrit.yaml
diff --git a/tests/fixtures/config/multi-driver/git/common-config/playbooks/project1-github.yaml b/tests/fixtures/config/multi-driver/git/org_common-config/playbooks/project1-github.yaml
similarity index 100%
rename from tests/fixtures/config/multi-driver/git/common-config/playbooks/project1-github.yaml
rename to tests/fixtures/config/multi-driver/git/org_common-config/playbooks/project1-github.yaml
diff --git a/tests/fixtures/config/multi-driver/git/common-config/zuul.yaml b/tests/fixtures/config/multi-driver/git/org_common-config/zuul.yaml
similarity index 100%
rename from tests/fixtures/config/multi-driver/git/common-config/zuul.yaml
rename to tests/fixtures/config/multi-driver/git/org_common-config/zuul.yaml
diff --git a/tests/fixtures/config/multi-driver/main.yaml b/tests/fixtures/config/multi-driver/main.yaml
index 301df38..4eed523 100644
--- a/tests/fixtures/config/multi-driver/main.yaml
+++ b/tests/fixtures/config/multi-driver/main.yaml
@@ -3,7 +3,7 @@
source:
github:
config-projects:
- - common-config
+ - org/common-config
untrusted-projects:
- org/project1
gerrit:
diff --git a/tests/fixtures/config/push-reqs/git/common-config/playbooks/job1.yaml b/tests/fixtures/config/push-reqs/git/org_common-config/playbooks/job1.yaml
similarity index 100%
rename from tests/fixtures/config/push-reqs/git/common-config/playbooks/job1.yaml
rename to tests/fixtures/config/push-reqs/git/org_common-config/playbooks/job1.yaml
diff --git a/tests/fixtures/config/push-reqs/git/common-config/zuul.yaml b/tests/fixtures/config/push-reqs/git/org_common-config/zuul.yaml
similarity index 100%
rename from tests/fixtures/config/push-reqs/git/common-config/zuul.yaml
rename to tests/fixtures/config/push-reqs/git/org_common-config/zuul.yaml
diff --git a/tests/fixtures/config/push-reqs/main.yaml b/tests/fixtures/config/push-reqs/main.yaml
index d9f1a42..b58db73 100644
--- a/tests/fixtures/config/push-reqs/main.yaml
+++ b/tests/fixtures/config/push-reqs/main.yaml
@@ -3,7 +3,7 @@
source:
github:
config-projects:
- - common-config
+ - org/common-config
untrusted-projects:
- org/project1
gerrit:
diff --git a/tests/fixtures/config/tenant-parser/unprotected-branches.yaml b/tests/fixtures/config/tenant-parser/unprotected-branches.yaml
new file mode 100644
index 0000000..bf2feef
--- /dev/null
+++ b/tests/fixtures/config/tenant-parser/unprotected-branches.yaml
@@ -0,0 +1,11 @@
+- tenant:
+ name: tenant-one
+ exclude-unprotected-branches: true
+ source:
+ gerrit:
+ config-projects:
+ - common-config:
+ exclude-unprotected-branches: false
+ untrusted-projects:
+ - org/project1
+ - org/project2
diff --git a/tests/fixtures/config/unprotected-branches/git/org_common-config/zuul.yaml b/tests/fixtures/config/unprotected-branches/git/org_common-config/zuul.yaml
new file mode 100644
index 0000000..c0fbf0d
--- /dev/null
+++ b/tests/fixtures/config/unprotected-branches/git/org_common-config/zuul.yaml
@@ -0,0 +1,19 @@
+- pipeline:
+ name: check
+ manager: independent
+ trigger:
+ github:
+ - event: pull_request
+ action:
+ - opened
+ - changed
+ - reopened
+ success:
+ github:
+ status: 'success'
+ failure:
+ github:
+ status: 'failure'
+ start:
+ github:
+ comment: true
diff --git a/tests/fixtures/config/unprotected-branches/git/org_project1/README b/tests/fixtures/config/unprotected-branches/git/org_project1/README
new file mode 100644
index 0000000..9daeafb
--- /dev/null
+++ b/tests/fixtures/config/unprotected-branches/git/org_project1/README
@@ -0,0 +1 @@
+test
diff --git a/tests/fixtures/config/multi-driver/git/common-config/playbooks/project-gerrit.yaml b/tests/fixtures/config/unprotected-branches/git/org_project1/playbooks/project-test.yaml
similarity index 100%
copy from tests/fixtures/config/multi-driver/git/common-config/playbooks/project-gerrit.yaml
copy to tests/fixtures/config/unprotected-branches/git/org_project1/playbooks/project-test.yaml
diff --git a/tests/fixtures/config/unprotected-branches/git/org_project1/zuul.yaml b/tests/fixtures/config/unprotected-branches/git/org_project1/zuul.yaml
new file mode 100644
index 0000000..31abadf
--- /dev/null
+++ b/tests/fixtures/config/unprotected-branches/git/org_project1/zuul.yaml
@@ -0,0 +1,8 @@
+- job:
+ name: project-test
+
+- project:
+ name: org/project1
+ check:
+ jobs:
+ - project-test
diff --git a/tests/fixtures/config/unprotected-branches/git/org_project2/README b/tests/fixtures/config/unprotected-branches/git/org_project2/README
new file mode 100644
index 0000000..9daeafb
--- /dev/null
+++ b/tests/fixtures/config/unprotected-branches/git/org_project2/README
@@ -0,0 +1 @@
+test
diff --git a/tests/fixtures/config/unprotected-branches/git/org_project2/zuul.yaml b/tests/fixtures/config/unprotected-branches/git/org_project2/zuul.yaml
new file mode 100644
index 0000000..64d316d
--- /dev/null
+++ b/tests/fixtures/config/unprotected-branches/git/org_project2/zuul.yaml
@@ -0,0 +1 @@
+This zuul.yaml is intentionally broken and should not be loaded on startup.
diff --git a/tests/fixtures/config/unprotected-branches/main.yaml b/tests/fixtures/config/unprotected-branches/main.yaml
new file mode 100644
index 0000000..8078d37
--- /dev/null
+++ b/tests/fixtures/config/unprotected-branches/main.yaml
@@ -0,0 +1,10 @@
+- tenant:
+ name: tenant-one
+ source:
+ github:
+ config-projects:
+ - org/common-config
+ untrusted-projects:
+ - org/project1
+ - org/project2:
+ exclude-unprotected-branches: true
diff --git a/tests/fixtures/layouts/reporting-multiple-github.yaml b/tests/fixtures/layouts/reporting-multiple-github.yaml
index f14000e..22fa1e7 100644
--- a/tests/fixtures/layouts/reporting-multiple-github.yaml
+++ b/tests/fixtures/layouts/reporting-multiple-github.yaml
@@ -25,7 +25,7 @@
- job:
name: project1-test1
- job:
- name: project2-test1
+ name: project2-test2
- project:
name: org/project1
diff --git a/tests/unit/test_configloader.py b/tests/unit/test_configloader.py
index d08c6a1..1ba4ed9 100644
--- a/tests/unit/test_configloader.py
+++ b/tests/unit/test_configloader.py
@@ -182,6 +182,7 @@
def test_tenant_groups3(self):
tenant = self.sched.abide.tenants.get('tenant-one')
+ self.assertEqual(False, tenant.exclude_unprotected_branches)
self.assertEqual(['common-config'],
[x.name for x in tenant.config_projects])
self.assertEqual(['org/project1', 'org/project2'],
@@ -212,6 +213,29 @@
project2_config.pipelines['check'].job_list.jobs)
+class TestTenantUnprotectedBranches(TenantParserTestCase):
+ tenant_config_file = 'config/tenant-parser/unprotected-branches.yaml'
+
+ def test_tenant_unprotected_branches(self):
+ tenant = self.sched.abide.tenants.get('tenant-one')
+ self.assertEqual(True, tenant.exclude_unprotected_branches)
+
+ self.assertEqual(['common-config'],
+ [x.name for x in tenant.config_projects])
+ self.assertEqual(['org/project1', 'org/project2'],
+ [x.name for x in tenant.untrusted_projects])
+
+ tpc = tenant.project_configs
+ project_name = tenant.config_projects[0].canonical_name
+ self.assertEqual(False, tpc[project_name].exclude_unprotected_branches)
+
+ project_name = tenant.untrusted_projects[0].canonical_name
+ self.assertIsNone(tpc[project_name].exclude_unprotected_branches)
+
+ project_name = tenant.untrusted_projects[1].canonical_name
+ self.assertIsNone(tpc[project_name].exclude_unprotected_branches)
+
+
class TestSplitConfig(ZuulTestCase):
tenant_config_file = 'config/split-config/main.yaml'
diff --git a/tests/unit/test_executor.py b/tests/unit/test_executor.py
index f691135..3793edc 100755
--- a/tests/unit/test_executor.py
+++ b/tests/unit/test_executor.py
@@ -277,6 +277,11 @@
'layouts/repo-checkout-no-timer-override.yaml')
self.sched.reconfigure(self.config)
self.waitUntilSettled()
+ # If APScheduler is in mid-event when we remove the job, we
+ # can end up with one more event firing, so give it an extra
+ # second to settle.
+ time.sleep(1)
+ self.waitUntilSettled()
self.assertEquals(1, len(self.builds), "One build is running")
@@ -315,6 +320,11 @@
'layouts/repo-checkout-no-timer.yaml')
self.sched.reconfigure(self.config)
self.waitUntilSettled()
+ # If APScheduler is in mid-event when we remove the job, we
+ # can end up with one more event firing, so give it an extra
+ # second to settle.
+ time.sleep(1)
+ self.waitUntilSettled()
self.assertEquals(2, len(self.builds), "Two builds are running")
diff --git a/tests/unit/test_github_driver.py b/tests/unit/test_github_driver.py
index 4077cca..538de58 100644
--- a/tests/unit/test_github_driver.py
+++ b/tests/unit/test_github_driver.py
@@ -262,14 +262,19 @@
@simple_layout('layouts/reporting-github.yaml', driver='github')
def test_reporting(self):
project = 'org/project'
+ github = self.fake_github.github_client
+
# pipeline reports pull status both on start and success
self.executor_server.hold_jobs_in_build = True
A = self.fake_github.openFakePullRequest(project, 'master', 'A')
self.fake_github.emitEvent(A.getPullRequestOpenedEvent())
self.waitUntilSettled()
+
# We should have a status container for the head sha
- statuses = self.fake_github.statuses[project][A.head_sha]
- self.assertIn(A.head_sha, self.fake_github.statuses[project].keys())
+ self.assertIn(
+ A.head_sha, github.repo_from_project(project)._commits.keys())
+ statuses = self.fake_github.getCommitStatuses(project, A.head_sha)
+
# We should only have one status for the head sha
self.assertEqual(1, len(statuses))
check_status = statuses[0]
@@ -286,7 +291,7 @@
self.executor_server.release()
self.waitUntilSettled()
# We should only have two statuses for the head sha
- statuses = self.fake_github.statuses[project][A.head_sha]
+ statuses = self.fake_github.getCommitStatuses(project, A.head_sha)
self.assertEqual(2, len(statuses))
check_status = statuses[0]
check_url = ('http://zuul.example.com/status/#%s,%s' %
@@ -305,7 +310,7 @@
self.fake_github.emitEvent(
A.getCommentAddedEvent('reporting check'))
self.waitUntilSettled()
- statuses = self.fake_github.statuses[project][A.head_sha]
+ statuses = self.fake_github.getCommitStatuses(project, A.head_sha)
self.assertEqual(2, len(statuses))
# comments increased by one for the start message
self.assertEqual(2, len(A.comments))
@@ -315,7 +320,7 @@
self.executor_server.release()
self.waitUntilSettled()
# pipeline reports success status
- statuses = self.fake_github.statuses[project][A.head_sha]
+ statuses = self.fake_github.getCommitStatuses(project, A.head_sha)
self.assertEqual(3, len(statuses))
report_status = statuses[0]
self.assertEqual('tenant-one/reporting', report_status['context'])
@@ -347,7 +352,7 @@
self.fake_github.emitEvent(
A.getCommentAddedEvent('long pipeline'))
self.waitUntilSettled()
- statuses = self.fake_github.statuses[project][A.head_sha]
+ statuses = self.fake_github.getCommitStatuses(project, A.head_sha)
self.assertEqual(1, len(statuses))
check_status = statuses[0]
# Status is truncated due to long pipeline name
@@ -358,7 +363,7 @@
self.executor_server.release()
self.waitUntilSettled()
# We should only have two statuses for the head sha
- statuses = self.fake_github.statuses[project][A.head_sha]
+ statuses = self.fake_github.getCommitStatuses(project, A.head_sha)
self.assertEqual(2, len(statuses))
check_status = statuses[0]
# Status is truncated due to long pipeline name
@@ -451,6 +456,8 @@
@simple_layout('layouts/reporting-multiple-github.yaml', driver='github')
def test_reporting_multiple_github(self):
project = 'org/project1'
+ github = self.fake_github.github_client
+
# pipeline reports pull status both on start and success
self.executor_server.hold_jobs_in_build = True
A = self.fake_github.openFakePullRequest(project, 'master', 'A')
@@ -461,8 +468,9 @@
self.fake_github.emitEvent(B.getPullRequestOpenedEvent())
self.waitUntilSettled()
# We should have a status container for the head sha
- statuses = self.fake_github.statuses[project][A.head_sha]
- self.assertIn(A.head_sha, self.fake_github.statuses[project].keys())
+ statuses = self.fake_github.getCommitStatuses(project, A.head_sha)
+ self.assertIn(
+ A.head_sha, github.repo_from_project(project)._commits.keys())
# We should only have one status for the head sha
self.assertEqual(1, len(statuses))
check_status = statuses[0]
@@ -478,7 +486,7 @@
self.executor_server.release()
self.waitUntilSettled()
# We should only have two statuses for the head sha
- statuses = self.fake_github.statuses[project][A.head_sha]
+ statuses = self.fake_github.getCommitStatuses(project, A.head_sha)
self.assertEqual(2, len(statuses))
check_status = statuses[0]
check_url = ('http://zuul.example.com/status/#%s,%s' %
@@ -666,7 +674,7 @@
@simple_layout('layouts/basic-github.yaml', driver='github')
def test_push_event_reconfigure(self):
- pevent = self.fake_github.getPushEvent(project='common-config',
+ pevent = self.fake_github.getPushEvent(project='org/common-config',
ref='refs/heads/master',
modified_files=['zuul.yaml'])
@@ -693,3 +701,20 @@
self.fake_github.emitEvent,
('ping', pevent),
)
+
+
+class TestGithubUnprotectedBranches(ZuulTestCase):
+ config_file = 'zuul-github-driver.conf'
+ tenant_config_file = 'config/unprotected-branches/main.yaml'
+
+ def test_unprotected_branches(self):
+ tenant = self.sched.abide.tenants.get('tenant-one')
+
+ project1 = tenant.untrusted_projects[0]
+ project2 = tenant.untrusted_projects[1]
+
+ # project1 should have parsed master
+ self.assertIn('master', project1.unparsed_branch_config.keys())
+
+ # project2 should have no parsed branch
+ self.assertEqual(0, len(project2.unparsed_branch_config.keys()))
diff --git a/tests/unit/test_multi_driver.py b/tests/unit/test_multi_driver.py
index e40591b..1844c33 100644
--- a/tests/unit/test_multi_driver.py
+++ b/tests/unit/test_multi_driver.py
@@ -46,7 +46,8 @@
# Check on reporting results
# github should have a success status (only).
- statuses = self.fake_github.statuses['org/project1'][B.head_sha]
+ statuses = self.fake_github.getCommitStatuses(
+ 'org/project1', B.head_sha)
self.assertEqual(1, len(statuses))
self.assertEqual('success', statuses[0]['state'])
diff --git a/tests/unit/test_scheduler.py b/tests/unit/test_scheduler.py
index d77a7be..93367b9 100755
--- a/tests/unit/test_scheduler.py
+++ b/tests/unit/test_scheduler.py
@@ -1898,6 +1898,11 @@
self.commitConfigUpdate('common-config', 'layouts/no-timer.yaml')
self.sched.reconfigure(self.config)
self.waitUntilSettled()
+ # If APScheduler is in mid-event when we remove the job, we
+ # can end up with one more event firing, so give it an extra
+ # second to settle.
+ time.sleep(1)
+ self.waitUntilSettled()
self.assertEqual(len(self.builds), 1, "One timer job")
@@ -2861,6 +2866,12 @@
# below don't race against more jobs being queued.
self.commitConfigUpdate('common-config', 'layouts/no-timer.yaml')
self.sched.reconfigure(self.config)
+ self.waitUntilSettled()
+ # If APScheduler is in mid-event when we remove the job, we
+ # can end up with one more event firing, so give it an extra
+ # second to settle.
+ time.sleep(1)
+ self.waitUntilSettled()
self.executor_server.release()
self.waitUntilSettled()
@@ -2908,6 +2919,11 @@
'layouts/no-timer.yaml')
self.sched.reconfigure(self.config)
self.waitUntilSettled()
+ # If APScheduler is in mid-event when we remove the job,
+ # we can end up with one more event firing, so give it an
+ # extra second to settle.
+ time.sleep(1)
+ self.waitUntilSettled()
self.assertEqual(len(self.builds), 1,
'Timer builds iteration #%d' % x)
self.executor_server.release('.*')
@@ -2986,6 +3002,11 @@
self.commitConfigUpdate('common-config', 'layouts/no-timer.yaml')
self.sched.reconfigure(self.config)
self.waitUntilSettled()
+ # If APScheduler is in mid-event when we remove the job, we
+ # can end up with one more event firing, so give it an extra
+ # second to settle.
+ time.sleep(1)
+ self.waitUntilSettled()
self.executor_server.release('.*')
self.waitUntilSettled()
@@ -3030,6 +3051,11 @@
self.sched.reconfigure(self.config)
self.registerJobs()
self.waitUntilSettled()
+ # If APScheduler is in mid-event when we remove the job, we
+ # can end up with one more event firing, so give it an extra
+ # second to settle.
+ time.sleep(1)
+ self.waitUntilSettled()
self.worker.release('.*')
self.waitUntilSettled()
@@ -5375,6 +5401,9 @@
in_repo_conf = textwrap.dedent(
"""
- job:
+ name: project-test1
+
+ - job:
name: project-test2
semaphore: test-semaphore
diff --git a/tests/unit/test_v3.py b/tests/unit/test_v3.py
index 44cd5f6..8555208 100755
--- a/tests/unit/test_v3.py
+++ b/tests/unit/test_v3.py
@@ -90,6 +90,9 @@
in_repo_conf = textwrap.dedent(
"""
- job:
+ name: project-test1
+
+ - job:
name: project-test2
- project:
@@ -135,6 +138,77 @@
dict(name='project-test2', result='SUCCESS', changes='1,1'),
dict(name='project-test2', result='SUCCESS', changes='2,1')])
+ def test_dynamic_config_non_existing_job(self):
+ """Test that requesting a non existent job fails"""
+ in_repo_conf = textwrap.dedent(
+ """
+ - job:
+ name: project-test1
+
+ - project:
+ name: org/project
+ check:
+ jobs:
+ - non-existent-job
+ """)
+
+ in_repo_playbook = textwrap.dedent(
+ """
+ - hosts: all
+ tasks: []
+ """)
+
+ file_dict = {'.zuul.yaml': in_repo_conf,
+ 'playbooks/project-test2.yaml': in_repo_playbook}
+ A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A',
+ files=file_dict)
+ self.fake_gerrit.addEvent(A.getPatchsetCreatedEvent(1))
+ self.waitUntilSettled()
+ self.assertEqual(A.reported, 1,
+ "A should report failure")
+ self.assertEqual(A.patchsets[0]['approvals'][0]['value'], "-1")
+ self.assertIn('Job non-existent-job not defined', A.messages[0],
+ "A should have failed the check pipeline")
+ self.assertHistory([])
+
+ def test_dynamic_config_non_existing_job_in_template(self):
+ """Test that requesting a non existent job fails"""
+ in_repo_conf = textwrap.dedent(
+ """
+ - job:
+ name: project-test1
+
+ - project-template:
+ name: test-template
+ check:
+ jobs:
+ - non-existent-job
+
+ - project:
+ name: org/project
+ templates:
+ - test-template
+ """)
+
+ in_repo_playbook = textwrap.dedent(
+ """
+ - hosts: all
+ tasks: []
+ """)
+
+ file_dict = {'.zuul.yaml': in_repo_conf,
+ 'playbooks/project-test2.yaml': in_repo_playbook}
+ A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A',
+ files=file_dict)
+ self.fake_gerrit.addEvent(A.getPatchsetCreatedEvent(1))
+ self.waitUntilSettled()
+ self.assertEqual(A.reported, 1,
+ "A should report failure")
+ self.assertEqual(A.patchsets[0]['approvals'][0]['value'], "-1")
+ self.assertIn('Job non-existent-job not defined', A.messages[0],
+ "A should have failed the check pipeline")
+ self.assertHistory([])
+
def test_dynamic_config_new_patchset(self):
self.executor_server.hold_jobs_in_build = True
@@ -144,6 +218,9 @@
in_repo_conf = textwrap.dedent(
"""
- job:
+ name: project-test1
+
+ - job:
name: project-test2
- project:
@@ -221,6 +298,9 @@
in_repo_conf = textwrap.dedent(
"""
- job:
+ name: project-test1
+
+ - job:
name: project-test2
- project:
@@ -260,6 +340,9 @@
in_repo_conf = textwrap.dedent(
"""
- job:
+ name: project-test1
+
+ - job:
name: project-test2
- project:
@@ -322,6 +405,9 @@
in_repo_conf = textwrap.dedent(
"""
- job:
+ name: project-test1
+
+ - job:
name: project-test2
- project:
diff --git a/zuul/configloader.py b/zuul/configloader.py
index 4b9b8a0..a09147c 100644
--- a/zuul/configloader.py
+++ b/zuul/configloader.py
@@ -644,6 +644,12 @@
raise Exception("Job must be a string or dictionary")
attrs['_source_context'] = source_context
attrs['_start_mark'] = start_mark
+
+ # validate that the job is existing
+ with configuration_exceptions('project or project-template',
+ attrs):
+ layout.getJob(attrs['name'])
+
job_list.addJob(JobParser.fromYaml(tenant, layout, attrs,
project_pipeline=True))
@@ -929,6 +935,7 @@
'include': to_list(classes),
'exclude': to_list(classes),
'shadow': to_list(str),
+ 'exclude-unprotected-branches': bool,
}}
project = vs.Any(str, project_dict)
@@ -965,7 +972,9 @@
def getSchema(connections=None):
tenant = {vs.Required('name'): str,
'max-nodes-per-job': int,
- 'source': TenantParser.validateTenantSources(connections)}
+ 'source': TenantParser.validateTenantSources(connections),
+ 'exclude-unprotected-branches': bool,
+ }
return vs.Schema(tenant)
@staticmethod
@@ -975,6 +984,10 @@
tenant = model.Tenant(conf['name'])
if conf.get('max-nodes-per-job') is not None:
tenant.max_nodes_per_job = conf['max-nodes-per-job']
+ if conf.get('exclude-unprotected-branches') is not None:
+ tenant.exclude_unprotected_branches = \
+ conf['exclude-unprotected-branches']
+
tenant.unparsed_config = conf
unparsed_config = model.UnparsedTenantConfig()
# tpcs is TenantProjectConfigs
@@ -993,7 +1006,7 @@
TenantParser._loadTenantInRepoLayouts(merger, connections,
tenant.config_projects,
tenant.untrusted_projects,
- cached)
+ cached, tenant)
unparsed_config.extend(tenant.config_projects_config)
unparsed_config.extend(tenant.untrusted_projects_config)
tenant.layout = TenantParser._parseLayout(base, tenant,
@@ -1065,6 +1078,7 @@
project = source.getProject(conf)
project_include = current_include
shadow_projects = []
+ project_exclude_unprotected_branches = None
else:
project_name = list(conf.keys())[0]
project = source.getProject(project_name)
@@ -1078,10 +1092,14 @@
as_list(conf[project_name].get('exclude', [])))
if project_exclude:
project_include = frozenset(project_include - project_exclude)
+ project_exclude_unprotected_branches = conf[project_name].get(
+ 'exclude-unprotected-branches', None)
tenant_project_config = model.TenantProjectConfig(project)
tenant_project_config.load_classes = frozenset(project_include)
tenant_project_config.shadow_projects = shadow_projects
+ tenant_project_config.exclude_unprotected_branches = \
+ project_exclude_unprotected_branches
return tenant_project_config
@@ -1148,7 +1166,7 @@
@staticmethod
def _loadTenantInRepoLayouts(merger, connections, config_projects,
- untrusted_projects, cached):
+ untrusted_projects, cached, tenant):
config_projects_config = model.UnparsedTenantConfig()
untrusted_projects_config = model.UnparsedTenantConfig()
jobs = []
@@ -1196,7 +1214,7 @@
# branch. Remember the branch and then implicitly add a
# branch selector to each job there. This makes the
# in-repo configuration apply only to that branch.
- for branch in project.source.getProjectBranches(project):
+ for branch in project.source.getProjectBranches(project, tenant):
project.unparsed_branch_config[branch] = \
model.UnparsedTenantConfig()
job = merger.getFiles(
@@ -1416,11 +1434,11 @@
new_abide.tenants[tenant.name] = new_tenant
return new_abide
- def _loadDynamicProjectData(self, config, project, files, trusted):
+ def _loadDynamicProjectData(self, config, project, files, trusted, tenant):
if trusted:
branches = ['master']
else:
- branches = project.source.getProjectBranches(project)
+ branches = project.source.getProjectBranches(project, tenant)
for branch in branches:
fns1 = []
@@ -1472,11 +1490,12 @@
if include_config_projects:
config = model.UnparsedTenantConfig()
for project in tenant.config_projects:
- self._loadDynamicProjectData(config, project, files, True)
+ self._loadDynamicProjectData(
+ config, project, files, True, tenant)
else:
config = tenant.config_projects_config.copy()
for project in tenant.untrusted_projects:
- self._loadDynamicProjectData(config, project, files, False)
+ self._loadDynamicProjectData(config, project, files, False, tenant)
layout = model.Layout(tenant)
# NOTE: the actual pipeline objects (complete with queues and
diff --git a/zuul/driver/gerrit/gerritconnection.py b/zuul/driver/gerrit/gerritconnection.py
index d23857f..6bf43d6 100644
--- a/zuul/driver/gerrit/gerritconnection.py
+++ b/zuul/driver/gerrit/gerritconnection.py
@@ -616,7 +616,7 @@
(record.get('number'),))
return changes
- def getProjectBranches(self, project: Project) -> List[str]:
+ def getProjectBranches(self, project: Project, tenant) -> List[str]:
refs = self.getInfoRefs(project)
heads = [str(k[len('refs/heads/'):]) for k in refs.keys()
if k.startswith('refs/heads/')]
diff --git a/zuul/driver/gerrit/gerritsource.py b/zuul/driver/gerrit/gerritsource.py
index e41859e..7141080 100644
--- a/zuul/driver/gerrit/gerritsource.py
+++ b/zuul/driver/gerrit/gerritsource.py
@@ -54,8 +54,8 @@
def getProjectOpenChanges(self, project):
return self.connection.getProjectOpenChanges(project)
- def getProjectBranches(self, project):
- return self.connection.getProjectBranches(project)
+ def getProjectBranches(self, project, tenant):
+ return self.connection.getProjectBranches(project, tenant)
def getGitUrl(self, project):
return self.connection.getGitUrl(project)
diff --git a/zuul/driver/git/gitconnection.py b/zuul/driver/git/gitconnection.py
index f4fe7e5..0624088 100644
--- a/zuul/driver/git/gitconnection.py
+++ b/zuul/driver/git/gitconnection.py
@@ -48,7 +48,7 @@
def addProject(self, project):
self.projects[project.name] = project
- def getProjectBranches(self, project):
+ def getProjectBranches(self, project, tenant):
# TODO(jeblair): implement; this will need to handle local or
# remote git urls.
raise NotImplemented()
diff --git a/zuul/driver/git/gitsource.py b/zuul/driver/git/gitsource.py
index 61a328e..8d85c08 100644
--- a/zuul/driver/git/gitsource.py
+++ b/zuul/driver/git/gitsource.py
@@ -45,8 +45,8 @@
self.connection.addProject(p)
return p
- def getProjectBranches(self, project):
- return self.connection.getProjectBranches(project)
+ def getProjectBranches(self, project, tenant):
+ return self.connection.getProjectBranches(project, tenant)
def getGitUrl(self, project):
return self.connection.getGitUrl(project)
diff --git a/zuul/driver/github/githubconnection.py b/zuul/driver/github/githubconnection.py
index 48603a0..616e774 100644
--- a/zuul/driver/github/githubconnection.py
+++ b/zuul/driver/github/githubconnection.py
@@ -326,25 +326,26 @@
self._data = None
def __getitem__(self, key):
- if self._data is None:
- self._data = self._init_data()
+ self._init_data()
return self._data[key]
def __iter__(self):
+ self._init_data()
return iter(self._data)
def __len__(self):
+ self._init_data()
return len(self._data)
def _init_data(self):
- user = self._github.user(self._username)
- log_rate_limit(self.log, self._github)
- data = {
- 'username': user.login,
- 'name': user.name,
- 'email': user.email
- }
- return data
+ if self._data is None:
+ user = self._github.user(self._username)
+ log_rate_limit(self.log, self._github)
+ self._data = {
+ 'username': user.login,
+ 'name': user.name,
+ 'email': user.email
+ }
class GithubConnection(BaseConnection):
@@ -697,11 +698,21 @@
def addProject(self, project):
self.projects[project.name] = project
- def getProjectBranches(self, project):
+ def getProjectBranches(self, project, tenant):
+
+ # Evaluate if unprotected branches should be excluded or not. The first
+ # match wins. The order is project -> tenant (default is false).
+ project_config = tenant.project_configs.get(project.canonical_name)
+ if project_config.exclude_unprotected_branches is not None:
+ exclude_unprotected = project_config.exclude_unprotected_branches
+ else:
+ exclude_unprotected = tenant.exclude_unprotected_branches
+
github = self.getGithubClient()
owner, proj = project.name.split('/')
repository = github.repository(owner, proj)
- branches = [branch.name for branch in repository.branches()]
+ branches = [branch.name for branch in repository.branches(
+ protected=exclude_unprotected)]
log_rate_limit(self.log, github)
return branches
diff --git a/zuul/driver/github/githubsource.py b/zuul/driver/github/githubsource.py
index 1bd280f..1e7e07a 100644
--- a/zuul/driver/github/githubsource.py
+++ b/zuul/driver/github/githubsource.py
@@ -68,8 +68,8 @@
self.connection.addProject(p)
return p
- def getProjectBranches(self, project):
- return self.connection.getProjectBranches(project)
+ def getProjectBranches(self, project, tenant):
+ return self.connection.getProjectBranches(project, tenant)
def getProjectOpenChanges(self, project):
"""Get the open changes for a project."""
diff --git a/zuul/driver/timer/__init__.py b/zuul/driver/timer/__init__.py
index 4489808..69cd508 100644
--- a/zuul/driver/timer/__init__.py
+++ b/zuul/driver/timer/__init__.py
@@ -81,7 +81,7 @@
def _onTrigger(self, tenant, pipeline_name, timespec):
for project_name in tenant.layout.project_configs.keys():
(trusted, project) = tenant.getProject(project_name)
- for branch in project.source.getProjectBranches(project):
+ for branch in project.source.getProjectBranches(project, tenant):
event = TimerTriggerEvent()
event.type = 'timer'
event.timespec = timespec
diff --git a/zuul/model.py b/zuul/model.py
index 9be8745..26a7963 100644
--- a/zuul/model.py
+++ b/zuul/model.py
@@ -2091,6 +2091,10 @@
self.load_classes = set()
self.shadow_projects = set()
+ # The tenant's default setting of exclude_unprotected_branches will
+ # be overridden by this one if not None.
+ self.exclude_unprotected_branches = None
+
class ProjectConfig(object):
# Represents a project cofiguration
@@ -2451,6 +2455,7 @@
def __init__(self, name):
self.name = name
self.max_nodes_per_job = 5
+ self.exclude_unprotected_branches = False
self.layout = None
# The unparsed configuration from the main zuul config for
# this tenant.
diff --git a/zuul/source/__init__.py b/zuul/source/__init__.py
index b37aeb4..0396aff 100644
--- a/zuul/source/__init__.py
+++ b/zuul/source/__init__.py
@@ -64,7 +64,7 @@
"""Get a project."""
@abc.abstractmethod
- def getProjectBranches(self, project):
+ def getProjectBranches(self, project, tenant):
"""Get branches for a project"""
@abc.abstractmethod
diff --git a/zuul/sphinx/zuul.py b/zuul/sphinx/zuul.py
index b4133d7..7946074 100644
--- a/zuul/sphinx/zuul.py
+++ b/zuul/sphinx/zuul.py
@@ -25,18 +25,18 @@
class ZuulConfigObject(ObjectDescription):
object_names = {
'attr': 'attribute',
+ 'var': 'variable',
}
def get_path(self):
- attr_path = self.env.ref_context.get('zuul:attr_path', [])
- path = []
- if attr_path:
- path.extend(attr_path)
- return path
+ return self.env.ref_context.get('zuul:attr_path', [])
+
+ def get_display_path(self):
+ return self.env.ref_context.get('zuul:display_attr_path', [])
@property
def parent_pathname(self):
- return '.'.join(self.get_path())
+ return '.'.join(self.get_display_path())
@property
def full_pathname(self):
@@ -81,14 +81,19 @@
def before_content(self):
path = self.env.ref_context.setdefault('zuul:attr_path', [])
path.append(self.names[-1])
+ path = self.env.ref_context.setdefault('zuul:display_attr_path', [])
+ path.append(self.names[-1])
def after_content(self):
path = self.env.ref_context.get('zuul:attr_path')
if path:
path.pop()
+ path = self.env.ref_context.get('zuul:display_attr_path')
+ if path:
+ path.pop()
def handle_signature(self, sig, signode):
- path = self.get_path()
+ path = self.get_display_path()
signode['is_multiline'] = True
line = addnodes.desc_signature_line()
line['add_permalink'] = True
@@ -115,6 +120,50 @@
return sig
+class ZuulVarDirective(ZuulConfigObject):
+ has_content = True
+
+ option_spec = {
+ 'type': lambda x: x,
+ 'hidden': lambda x: x,
+ }
+
+ type_map = {
+ 'list': '[]',
+ 'dict': '{}',
+ }
+
+ def get_type_str(self):
+ if 'type' in self.options:
+ return self.type_map[self.options['type']]
+ return ''
+
+ def before_content(self):
+ path = self.env.ref_context.setdefault('zuul:attr_path', [])
+ element = self.names[-1]
+ path.append(element)
+ path = self.env.ref_context.setdefault('zuul:display_attr_path', [])
+ element = self.names[-1] + self.get_type_str()
+ path.append(element)
+
+ def after_content(self):
+ path = self.env.ref_context.get('zuul:attr_path')
+ if path:
+ path.pop()
+ path = self.env.ref_context.get('zuul:display_attr_path')
+ if path:
+ path.pop()
+
+ def handle_signature(self, sig, signode):
+ if 'hidden' in self.options:
+ return sig
+ path = self.get_display_path()
+ for x in path:
+ signode += addnodes.desc_addname(x + '.', x + '.')
+ signode += addnodes.desc_name(sig, sig)
+ return sig
+
+
class ZuulDomain(Domain):
name = 'zuul'
label = 'Zuul'
@@ -122,6 +171,7 @@
directives = {
'attr': ZuulAttrDirective,
'value': ZuulValueDirective,
+ 'var': ZuulVarDirective,
}
roles = {
@@ -129,6 +179,8 @@
warn_dangling=True),
'value': XRefRole(innernodeclass=nodes.inline, # type: ignore
warn_dangling=True),
+ 'var': XRefRole(innernodeclass=nodes.inline, # type: ignore
+ warn_dangling=True),
}
initial_data = {