Merge "Add stackdumphandler to zuul-web" into feature/zuulv3
diff --git a/doc/source/admin/components.rst b/doc/source/admin/components.rst
index b3c2e44..86b01ef 100644
--- a/doc/source/admin/components.rst
+++ b/doc/source/admin/components.rst
@@ -224,6 +224,11 @@
.. attr:: scheduler
+ .. attr:: command_socket
+ :default: /var/lib/zuul/scheduler.socket
+
+ Path to command socket file for the scheduler process.
+
.. attr:: tenant_config
:required:
@@ -282,6 +287,11 @@
.. attr:: merger
+ ,, attr:: command_socket
+ :default: /var/lib/zuul/merger.socket
+
+ Path to command socket file for the merger process.
+
.. attr:: git_dir
Directory in which Zuul should clone git repositories.
@@ -392,6 +402,11 @@
.. attr:: executor
+ .. attr:: command_socket
+ :default: /var/lib/zuul/executor.socket
+
+ Path to command socket file for the executor process.
+
.. attr:: finger_port
:default: 79
diff --git a/doc/source/admin/drivers/github.rst b/doc/source/admin/drivers/github.rst
index 7eebbdc..8dd7764 100644
--- a/doc/source/admin/drivers/github.rst
+++ b/doc/source/admin/drivers/github.rst
@@ -7,18 +7,95 @@
interact with the public GitHub service as well as site-local
installations of GitHub enterprise.
-.. TODO: make this section more user friendly
+Configure GitHub
+----------------
-Configure GitHub `webhook events
-<https://developer.github.com/webhooks/creating/>`_.
+There are two options currently available. GitHub's project owner can either
+manually setup web-hook or install a GitHub Application. In the first case,
+the project's owner needs to know the zuul endpoint and the webhook secrets.
-Set *Payload URL* to
-``http://<zuul-hostname>/connection/<connection-name>/payload``.
-Set *Content Type* to ``application/json``.
+Web-Hook
+........
+
+To configure a project's `webhook events <https://developer.github.com/webhooks/creating/>`_:
+
+* Set *Payload URL* to ``http://<zuul-hostname>/connection/<connection-name>/payload``.
+
+* Set *Content Type* to ``application/json``.
Select *Events* you are interested in. See below for the supported events.
+You will also need to have a GitHub user created for your zuul:
+
+* Zuul public key needs to be added to the GitHub account
+
+* A api_token needs to be created too, see this `article <See https://help.github.com/articles/creating-an-access-token-for-command-line-use/>`_
+
+Then in the zuul.conf, set webhook_token and api_token.
+
+Application
+...........
+
+To create a `GitHub application <https://developer.github.com/apps/building-integrations/setting-up-and-registering-github-apps/registering-github-apps/>`_:
+
+* Go to your organization settings page to create the application, e.g.: https://github.com/organizations/my-org/settings/apps/new
+
+* Set GitHub App name to "my-org-zuul"
+
+* Set Setup URL to your setup documentation, when user install the application they are redirected to this url
+
+* Set Webhook URL to ``http://<zuul-hostname>/connection/<connection-name>/payload``.
+
+* Create a Webhook secret
+
+* Set permissions:
+
+ * Commit statuses: Read & Write
+
+ * Issues: Read & Write
+
+ * Pull requests: Read & Write
+
+ * Repository contents: Read & Write (write to let zuul merge change)
+
+* Set events subscription:
+
+ * Label
+
+ * Status
+
+ * Issue comment
+
+ * Issues
+
+ * Pull request
+
+ * Pull request review
+
+ * Pull request review comment
+
+ * Commit comment
+
+ * Create
+
+ * Push
+
+ * Release
+
+* Set Where can this GitHub App be installed to "Any account"
+
+* Create the App
+
+* Generate a Private key in the app settings page
+
+Then in the zuul.conf, set webhook_token, app_id and app_key.
+After restarting zuul-scheduler, verify in the 'Advanced' tab that the
+Ping payload works (green tick and 200 response)
+
+Users can now install the application using its public page, e.g.: https://github.com/apps/my-org-zuul
+
+
Connection Configuration
------------------------
diff --git a/doc/source/user/config.rst b/doc/source/user/config.rst
index 3ea20ab..4151eda 100644
--- a/doc/source/user/config.rst
+++ b/doc/source/user/config.rst
@@ -798,13 +798,6 @@
are run after the parent's. See :ref:`job` for more
information.
- .. warning::
-
- If the path as specified does not exist, Zuul will try
- appending the extensions ``.yaml`` and ``.yml``. This
- behavior is deprecated and will be removed in the future all
- playbook paths should include the file extension.
-
.. attr:: post-run
The name of a playbook or list of playbooks to run after the
@@ -815,13 +808,6 @@
playbooks are run before the parent's. See :ref:`job` for more
information.
- .. warning::
-
- If the path as specified does not exist, Zuul will try
- appending the extensions ``.yaml`` and ``.yml``. This
- behavior is deprecated and will be removed in the future all
- playbook paths should include the file extension.
-
.. attr:: run
The name of the main playbook for this job. If it is not
@@ -833,13 +819,6 @@
run: playbooks/job-playbook.yaml
- .. warning::
-
- If the path as specified does not exist, Zuul will try
- appending the extensions ``.yaml`` and ``.yml``. This
- behavior is deprecated and will be removed in the future all
- playbook paths should include the file extension.
-
.. attr:: roles
A list of Ansible roles to prepare for the job. Because a job
@@ -1210,7 +1189,9 @@
label: controller-label
- name: compute1
label: compute-label
- - name: compute2
+ - name:
+ - compute2
+ - web
label: compute-label
groups:
- name: ceph-osd
@@ -1221,6 +1202,9 @@
- controller
- compute1
- compute2
+ - name: ceph-web
+ nodes:
+ - web
.. attr:: nodeset
@@ -1242,6 +1226,9 @@
The name of the node. This will appear in the Ansible inventory
for the job.
+ This can also be as a list of strings. If so, then the list of hosts in
+ the Ansible inventory will share a common ansible_host address.
+
.. attr:: label
:required:
diff --git a/doc/source/user/jobs.rst b/doc/source/user/jobs.rst
index 989338a..278c4f4 100644
--- a/doc/source/user/jobs.rst
+++ b/doc/source/user/jobs.rst
@@ -220,14 +220,15 @@
`src/git.example.com/org/project`.
.. var:: projects
- :type: list
+ :type: dict
- A list of all projects prepared by Zuul for the item. It
+ A dictionary of all projects prepared by Zuul for the item. It
includes, at least, the item's own project. It also includes
the projects of any items this item depends on, as well as the
projects that appear in :attr:`job.required-projects`.
- This is a list of dictionaries, with each element consisting of:
+ This is a dictionary of dictionaries. Each value has a key of
+ the `canonical_name`, then each entry consists of:
.. var:: name
@@ -264,6 +265,20 @@
This may be influenced by the branch or tag associated with
the item as well as the job configuration.
+ For example, to access the source directory of a single known
+ project, you might use::
+
+ {{ zuul.projects['git.example.com/org/project'].src_dir }}
+
+ To iterate over the project list, you might write a task
+ something like::
+
+ - name: Sample project iteration
+ debug:
+ msg: "Project {{ item.name }} is at {{ item.src_dir }}
+ with_items: {{ zuul.projects.values() | list }}
+
+
.. var:: _projects
:type: dict
diff --git a/etc/status/public_html/zuul.app.js b/etc/status/public_html/zuul.app.js
index 7ceb2dd..bf90a4d 100644
--- a/etc/status/public_html/zuul.app.js
+++ b/etc/status/public_html/zuul.app.js
@@ -28,8 +28,6 @@
function zuul_build_dom($, container) {
// Build a default-looking DOM
var default_layout = '<div class="container">'
- + '<h1>Zuul Status</h1>'
- + '<p>Real-time status monitor of Zuul, the pipeline manager between Gerrit and Workers.</p>'
+ '<div class="zuul-container" id="zuul-container">'
+ '<div style="display: none;" class="alert" id="zuul_msg"></div>'
+ '<button class="btn pull-right zuul-spinner">updating <span class="glyphicon glyphicon-refresh"></span></button>'
diff --git a/playbooks/zuul-stream/templates/ansible.cfg.j2 b/playbooks/zuul-stream/templates/ansible.cfg.j2
index 24f459e..41ffc0c 100644
--- a/playbooks/zuul-stream/templates/ansible.cfg.j2
+++ b/playbooks/zuul-stream/templates/ansible.cfg.j2
@@ -1,5 +1,5 @@
[defaults]
-hostfile = {{ ansible_user_dir }}/inventory.yaml
+inventory = {{ ansible_user_dir }}/inventory.yaml
gathering = smart
gather_subset = !all
lookup_plugins = {{ ansible_user_dir }}/src/git.openstack.org/openstack-infra/zuul/zuul/ansible/lookup
diff --git a/tests/base.py b/tests/base.py
index f274ed6..ea01d20 100755
--- a/tests/base.py
+++ b/tests/base.py
@@ -1435,7 +1435,7 @@
host['host_vars']['ansible_connection'] = 'local'
hosts.append(dict(
- name='localhost',
+ name=['localhost'],
host_vars=dict(ansible_connection='local'),
host_keys=[]))
return hosts
@@ -2066,10 +2066,16 @@
FIXTURE_DIR,
self.config.get('scheduler', 'tenant_config')))
self.config.set('scheduler', 'state_dir', self.state_root)
+ self.config.set(
+ 'scheduler', 'command_socket',
+ os.path.join(self.test_root, 'scheduler.socket'))
self.config.set('merger', 'git_dir', self.merger_src_root)
self.config.set('executor', 'git_dir', self.executor_src_root)
self.config.set('executor', 'private_key_file', self.private_key_file)
self.config.set('executor', 'state_dir', self.executor_state_root)
+ self.config.set(
+ 'executor', 'command_socket',
+ os.path.join(self.test_root, 'executor.socket'))
self.statsd = FakeStatsd()
if self.config.has_section('statsd'):
@@ -2256,13 +2262,13 @@
branch='master', tag='init')
if 'job' in item:
if 'run' in item['job']:
- files['%s.yaml' % item['job']['run']] = ''
+ files['%s' % item['job']['run']] = ''
for fn in zuul.configloader.as_list(
item['job'].get('pre-run', [])):
- files['%s.yaml' % fn] = ''
+ files['%s' % fn] = ''
for fn in zuul.configloader.as_list(
item['job'].get('post-run', [])):
- files['%s.yaml' % fn] = ''
+ files['%s' % fn] = ''
root = os.path.join(self.test_root, "config")
if not os.path.exists(root):
diff --git a/tests/fixtures/config/ansible/git/common-config/zuul.yaml b/tests/fixtures/config/ansible/git/common-config/zuul.yaml
index 28bfce1..d0a8f7b 100644
--- a/tests/fixtures/config/ansible/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/ansible/git/common-config/zuul.yaml
@@ -129,10 +129,10 @@
parent: base-urls
name: hello
run: playbooks/hello-post.yaml
- post-run: playbooks/hello-post
+ post-run: playbooks/hello-post.yaml
- job:
parent: python27
name: failpost
run: playbooks/post-broken.yaml
- post-run: playbooks/post-broken
+ post-run: playbooks/post-broken.yaml
diff --git a/tests/fixtures/config/branch-variants/git/project-config/zuul.yaml b/tests/fixtures/config/branch-variants/git/project-config/zuul.yaml
index 161e5a1..48da2d4 100644
--- a/tests/fixtures/config/branch-variants/git/project-config/zuul.yaml
+++ b/tests/fixtures/config/branch-variants/git/project-config/zuul.yaml
@@ -34,10 +34,10 @@
- job:
name: base
parent: null
- pre-run: playbooks/base/pre
+ pre-run: playbooks/base/pre.yaml
post-run:
- - playbooks/base/post-ssh
- - playbooks/base/post-logs
+ - playbooks/base/post-ssh.yaml
+ - playbooks/base/post-logs.yaml
- project:
name: project-config
diff --git a/tests/fixtures/config/branch-variants/git/puppet-integration/.zuul.yaml b/tests/fixtures/config/branch-variants/git/puppet-integration/.zuul.yaml
index 322927f..7e9cbc3 100644
--- a/tests/fixtures/config/branch-variants/git/puppet-integration/.zuul.yaml
+++ b/tests/fixtures/config/branch-variants/git/puppet-integration/.zuul.yaml
@@ -1,16 +1,16 @@
- job:
name: puppet-base
- pre-run: playbooks/prepare-node-common
+ pre-run: playbooks/prepare-node-common.yaml
- job:
name: puppet-module-base
parent: puppet-base
- pre-run: playbooks/prepare-node-unit
+ pre-run: playbooks/prepare-node-unit.yaml
- job:
name: puppet-lint
parent: puppet-module-base
- run: playbooks/run-lint
+ run: playbooks/run-lint.yaml
tags:
- master
diff --git a/tests/fixtures/config/branch-variants/git/puppet-integration/stable.zuul.yaml b/tests/fixtures/config/branch-variants/git/puppet-integration/stable.zuul.yaml
index 4701b80..74704a0 100644
--- a/tests/fixtures/config/branch-variants/git/puppet-integration/stable.zuul.yaml
+++ b/tests/fixtures/config/branch-variants/git/puppet-integration/stable.zuul.yaml
@@ -1,16 +1,16 @@
- job:
name: puppet-base
- pre-run: playbooks/prepare-node-common
+ pre-run: playbooks/prepare-node-common.yaml
- job:
name: puppet-module-base
parent: puppet-base
- pre-run: playbooks/prepare-node-unit
+ pre-run: playbooks/prepare-node-unit.yaml
- job:
name: puppet-lint
parent: puppet-module-base
- run: playbooks/run-lint
+ run: playbooks/run-lint.yaml
tags:
- stable
diff --git a/tests/fixtures/config/inventory/git/common-config/zuul.yaml b/tests/fixtures/config/inventory/git/common-config/zuul.yaml
index 74ddf2d..ad530a7 100644
--- a/tests/fixtures/config/inventory/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/inventory/git/common-config/zuul.yaml
@@ -52,6 +52,16 @@
run: playbooks/single-inventory.yaml
- job:
+ name: single-inventory-list
+ nodeset:
+ nodes:
+ - name:
+ - compute
+ - controller
+ label: ubuntu-xenial
+ run: playbooks/single-inventory.yaml
+
+- job:
name: group-inventory
nodeset: nodeset1
run: playbooks/group-inventory.yaml
diff --git a/tests/fixtures/config/inventory/git/org_project/.zuul.yaml b/tests/fixtures/config/inventory/git/org_project/.zuul.yaml
index 1a8bf5d..6a29049 100644
--- a/tests/fixtures/config/inventory/git/org_project/.zuul.yaml
+++ b/tests/fixtures/config/inventory/git/org_project/.zuul.yaml
@@ -3,5 +3,6 @@
check:
jobs:
- single-inventory
+ - single-inventory-list
- group-inventory
- hostvars-inventory
diff --git a/tests/fixtures/config/job-output/git/common-config/zuul.yaml b/tests/fixtures/config/job-output/git/common-config/zuul.yaml
index 4df0020..9373038 100644
--- a/tests/fixtures/config/job-output/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/job-output/git/common-config/zuul.yaml
@@ -23,8 +23,8 @@
- job:
name: job-output-failure
- run: playbooks/job-output
- post-run: playbooks/job-output-failure-post
+ run: playbooks/job-output.yaml
+ post-run: playbooks/job-output-failure-post.yaml
- project:
name: org/project
diff --git a/tests/fixtures/config/post-playbook/git/common-config/zuul.yaml b/tests/fixtures/config/post-playbook/git/common-config/zuul.yaml
index 16d7dee..b00d4c2 100644
--- a/tests/fixtures/config/post-playbook/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/post-playbook/git/common-config/zuul.yaml
@@ -18,8 +18,8 @@
- job:
name: python27
- pre-run: playbooks/pre
- post-run: playbooks/post
+ pre-run: playbooks/pre.yaml
+ post-run: playbooks/post.yaml
vars:
waitpath: '{{zuul._test.test_root}}/{{zuul.build}}/test_wait'
run: playbooks/python27.yaml
diff --git a/tests/fixtures/config/pre-playbook/git/common-config/zuul.yaml b/tests/fixtures/config/pre-playbook/git/common-config/zuul.yaml
index 7817745..16f48b1 100644
--- a/tests/fixtures/config/pre-playbook/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/pre-playbook/git/common-config/zuul.yaml
@@ -18,6 +18,6 @@
- job:
name: python27
- pre-run: playbooks/pre
- post-run: playbooks/post
+ pre-run: playbooks/pre.yaml
+ post-run: playbooks/post.yaml
run: playbooks/python27.yaml
diff --git a/tests/fixtures/config/tenant-parser/git/common-config/zuul.yaml b/tests/fixtures/config/tenant-parser/git/common-config/zuul.yaml
index e21f967..a28ef54 100644
--- a/tests/fixtures/config/tenant-parser/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/tenant-parser/git/common-config/zuul.yaml
@@ -18,8 +18,10 @@
- job:
name: common-config-job
+# Use the canonical name here. This should be merged with the org/project1 in
+# the other repo.
- project:
- name: org/project1
+ name: review.example.com/org/project1
check:
jobs:
- common-config-job
diff --git a/tests/unit/test_executor.py b/tests/unit/test_executor.py
index 5d27663..474859d 100755
--- a/tests/unit/test_executor.py
+++ b/tests/unit/test_executor.py
@@ -416,15 +416,15 @@
job)
def test_getHostList_host_keys(self):
- # Test without ssh_port set
+ # Test without connection_port set
node = {'name': 'fake-host',
'host_keys': ['fake-host-key'],
'interface_ip': 'localhost'}
keys = self.test_job.getHostList({'nodes': [node]})[0]['host_keys']
self.assertEqual(keys[0], 'localhost fake-host-key')
- # Test with custom ssh_port set
- node['ssh_port'] = 22022
+ # Test with custom connection_port set
+ node['connection_port'] = 22022
keys = self.test_job.getHostList({'nodes': [node]})[0]['host_keys']
self.assertEqual(keys[0], '[localhost]:22022 fake-host-key')
diff --git a/tests/unit/test_inventory.py b/tests/unit/test_inventory.py
index 04dcb05..1c41f5f 100644
--- a/tests/unit/test_inventory.py
+++ b/tests/unit/test_inventory.py
@@ -57,6 +57,26 @@
self.executor_server.release()
self.waitUntilSettled()
+ def test_single_inventory_list(self):
+
+ inventory = self._get_build_inventory('single-inventory-list')
+
+ all_nodes = ('compute', 'controller')
+ self.assertIn('all', inventory)
+ self.assertIn('hosts', inventory['all'])
+ self.assertIn('vars', inventory['all'])
+ for node_name in all_nodes:
+ self.assertIn(node_name, inventory['all']['hosts'])
+ self.assertIn('zuul', inventory['all']['vars'])
+ z_vars = inventory['all']['vars']['zuul']
+ self.assertIn('executor', z_vars)
+ self.assertIn('src_root', z_vars['executor'])
+ self.assertIn('job', z_vars)
+ self.assertEqual(z_vars['job'], 'single-inventory-list')
+
+ self.executor_server.release()
+ self.waitUntilSettled()
+
def test_group_inventory(self):
inventory = self._get_build_inventory('group-inventory')
diff --git a/tests/unit/test_nodepool.py b/tests/unit/test_nodepool.py
index d3f9ddb..aa0f082 100644
--- a/tests/unit/test_nodepool.py
+++ b/tests/unit/test_nodepool.py
@@ -67,8 +67,8 @@
# Test a simple node request
nodeset = model.NodeSet()
- nodeset.addNode(model.Node('controller', 'ubuntu-xenial'))
- nodeset.addNode(model.Node('compute', 'ubuntu-xenial'))
+ nodeset.addNode(model.Node(['controller', 'foo'], 'ubuntu-xenial'))
+ nodeset.addNode(model.Node(['compute'], 'ubuntu-xenial'))
job = model.Job('testjob')
job.nodeset = nodeset
request = self.nodepool.requestNodes(None, job)
@@ -99,8 +99,8 @@
# Test that node requests are re-submitted after disconnect
nodeset = model.NodeSet()
- nodeset.addNode(model.Node('controller', 'ubuntu-xenial'))
- nodeset.addNode(model.Node('compute', 'ubuntu-xenial'))
+ nodeset.addNode(model.Node(['controller'], 'ubuntu-xenial'))
+ nodeset.addNode(model.Node(['compute'], 'ubuntu-xenial'))
job = model.Job('testjob')
job.nodeset = nodeset
self.fake_nodepool.paused = True
@@ -116,8 +116,8 @@
# Test that node requests can be canceled
nodeset = model.NodeSet()
- nodeset.addNode(model.Node('controller', 'ubuntu-xenial'))
- nodeset.addNode(model.Node('compute', 'ubuntu-xenial'))
+ nodeset.addNode(model.Node(['controller'], 'ubuntu-xenial'))
+ nodeset.addNode(model.Node(['compute'], 'ubuntu-xenial'))
job = model.Job('testjob')
job.nodeset = nodeset
self.fake_nodepool.paused = True
@@ -131,8 +131,8 @@
# Test that a resubmitted request would not lock nodes
nodeset = model.NodeSet()
- nodeset.addNode(model.Node('controller', 'ubuntu-xenial'))
- nodeset.addNode(model.Node('compute', 'ubuntu-xenial'))
+ nodeset.addNode(model.Node(['controller'], 'ubuntu-xenial'))
+ nodeset.addNode(model.Node(['compute'], 'ubuntu-xenial'))
job = model.Job('testjob')
job.nodeset = nodeset
request = self.nodepool.requestNodes(None, job)
@@ -152,8 +152,8 @@
# Test that a lost request would not lock nodes
nodeset = model.NodeSet()
- nodeset.addNode(model.Node('controller', 'ubuntu-xenial'))
- nodeset.addNode(model.Node('compute', 'ubuntu-xenial'))
+ nodeset.addNode(model.Node(['controller'], 'ubuntu-xenial'))
+ nodeset.addNode(model.Node(['compute'], 'ubuntu-xenial'))
job = model.Job('testjob')
job.nodeset = nodeset
request = self.nodepool.requestNodes(None, job)
diff --git a/tests/unit/test_v3.py b/tests/unit/test_v3.py
index 54cf111..b9c9b32 100755
--- a/tests/unit/test_v3.py
+++ b/tests/unit/test_v3.py
@@ -1935,8 +1935,8 @@
name: parent
roles:
- zuul: bare-role
- pre-run: playbooks/parent-pre
- post-run: playbooks/parent-post
+ pre-run: playbooks/parent-pre.yaml
+ post-run: playbooks/parent-post.yaml
- job:
name: project-test
diff --git a/tools/test-logs.sh b/tools/test-logs.sh
index bf2147d..a514dd8 100644
--- a/tools/test-logs.sh
+++ b/tools/test-logs.sh
@@ -42,7 +42,7 @@
cat >$WORK_DIR/ansible.cfg <<EOF
[defaults]
-hostfile = $INVENTORY
+inventory = $INVENTORY
gathering = smart
gather_subset = !all
fact_caching = jsonfile
diff --git a/tox.ini b/tox.ini
index 28d6000..5efc4c0 100644
--- a/tox.ini
+++ b/tox.ini
@@ -41,9 +41,6 @@
[testenv:venv]
commands = {posargs}
-[testenv:validate-layout]
-commands = zuul-server -c etc/zuul.conf-sample -t -l {posargs}
-
[testenv:nodepool]
setenv =
OS_TEST_PATH = ./tests/nodepool
diff --git a/zuul/ansible/callback/zuul_stream.py b/zuul/ansible/callback/zuul_stream.py
index 8845e9b..df28a57 100644
--- a/zuul/ansible/callback/zuul_stream.py
+++ b/zuul/ansible/callback/zuul_stream.py
@@ -150,7 +150,7 @@
buff += more
if buff:
self._log_streamline(
- host, line.decode("utf-8", "backslashreplace"))
+ host, buff.decode("utf-8", "backslashreplace"))
def _log_streamline(self, host, line):
if "[Zuul] Task exit code" in line:
diff --git a/zuul/ansible/library/zuul_return.py b/zuul/ansible/library/zuul_return.py
index 9f3332b..4935226 100644
--- a/zuul/ansible/library/zuul_return.py
+++ b/zuul/ansible/library/zuul_return.py
@@ -63,7 +63,7 @@
path = os.path.join(os.environ['ZUUL_JOBDIR'], 'work',
'results.json')
set_value(path, p['data'], p['file'])
- module.exit_json(changed=True, e=os.environ)
+ module.exit_json(changed=True, e=os.environ.copy())
from ansible.module_utils.basic import * # noqa
from ansible.module_utils.basic import AnsibleModule
diff --git a/zuul/cmd/__init__.py b/zuul/cmd/__init__.py
index e150f9c..236fd9f 100755
--- a/zuul/cmd/__init__.py
+++ b/zuul/cmd/__init__.py
@@ -23,6 +23,7 @@
import logging.config
import os
import signal
+import socket
import sys
import traceback
import threading
@@ -184,3 +185,12 @@
pass
with daemon.DaemonContext(pidfile=pid):
self.run()
+
+ def send_command(self, cmd):
+ command_socket = get_default(
+ self.config, self.app_name, 'command_socket',
+ '/var/lib/zuul/%s.socket' % self.app_name)
+ s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+ s.connect(command_socket)
+ cmd = '%s\n' % cmd
+ s.sendall(cmd.encode('utf8'))
diff --git a/zuul/cmd/executor.py b/zuul/cmd/executor.py
index aef8c95..ade9715 100755
--- a/zuul/cmd/executor.py
+++ b/zuul/cmd/executor.py
@@ -18,7 +18,6 @@
import logging
import os
import pwd
-import socket
import sys
import signal
import tempfile
@@ -52,15 +51,6 @@
if self.args.command:
self.args.nodaemon = True
- def send_command(self, cmd):
- state_dir = get_default(self.config, 'executor', 'state_dir',
- '/var/lib/zuul', expand_user=True)
- path = os.path.join(state_dir, 'executor.socket')
- s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
- s.connect(path)
- cmd = '%s\n' % cmd
- s.sendall(cmd.encode('utf8'))
-
def exit_handler(self):
self.executor.stop()
self.executor.join()
diff --git a/zuul/cmd/merger.py b/zuul/cmd/merger.py
index 56b6b44..7db1bee 100755
--- a/zuul/cmd/merger.py
+++ b/zuul/cmd/merger.py
@@ -15,8 +15,10 @@
# under the License.
import signal
+import sys
import zuul.cmd
+import zuul.merger.server
# No zuul imports here because they pull in paramiko which must not be
# imported until after the daemonization.
@@ -28,14 +30,28 @@
app_name = 'merger'
app_description = 'A standalone Zuul merger.'
- def exit_handler(self, signum, frame):
- signal.signal(signal.SIGUSR1, signal.SIG_IGN)
+ def createParser(self):
+ parser = super(Merger, self).createParser()
+ parser.add_argument('command',
+ choices=zuul.merger.server.COMMANDS,
+ nargs='?')
+ return parser
+
+ def parseArguments(self, args=None):
+ super(Merger, self).parseArguments()
+ if self.args.command:
+ self.args.nodaemon = True
+
+ def exit_handler(self):
self.merger.stop()
self.merger.join()
def run(self):
# See comment at top of file about zuul imports
import zuul.merger.server
+ if self.args.command in zuul.merger.server.COMMANDS:
+ self.send_command(self.args.command)
+ sys.exit(0)
self.configure_connections(source_only=True)
@@ -45,14 +61,18 @@
self.connections)
self.merger.start()
- signal.signal(signal.SIGUSR1, self.exit_handler)
signal.signal(signal.SIGUSR2, zuul.cmd.stack_dump_handler)
- while True:
- try:
- signal.pause()
- except KeyboardInterrupt:
- print("Ctrl + C: asking merger to exit nicely...\n")
- self.exit_handler(signal.SIGINT, None)
+
+ if self.args.nodaemon:
+ while True:
+ try:
+ signal.pause()
+ except KeyboardInterrupt:
+ print("Ctrl + C: asking merger to exit nicely...\n")
+ self.exit_handler()
+ sys.exit(0)
+ else:
+ self.merger.join()
def main():
diff --git a/zuul/cmd/scheduler.py b/zuul/cmd/scheduler.py
index 539d55b..7722d6e 100755
--- a/zuul/cmd/scheduler.py
+++ b/zuul/cmd/scheduler.py
@@ -22,6 +22,7 @@
import zuul.cmd
from zuul.lib.config import get_default
from zuul.lib.statsd import get_statsd_config
+import zuul.scheduler
# No zuul imports here because they pull in paramiko which must not be
# imported until after the daemonization.
@@ -37,6 +38,18 @@
super(Scheduler, self).__init__()
self.gear_server_pid = None
+ def createParser(self):
+ parser = super(Scheduler, self).createParser()
+ parser.add_argument('command',
+ choices=zuul.scheduler.COMMANDS,
+ nargs='?')
+ return parser
+
+ def parseArguments(self, args=None):
+ super(Scheduler, self).parseArguments()
+ if self.args.command:
+ self.args.nodaemon = True
+
def reconfigure_handler(self, signum, frame):
signal.signal(signal.SIGHUP, signal.SIG_IGN)
self.log.debug("Reconfiguration triggered")
@@ -48,8 +61,7 @@
self.log.exception("Reconfiguration failed:")
signal.signal(signal.SIGHUP, self.reconfigure_handler)
- def exit_handler(self, signum, frame):
- signal.signal(signal.SIGUSR1, signal.SIG_IGN)
+ def exit_handler(self):
self.sched.exit()
self.sched.join()
self.stop_gear_server()
@@ -104,6 +116,10 @@
def run(self):
# See comment at top of file about zuul imports
import zuul.scheduler
+ if self.args.command in zuul.scheduler.COMMANDS:
+ self.send_command(self.args.command)
+ sys.exit(0)
+ # See comment at top of file about zuul imports
import zuul.executor.client
import zuul.merger.client
import zuul.nodepool
@@ -162,14 +178,17 @@
webapp.start()
signal.signal(signal.SIGHUP, self.reconfigure_handler)
- signal.signal(signal.SIGUSR1, self.exit_handler)
- signal.signal(signal.SIGTERM, self.term_handler)
- while True:
- try:
- signal.pause()
- except KeyboardInterrupt:
- print("Ctrl + C: asking scheduler to exit nicely...\n")
- self.exit_handler(signal.SIGINT, None)
+
+ if self.args.nodaemon:
+ while True:
+ try:
+ signal.pause()
+ except KeyboardInterrupt:
+ print("Ctrl + C: asking scheduler to exit nicely...\n")
+ self.exit_handler()
+ sys.exit(0)
+ else:
+ self.sched.join()
def main():
diff --git a/zuul/configloader.py b/zuul/configloader.py
index e034329..fb1695c 100644
--- a/zuul/configloader.py
+++ b/zuul/configloader.py
@@ -340,7 +340,7 @@
class NodeSetParser(object):
@staticmethod
def getSchema(anonymous=False):
- node = {vs.Required('name'): str,
+ node = {vs.Required('name'): to_list(str),
vs.Required('label'): str,
}
@@ -365,11 +365,13 @@
node_names = set()
group_names = set()
for conf_node in as_list(conf['nodes']):
- if conf_node['name'] in node_names:
- raise DuplicateNodeError(conf['name'], conf_node['name'])
- node = model.Node(conf_node['name'], conf_node['label'])
+ for name in as_list(conf_node['name']):
+ if name in node_names:
+ raise DuplicateNodeError(name, conf_node['name'])
+ node = model.Node(as_list(conf_node['name']), conf_node['label'])
ns.addNode(node)
- node_names.add(conf_node['name'])
+ for name in as_list(conf_node['name']):
+ node_names.add(name)
for conf_group in as_list(conf.get('groups', [])):
for node_name in as_list(conf_group['nodes']):
if node_name not in node_names:
@@ -1162,8 +1164,8 @@
tenant.config_projects,
tenant.untrusted_projects,
cached, tenant)
- unparsed_config.extend(tenant.config_projects_config)
- unparsed_config.extend(tenant.untrusted_projects_config)
+ unparsed_config.extend(tenant.config_projects_config, tenant=tenant)
+ unparsed_config.extend(tenant.untrusted_projects_config, tenant=tenant)
tenant.layout = TenantParser._parseLayout(base, tenant,
unparsed_config,
scheduler,
diff --git a/zuul/executor/client.py b/zuul/executor/client.py
index a8b94f0..06c2087 100644
--- a/zuul/executor/client.py
+++ b/zuul/executor/client.py
@@ -180,8 +180,7 @@
if (hasattr(item.change, 'newrev') and item.change.newrev
and item.change.newrev != '0' * 40):
zuul_params['newrev'] = item.change.newrev
- zuul_params['projects'] = [] # Set below
- zuul_params['_projects'] = {} # transitional to convert to dict
+ zuul_params['projects'] = {} # Set below
zuul_params['items'] = dependent_changes
params = dict()
@@ -253,7 +252,7 @@
params['projects'].append(make_project_dict(project))
projects.add(project)
for p in projects:
- zuul_params['_projects'][p.canonical_name] = (dict(
+ zuul_params['projects'][p.canonical_name] = (dict(
name=p.name,
short_name=p.name.split('/')[-1],
# Duplicate this into the dict too, so that iterating
@@ -265,12 +264,10 @@
))
# We are transitioning "projects" from a list to a dict
# indexed by canonical name, as it is much easier to access
- # values in ansible. Existing callers are converted to
- # "_projects", then once "projects" is unused we switch it,
- # then convert callers back. Finally when "_projects" is
- # unused it will be removed.
- for cn, p in zuul_params['_projects'].items():
- zuul_params['projects'].append(p)
+ # values in ansible. Existing callers have been converted to
+ # "_projects" and "projects" is swapped; we will convert users
+ # back to "projects" and remove this soon.
+ zuul_params['_projects'] = zuul_params['projects']
build = Build(job, uuid)
build.parameters = params
diff --git a/zuul/executor/server.py b/zuul/executor/server.py
index 016d0e6..7a93f89 100644
--- a/zuul/executor/server.py
+++ b/zuul/executor/server.py
@@ -497,7 +497,8 @@
hosts = {}
for node in nodes:
- hosts[node['name']] = node['host_vars']
+ for name in node['name']:
+ hosts[name] = node['host_vars']
inventory = {
'all': {
@@ -910,7 +911,7 @@
# results in the wrong thing being in interface_ip
# TODO(jeblair): Move this notice to the docs.
ip = node.get('interface_ip')
- port = node.get('ssh_port', 22)
+ port = node.get('connection_port', node.get('ssh_port', 22))
host_vars = dict(
ansible_host=ip,
ansible_user=self.executor_server.default_username,
@@ -958,13 +959,11 @@
"non-trusted repo." % (entry, path))
def findPlaybook(self, path, trusted=False):
- for ext in ['', '.yaml', '.yml']:
- fn = path + ext
- if os.path.exists(fn):
- if not trusted:
- playbook_dir = os.path.dirname(os.path.abspath(fn))
- self._blockPluginDirs(playbook_dir)
- return fn
+ if os.path.exists(path):
+ if not trusted:
+ playbook_dir = os.path.dirname(os.path.abspath(path))
+ self._blockPluginDirs(playbook_dir)
+ return path
raise ExecutorError("Unable to find playbook %s" % path)
def preparePlaybooks(self, args):
@@ -1187,7 +1186,7 @@
callback_path = self.executor_server.callback_dir
with open(jobdir_playbook.ansible_config, 'w') as config:
config.write('[defaults]\n')
- config.write('hostfile = %s\n' % self.jobdir.inventory)
+ config.write('inventory = %s\n' % self.jobdir.inventory)
config.write('local_tmp = %s/local_tmp\n' %
self.jobdir.ansible_cache_root)
config.write('retry_files_enabled = False\n')
@@ -1607,10 +1606,13 @@
self.merger = self._getMerger(self.merge_root)
self.update_queue = DeduplicateQueue()
+ command_socket = get_default(
+ self.config, 'executor', 'command_socket',
+ '/var/lib/zuul/executor.socket')
+ self.command_socket = commandsocket.CommandSocket(command_socket)
+
state_dir = get_default(self.config, 'executor', 'state_dir',
'/var/lib/zuul', expand_user=True)
- path = os.path.join(state_dir, 'executor.socket')
- self.command_socket = commandsocket.CommandSocket(path)
ansible_dir = os.path.join(state_dir, 'ansible')
self.ansible_dir = ansible_dir
if os.path.exists(ansible_dir):
diff --git a/zuul/merger/server.py b/zuul/merger/server.py
index 765d9e0..576d41e 100644
--- a/zuul/merger/server.py
+++ b/zuul/merger/server.py
@@ -19,10 +19,14 @@
import gear
+from zuul.lib import commandsocket
from zuul.lib.config import get_default
from zuul.merger import merger
+COMMANDS = ['stop']
+
+
class MergeServer(object):
log = logging.getLogger("zuul.MergeServer")
@@ -40,9 +44,16 @@
self.merger = merger.Merger(
merge_root, connections, merge_email, merge_name, speed_limit,
speed_time)
+ self.command_map = dict(
+ stop=self.stop)
+ command_socket = get_default(
+ self.config, 'merger', 'command_socket',
+ '/var/lib/zuul/merger.socket')
+ self.command_socket = commandsocket.CommandSocket(command_socket)
def start(self):
self._running = True
+ self._command_running = True
server = self.config.get('gearman', 'server')
port = get_default(self.config, 'gearman', 'port', 4730)
ssl_key = get_default(self.config, 'gearman', 'ssl_key')
@@ -54,6 +65,13 @@
self.worker.waitForServer()
self.log.debug("Registering")
self.register()
+ self.log.debug("Starting command processor")
+ self.command_socket.start()
+ self.command_thread = threading.Thread(
+ target=self.runCommand, name='command')
+ self.command_thread.daemon = True
+ self.command_thread.start()
+
self.log.debug("Starting worker")
self.thread = threading.Thread(target=self.run)
self.thread.daemon = True
@@ -67,12 +85,23 @@
def stop(self):
self.log.debug("Stopping")
self._running = False
+ self._command_running = False
+ self.command_socket.stop()
self.worker.shutdown()
self.log.debug("Stopped")
def join(self):
self.thread.join()
+ def runCommand(self):
+ while self._command_running:
+ try:
+ command = self.command_socket.get().decode('utf8')
+ if command != '_stop':
+ self.command_map[command]()
+ except Exception:
+ self.log.exception("Exception while processing command")
+
def run(self):
self.log.debug("Starting merge listener")
while self._running:
diff --git a/zuul/model.py b/zuul/model.py
index 6426bbe..56d08a1 100644
--- a/zuul/model.py
+++ b/zuul/model.py
@@ -383,7 +383,7 @@
self.public_ipv4 = None
self.private_ipv4 = None
self.public_ipv6 = None
- self.ssh_port = 22
+ self.connection_port = 22
self._keys = []
self.az = None
self.provider = None
@@ -498,9 +498,10 @@
return n
def addNode(self, node):
- if node.name in self.nodes:
- raise Exception("Duplicate node in %s" % (self,))
- self.nodes[node.name] = node
+ for name in node.name:
+ if name in self.nodes:
+ raise Exception("Duplicate node in %s" % (self,))
+ self.nodes[tuple(node.name)] = node
def getNodes(self):
return list(self.nodes.values())
@@ -2385,14 +2386,25 @@
r.semaphores = copy.deepcopy(self.semaphores)
return r
- def extend(self, conf):
+ def extend(self, conf, tenant=None):
if isinstance(conf, UnparsedTenantConfig):
self.pragmas.extend(conf.pragmas)
self.pipelines.extend(conf.pipelines)
self.jobs.extend(conf.jobs)
self.project_templates.extend(conf.project_templates)
for k, v in conf.projects.items():
- self.projects.setdefault(k, []).extend(v)
+ name = k
+ # If we have the tenant add the projects to
+ # the according canonical name instead of the given project
+ # name. If it is not found, it's ok to add this to the given
+ # name. We also don't need to throw the
+ # ProjectNotFoundException here as semantic validation occurs
+ # later where it will fail then.
+ if tenant is not None:
+ trusted, project = tenant.getProject(k)
+ if project is not None:
+ name = project.canonical_name
+ self.projects.setdefault(name, []).extend(v)
self.nodesets.extend(conf.nodesets)
self.secrets.extend(conf.secrets)
self.semaphores.extend(conf.semaphores)
diff --git a/zuul/scheduler.py b/zuul/scheduler.py
index 7dee00d..b978979 100644
--- a/zuul/scheduler.py
+++ b/zuul/scheduler.py
@@ -30,10 +30,13 @@
from zuul import exceptions
from zuul import version as zuul_version
from zuul import rpclistener
+from zuul.lib import commandsocket
from zuul.lib.config import get_default
from zuul.lib.statsd import get_statsd
import zuul.lib.queue
+COMMANDS = ['stop']
+
class ManagementEvent(object):
"""An event that should be processed within the main queue run loop"""
@@ -215,6 +218,9 @@
self.wake_event = threading.Event()
self.layout_lock = threading.Lock()
self.run_handler_lock = threading.Lock()
+ self.command_map = dict(
+ stop=self.stop,
+ )
self._pause = False
self._exit = False
self._stopped = False
@@ -243,6 +249,11 @@
time_dir = self._get_time_database_dir()
self.time_database = model.TimeDataBase(time_dir)
+ command_socket = get_default(
+ self.config, 'scheduler', 'command_socket',
+ '/var/lib/zuul/scheduler.socket')
+ self.command_socket = commandsocket.CommandSocket(command_socket)
+
self.zuul_version = zuul_version.version_info.release_string()
self.last_reconfigured = None
self.tenant_last_reconfigured = {}
@@ -250,6 +261,14 @@
def start(self):
super(Scheduler, self).start()
+ self._command_running = True
+ self.log.debug("Starting command processor")
+ self.command_socket.start()
+ self.command_thread = threading.Thread(target=self.runCommand,
+ name='command')
+ self.command_thread.daemon = True
+ self.command_thread.start()
+
self.rpc.start()
self.stats_thread.start()
@@ -261,6 +280,17 @@
self.stats_thread.join()
self.rpc.stop()
self.rpc.join()
+ self._command_running = False
+ self.command_socket.stop()
+
+ def runCommand(self):
+ while self._command_running:
+ try:
+ command = self.command_socket.get().decode('utf8')
+ if command != '_stop':
+ self.command_map[command]()
+ except Exception:
+ self.log.exception("Exception while processing command")
def registerConnections(self, connections, webapp, load=True):
# load: whether or not to trigger the onLoad for the connection. This
diff --git a/zuul/web/static/builds.html b/zuul/web/static/builds.html
index 921c9e2..5b9ba35 100644
--- a/zuul/web/static/builds.html
+++ b/zuul/web/static/builds.html
@@ -17,10 +17,10 @@
<html>
<head>
<title>Zuul Builds</title>
- <link rel="stylesheet" href="/static/bootstrap/css/bootstrap.min.css">
+ <link rel="stylesheet" href="../static/bootstrap/css/bootstrap.min.css">
<link rel="stylesheet" href="../static/styles/zuul.css" />
- <script src="/static/js/jquery.min.js"></script>
- <script src="/static/js/angular.min.js"></script>
+ <script src="../static/js/jquery.min.js"></script>
+ <script src="../static/js/angular.min.js"></script>
<script src="../static/javascripts/zuul.angular.js"></script>
</head>
<body ng-app="zuulBuilds" ng-controller="mainController"><div class="container-fluid">
@@ -55,12 +55,9 @@
<th>Project</th>
<th>Pipeline</th>
<th>Change</th>
- <th>Newrev</th>
<th>Duration</th>
<th>Log url</th>
- <th>Node name</th>
<th>Start time</th>
- <th>End time</th>
<th>Result</th>
</tr>
</thead>
@@ -71,12 +68,9 @@
<td>{{ build.project }}</td>
<td>{{ build.pipeline }}</td>
<td><a href="{{ build.ref_url }}" target="_self">change</a></td>
- <td>{{ build.newrev }}</td>
<td>{{ build.duration }} seconds</td>
<td><a ng-if="build.log_url" href="{{ build.log_url }}" target="_self">logs</a></td>
- <td>{{ build.node_name }}</td>
<td>{{ build.start_time }}</td>
- <td>{{ build.end_time }}</td>
<td>{{ build.result }}</td>
</tr>
</tbody>
diff --git a/zuul/web/static/index.html b/zuul/web/static/index.html
index 6747e66..d20a1ea 100644
--- a/zuul/web/static/index.html
+++ b/zuul/web/static/index.html
@@ -17,10 +17,10 @@
<html>
<head>
<title>Zuul Tenants</title>
- <link rel="stylesheet" href="/static/bootstrap/css/bootstrap.min.css">
+ <link rel="stylesheet" href="static/bootstrap/css/bootstrap.min.css">
<link rel="stylesheet" href="static/styles/zuul.css" />
- <script src="/static/js/jquery.min.js"></script>
- <script src="/static/js/angular.min.js"></script>
+ <script src="static/js/jquery.min.js"></script>
+ <script src="static/js/angular.min.js"></script>
<script src="static/javascripts/zuul.angular.js"></script>
</head>
<body ng-app="zuulTenants" ng-controller="mainController"><div class="container-fluid">
diff --git a/zuul/web/static/javascripts/zuul.app.js b/zuul/web/static/javascripts/zuul.app.js
index 7ceb2dd..bf90a4d 100644
--- a/zuul/web/static/javascripts/zuul.app.js
+++ b/zuul/web/static/javascripts/zuul.app.js
@@ -28,8 +28,6 @@
function zuul_build_dom($, container) {
// Build a default-looking DOM
var default_layout = '<div class="container">'
- + '<h1>Zuul Status</h1>'
- + '<p>Real-time status monitor of Zuul, the pipeline manager between Gerrit and Workers.</p>'
+ '<div class="zuul-container" id="zuul-container">'
+ '<div style="display: none;" class="alert" id="zuul_msg"></div>'
+ '<button class="btn pull-right zuul-spinner">updating <span class="glyphicon glyphicon-refresh"></span></button>'
diff --git a/zuul/web/static/jobs.html b/zuul/web/static/jobs.html
index 6946723..b27d882 100644
--- a/zuul/web/static/jobs.html
+++ b/zuul/web/static/jobs.html
@@ -17,10 +17,10 @@
<html>
<head>
<title>Zuul Builds</title>
- <link rel="stylesheet" href="/static/bootstrap/css/bootstrap.min.css">
+ <link rel="stylesheet" href="../static/bootstrap/css/bootstrap.min.css">
<link rel="stylesheet" href="../static/styles/zuul.css" />
- <script src="/static/js/jquery.min.js"></script>
- <script src="/static/js/angular.min.js"></script>
+ <script src="../static/js/jquery.min.js"></script>
+ <script src="../static/js/angular.min.js"></script>
<script src="../static/javascripts/zuul.angular.js"></script>
</head>
<body ng-app="zuulJobs" ng-controller="mainController"><div class="container-fluid">
diff --git a/zuul/web/static/status.html b/zuul/web/static/status.html
index 7cb9536..8471fd1 100644
--- a/zuul/web/static/status.html
+++ b/zuul/web/static/status.html
@@ -19,11 +19,11 @@
<html>
<head>
<title>Zuul Status</title>
- <link rel="stylesheet" href="/static/bootstrap/css/bootstrap.min.css">
+ <link rel="stylesheet" href="../static/bootstrap/css/bootstrap.min.css">
<link rel="stylesheet" href="../static/styles/zuul.css" />
- <script src="/static/js/jquery.min.js"></script>
- <script src="/static/js/jquery-visibility.min.js"></script>
- <script src="/static/js/jquery.graphite.min.js"></script>
+ <script src="../static/js/jquery.min.js"></script>
+ <script src="../static/js/jquery-visibility.min.js"></script>
+ <script src="../static/js/jquery.graphite.min.js"></script>
<script src="../static/javascripts/jquery.zuul.js"></script>
<script src="../static/javascripts/zuul.app.js"></script>
</head>