Merge "Add log streaming test" into feature/zuulv3
diff --git a/.gitignore b/.gitignore
index d6a7477..a2dd0a3 100644
--- a/.gitignore
+++ b/.gitignore
@@ -7,9 +7,12 @@
.testrepository
.tox
.venv
+.coverage
AUTHORS
build/*
ChangeLog
doc/build/*
zuul/versioninfo
dist/
+cover/
+htmlcov/
diff --git a/.zuul.yaml b/.zuul.yaml
index c21b30f..e8b070f 100644
--- a/.zuul.yaml
+++ b/.zuul.yaml
@@ -5,6 +5,5 @@
- tox-docs
- tox-cover
- tox-linters
- - tox-py27
- tox-py35
- tox-tarball
diff --git a/README.rst b/README.rst
index c55f7b3..16e7385 100644
--- a/README.rst
+++ b/README.rst
@@ -134,6 +134,16 @@
is too cryptic. In your own work, feel free to leave TODOv3 notes
if a change would otherwise become too large or unweildy.
+Python Version Support
+----------------------
+
+Zuul v3 requires Python 3. It does not support Python 2.
+
+As Ansible is used for the execution of jobs, it's important to note that
+while Ansible does support Python 3, not all of Ansible's modules do. Zuul
+currently sets ``ansible_python_interpreter`` to python2 so that remote
+content will be executed with Python2.
+
Roadmap
-------
diff --git a/bindep.txt b/bindep.txt
index 5db144b..8dffd0f 100644
--- a/bindep.txt
+++ b/bindep.txt
@@ -15,3 +15,4 @@
python-dev [platform:dpkg]
python-devel [platform:rpm]
bubblewrap [platform:rpm]
+redhat-rpm-config [platform:rpm]
diff --git a/doc/source/reporters.rst b/doc/source/reporters.rst
index dd053fa..ae6ab1c 100644
--- a/doc/source/reporters.rst
+++ b/doc/source/reporters.rst
@@ -31,10 +31,10 @@
GitHub
------
-Zuul reports back to GitHub pull requests via GitHub API.
-On success and failure, it creates a comment containing the build results.
-It also sets the status on start, success and failure. Status name and
-description is taken from the pipeline.
+Zuul reports back to GitHub via GitHub API. Available reports include a PR
+comment containing the build results, a commit status on start, success and
+failure, an issue label addition/removal on the PR, and a merge of the PR
+itself. Status name, description, and context is taken from the pipeline.
A :ref:`connection` that uses the github driver must be supplied to the
reporter. It has the following options:
@@ -51,22 +51,23 @@
**comment**
Boolean value (``true`` or ``false``) that determines if the reporter should
add a comment to the pipeline status to the github pull request. Defaults
- to ``true``.
+ to ``true``. Only used for Pull Request based events.
``comment: false``
**merge**
Boolean value (``true`` or ``false``) that determines if the reporter should
- merge the pull reqeust. Defaults to ``false``.
+ merge the pull reqeust. Defaults to ``false``. Only used for Pull Request based
+ events.
``merge=true``
**label**
List of strings each representing an exact label name which should be added
- to the pull request by reporter.
+ to the pull request by reporter. Only used for Pull Request based events.
``label: 'test successful'``
**unlabel**
List of strings each representing an exact label name which should be removed
- from the pull request by reporter.
+ from the pull request by reporter. Only used for Pull Request based events.
``unlabel: 'test failed'``
SMTP
diff --git a/doc/source/zuul.rst b/doc/source/zuul.rst
index a7dfb44..f07a859 100644
--- a/doc/source/zuul.rst
+++ b/doc/source/zuul.rst
@@ -52,6 +52,16 @@
Port on which the Gearman server is listening.
``port=4730`` (optional)
+**ssl_ca**
+ Optional: An openssl file containing a set of concatenated “certification authority” certificates
+ in PEM formet.
+
+**ssl_cert**
+ Optional: An openssl file containing the client public certificate in PEM format.
+
+**ssl_key**
+ Optional: An openssl file containing the client private key in PEM format.
+
gearman_server
""""""""""""""
@@ -70,6 +80,16 @@
Path to log config file for internal Gearman server.
``log_config=/etc/zuul/gearman-logging.yaml``
+**ssl_ca**
+ Optional: An openssl file containing a set of concatenated “certification authority” certificates
+ in PEM formet.
+
+**ssl_cert**
+ Optional: An openssl file containing the server public certificate in PEM format.
+
+**ssl_key**
+ Optional: An openssl file containing the server private key in PEM format.
+
webapp
""""""
@@ -128,11 +148,6 @@
optional value and ``1`` is used by default.
``status_expiry=1``
-**job_name_in_report**
- Boolean value (``true`` or ``false``) that indicates whether the
- job name should be included in the report (normally only the URL
- is included). Defaults to ``false``. Used by zuul-server only.
- ``job_name_in_report=true``
merger
""""""
diff --git a/etc/zuul.conf-sample b/etc/zuul.conf-sample
index 1065cec..2909ea6 100644
--- a/etc/zuul.conf-sample
+++ b/etc/zuul.conf-sample
@@ -1,8 +1,14 @@
[gearman]
server=127.0.0.1
+;ssl_ca=/path/to/ca.pem
+;ssl_cert=/path/to/client.pem
+;ssl_key=/path/to/client.key
[gearman_server]
start=true
+;ssl_ca=/path/to/ca.pem
+;ssl_cert=/path/to/server.pem
+;ssl_key=/path/to/server.key
[zuul]
layout_config=/etc/zuul/layout.yaml
@@ -20,6 +26,8 @@
[executor]
default_username=zuul
+trusted_ro_dirs=/opt/zuul-scripts:/var/cache
+trusted_rw_dirs=/opt/zuul-logs
[webapp]
listen_address=0.0.0.0
diff --git a/requirements.txt b/requirements.txt
index 746bbcb..5caa1b5 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -8,7 +8,6 @@
WebOb>=1.2.3
paramiko>=1.8.0,<2.0.0
GitPython>=0.3.3,<2.1.2
-ordereddict
python-daemon>=2.0.4,<2.1.0
extras
statsd>=1.0.0,<3.0
@@ -17,8 +16,7 @@
apscheduler>=3.0
PrettyTable>=0.6,<0.8
babel>=1.0
-six>=1.6.0
-ansible>=2.0.0.1
+ansible>=2.3.0.0
kazoo
sqlalchemy
alembic
diff --git a/setup.cfg b/setup.cfg
index 5ae0903..0d22cb1 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -12,9 +12,8 @@
License :: OSI Approved :: Apache Software License
Operating System :: POSIX :: Linux
Programming Language :: Python
- Programming Language :: Python :: 2
- Programming Language :: Python :: 2.7
- Programming Language :: Python :: 2.6
+ Programming Language :: Python :: 3
+ Programming Language :: Python :: 3.5
[pbr]
warnerrors = True
diff --git a/tests/base.py b/tests/base.py
index 7d33ffc..ff1f531 100755
--- a/tests/base.py
+++ b/tests/base.py
@@ -15,24 +15,20 @@
# License for the specific language governing permissions and limitations
# under the License.
-from six.moves import configparser as ConfigParser
+import configparser
import datetime
import gc
import hashlib
+import importlib
+from io import StringIO
import json
import logging
import os
-from six.moves import queue as Queue
-from six.moves import urllib
+import queue
import random
import re
import select
import shutil
-from six.moves import reload_module
-try:
- from cStringIO import StringIO
-except Exception:
- from six import StringIO
import socket
import string
import subprocess
@@ -42,6 +38,7 @@
import traceback
import time
import uuid
+import urllib
import git
@@ -463,7 +460,7 @@
super(FakeGerritConnection, self).__init__(driver, connection_name,
connection_config)
- self.event_queue = Queue.Queue()
+ self.event_queue = queue.Queue()
self.fixture_dir = os.path.join(FIXTURE_DIR, 'gerrit')
self.change_number = 0
self.changes = changes_db
@@ -936,7 +933,8 @@
'full_name': pr.project
}
},
- 'files': pr.files
+ 'files': pr.files,
+ 'labels': pr.labels
}
return data
@@ -1228,6 +1226,7 @@
self.build_history = []
self.fail_tests = {}
self.job_builds = {}
+ self.hostname = 'zl.example.com'
def failJob(self, name, change):
"""Instruct the executor to report matching builds as failures.
@@ -1356,13 +1355,24 @@
"""
- def __init__(self):
+ def __init__(self, use_ssl=False):
self.hold_jobs_in_queue = False
- super(FakeGearmanServer, self).__init__(0)
+ if use_ssl:
+ ssl_ca = os.path.join(FIXTURE_DIR, 'gearman/root-ca.pem')
+ ssl_cert = os.path.join(FIXTURE_DIR, 'gearman/server.pem')
+ ssl_key = os.path.join(FIXTURE_DIR, 'gearman/server.key')
+ else:
+ ssl_ca = None
+ ssl_cert = None
+ ssl_key = None
+
+ super(FakeGearmanServer, self).__init__(0, ssl_key=ssl_key,
+ ssl_cert=ssl_cert,
+ ssl_ca=ssl_ca)
def getJobForConnection(self, connection, peek=False):
- for queue in [self.high_queue, self.normal_queue, self.low_queue]:
- for job in queue:
+ for job_queue in [self.high_queue, self.normal_queue, self.low_queue]:
+ for job in job_queue:
if not hasattr(job, 'waiting'):
if job.name.startswith(b'executor:execute'):
job.waiting = self.hold_jobs_in_queue
@@ -1372,7 +1382,7 @@
continue
if job.name in connection.functions:
if not peek:
- queue.remove(job)
+ job_queue.remove(job)
connection.related_jobs[job.handle] = job
job.worker_connection = connection
job.running = True
@@ -1812,6 +1822,7 @@
config_file = 'zuul.conf'
run_ansible = False
create_project_keys = False
+ use_ssl = False
def _startMerger(self):
self.merge_server = zuul.merger.server.MergeServer(self.config,
@@ -1866,14 +1877,25 @@
os.environ['STATSD_PORT'] = str(self.statsd.port)
self.statsd.start()
# the statsd client object is configured in the statsd module import
- reload_module(statsd)
- reload_module(zuul.scheduler)
+ importlib.reload(statsd)
+ importlib.reload(zuul.scheduler)
- self.gearman_server = FakeGearmanServer()
+ self.gearman_server = FakeGearmanServer(self.use_ssl)
self.config.set('gearman', 'port', str(self.gearman_server.port))
self.log.info("Gearman server on port %s" %
(self.gearman_server.port,))
+ if self.use_ssl:
+ self.log.info('SSL enabled for gearman')
+ self.config.set(
+ 'gearman', 'ssl_ca',
+ os.path.join(FIXTURE_DIR, 'gearman/root-ca.pem'))
+ self.config.set(
+ 'gearman', 'ssl_cert',
+ os.path.join(FIXTURE_DIR, 'gearman/client.pem'))
+ self.config.set(
+ 'gearman', 'ssl_key',
+ os.path.join(FIXTURE_DIR, 'gearman/client.key'))
gerritsource.GerritSource.replication_timeout = 1.5
gerritsource.GerritSource.replication_retry_interval = 0.5
@@ -1984,7 +2006,7 @@
# This creates the per-test configuration object. It can be
# overriden by subclasses, but should not need to be since it
# obeys the config_file and tenant_config_file attributes.
- self.config = ConfigParser.ConfigParser()
+ self.config = configparser.ConfigParser()
self.config.read(os.path.join(FIXTURE_DIR, self.config_file))
if not self.setupSimpleLayout():
@@ -2000,6 +2022,8 @@
project = reponame.replace('_', '/')
self.copyDirToRepo(project,
os.path.join(git_path, reponame))
+ # Make test_root persist after ansible run for .flag test
+ self.config.set('executor', 'trusted_rw_dirs', self.test_root)
self.setupAllProjectKeys()
def setupSimpleLayout(self):
@@ -2359,12 +2383,12 @@
return True
def eventQueuesEmpty(self):
- for queue in self.event_queues:
- yield queue.empty()
+ for event_queue in self.event_queues:
+ yield event_queue.empty()
def eventQueuesJoin(self):
- for queue in self.event_queues:
- queue.join()
+ for event_queue in self.event_queues:
+ event_queue.join()
def waitUntilSettled(self):
self.log.debug("Waiting until settled...")
@@ -2373,8 +2397,9 @@
if time.time() - start > self.wait_timeout:
self.log.error("Timeout waiting for Zuul to settle")
self.log.error("Queue status:")
- for queue in self.event_queues:
- self.log.error(" %s: %s" % (queue, queue.empty()))
+ for event_queue in self.event_queues:
+ self.log.error(" %s: %s" %
+ (event_queue, event_queue.empty()))
self.log.error("All builds waiting: %s" %
(self.areAllBuildsWaiting(),))
self.log.error("All builds reported: %s" %
@@ -2433,11 +2458,12 @@
# Make sure there are no orphaned jobs
for tenant in self.sched.abide.tenants.values():
for pipeline in tenant.layout.pipelines.values():
- for queue in pipeline.queues:
- if len(queue.queue) != 0:
+ for pipeline_queue in pipeline.queues:
+ if len(pipeline_queue.queue) != 0:
print('pipeline %s queue %s contents %s' % (
- pipeline.name, queue.name, queue.queue))
- self.assertEqual(len(queue.queue), 0,
+ pipeline.name, pipeline_queue.name,
+ pipeline_queue.queue))
+ self.assertEqual(len(pipeline_queue.queue), 0,
"Pipelines queues should be empty")
def assertReportedStat(self, key, value=None, kind=None):
@@ -2683,6 +2709,11 @@
run_ansible = True
+class SSLZuulTestCase(ZuulTestCase):
+ """ZuulTestCase but using SSL when possible"""
+ use_ssl = True
+
+
class ZuulDBTestCase(ZuulTestCase):
def setup_config(self):
super(ZuulDBTestCase, self).setup_config()
diff --git a/tests/fixtures/config/ansible/git/common-config/playbooks/check-vars.yaml b/tests/fixtures/config/ansible/git/common-config/playbooks/check-vars.yaml
index 1f8fdf3..ce392a4 100644
--- a/tests/fixtures/config/ansible/git/common-config/playbooks/check-vars.yaml
+++ b/tests/fixtures/config/ansible/git/common-config/playbooks/check-vars.yaml
@@ -3,9 +3,9 @@
- name: Assert nodepool variables are valid.
assert:
that:
- - nodepool_az == 'test-az'
- - nodepool_region == 'test-region'
- - nodepool_provider == 'test-provider'
+ - nodepool.az == 'test-az'
+ - nodepool.region == 'test-region'
+ - nodepool.provider == 'test-provider'
- name: Assert zuul-executor variables are valid.
assert:
diff --git a/tests/fixtures/config/ansible/git/common-config/zuul.yaml b/tests/fixtures/config/ansible/git/common-config/zuul.yaml
index b31c148..fd3fc6d 100644
--- a/tests/fixtures/config/ansible/git/common-config/zuul.yaml
+++ b/tests/fixtures/config/ansible/git/common-config/zuul.yaml
@@ -51,8 +51,8 @@
- job:
name: python27
- pre-run: pre
- post-run: post
+ pre-run: playbooks/pre
+ post-run: playbooks/post
vars:
flagpath: '{{zuul._test.test_root}}/{{zuul.uuid}}.flag'
roles:
@@ -75,4 +75,4 @@
- job:
name: hello
- post-run: hello-post
+ post-run: playbooks/hello-post
diff --git a/tests/fixtures/gearman/README.rst b/tests/fixtures/gearman/README.rst
new file mode 100644
index 0000000..a3921ea
--- /dev/null
+++ b/tests/fixtures/gearman/README.rst
@@ -0,0 +1,21 @@
+Steps used to create our certs
+
+# Generate CA cert
+$ openssl req -new -newkey rsa:2048 -nodes -keyout root-ca.key -x509 -days 3650 -out root-ca.pem -subj "/C=US/ST=Texas/L=Austin/O=OpenStack Foundation/CN=gearman-ca"
+
+# Generate server keys
+$ CLIENT='server'
+$ openssl req -new -newkey rsa:2048 -nodes -keyout $CLIENT.key -out $CLIENT.csr -subj "/C=US/ST=Texas/L=Austin/O=OpenStack Foundation/CN=nodepool-$CLIENT"
+$ openssl x509 -req -days 3650 -in $CLIENT.csr -out $CLIENT.pem -CA root-ca.pem -CAkey root-ca.key -CAcreateserial
+
+
+# Generate client keys
+$ CLIENT='client'
+$ openssl req -new -newkey rsa:2048 -nodes -keyout $CLIENT.key -out $CLIENT.csr -subj "/C=US/ST=Texas/L=Austin/O=OpenStack Foundation/CN=gearman-$CLIENT"
+$ openssl x509 -req -days 3650 -in $CLIENT.csr -out $CLIENT.pem -CA root-ca.pem -CAkey root-ca.key -CAcreateserial
+
+
+# Test with geard
+# You'll need 2 terminal windows
+geard --ssl-ca root-ca.pem --ssl-cert server.pem --ssl-key server.key -d
+openssl s_client -connect localhost:4730 -key client.key -cert client.pem -CAfile root-ca.pem
diff --git a/tests/fixtures/gearman/client.csr b/tests/fixtures/gearman/client.csr
new file mode 100644
index 0000000..fadb857
--- /dev/null
+++ b/tests/fixtures/gearman/client.csr
@@ -0,0 +1,17 @@
+-----BEGIN CERTIFICATE REQUEST-----
+MIICqzCCAZMCAQAwZjELMAkGA1UEBhMCVVMxDjAMBgNVBAgMBVRleGFzMQ8wDQYD
+VQQHDAZBdXN0aW4xHTAbBgNVBAoMFE9wZW5TdGFjayBGb3VuZGF0aW9uMRcwFQYD
+VQQDDA5nZWFybWFuLWNsaWVudDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
+ggEBALe+ByAkac9cYjeV8lcWXhDxdFqb7Om+6cWSJ/hpM4Z5QyGJ9XHDWyhrmt5W
+X2jvE/bAxEWXxWj3v8xR5HbjS6XHBHouQxz+FSDcG1GZjOLK5fwnO5tKG5eLdrAN
+WgOqJynJAsA0IuxURI4LiBUnzdi/10VeygwSIHOBLVWfrTZNKiE8siiQIaUAerLT
+T8BEUEAUI38UhS4OT83QGUbcCPOkioE5/Q8VVpvlu3eIIEkkacs5293EfUvQRVSG
++GYjSHfFBV7ECX7gu/4nosa/bLfQw7F9O1C2E6QEoUqVNEtURXT0ALlGkUylq6H9
+ctVjoJS9iW8ToMtajW2PZVI/d6MCAwEAAaAAMA0GCSqGSIb3DQEBCwUAA4IBAQBc
+v3/Z9Exc7pnbwyL31ZGv+gF0Z1l9CaSobdI3JAMzKxYGK9SxYOAwcwuUL0+zAJEE
+VPAaWM0p6eVF6j0d97Q79XsHvIKvyVYFxZ9rYSI+cvAIxhws1b4YtRoPBlY3AajV
+u2CQDVos/8JB28X3DpM4MJRua2tnTfAGLCkEp1psAoND+rr5eL7j+naUcPvNMv7Z
+WnTbIJYmP/6N+8gGGtAiiibXP3/Z92kFUZZxKNt3YSHfhkGY57/p+d8i3/8B+qeA
+/YfohA4hNLPydcw32kzo7865+h3SMdbX7VF8xB9grbZXvkT26rtrFJxWLOf5Vmzi
+aGPrVyPIeyVJvW3EeJQ9
+-----END CERTIFICATE REQUEST-----
diff --git a/tests/fixtures/gearman/client.key b/tests/fixtures/gearman/client.key
new file mode 100644
index 0000000..656cfc7
--- /dev/null
+++ b/tests/fixtures/gearman/client.key
@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC3vgcgJGnPXGI3
+lfJXFl4Q8XRam+zpvunFkif4aTOGeUMhifVxw1soa5reVl9o7xP2wMRFl8Vo97/M
+UeR240ulxwR6LkMc/hUg3BtRmYziyuX8JzubShuXi3awDVoDqicpyQLANCLsVESO
+C4gVJ83Yv9dFXsoMEiBzgS1Vn602TSohPLIokCGlAHqy00/ARFBAFCN/FIUuDk/N
+0BlG3AjzpIqBOf0PFVab5bt3iCBJJGnLOdvdxH1L0EVUhvhmI0h3xQVexAl+4Lv+
+J6LGv2y30MOxfTtQthOkBKFKlTRLVEV09AC5RpFMpauh/XLVY6CUvYlvE6DLWo1t
+j2VSP3ejAgMBAAECggEAF5cAFzJVm1fDDFvl9yRaA1bcl115dzEZllIDa7Ml+FfN
+NJsftfFc3L2j7nOsYC6Bo6ZwDHdF0worx7Gj4VehOLFqc71IxIoicEuR/lH2co+W
+I19uGavUCwrOvB+atOm9iXHTNpX6/dh7zLjSSdUIapGGs9NNoWsaW3n0NhAADv51
+SQYemDgG9/vLGPoouUGTBkMNCuI+uHP1C+nRSs/kikebjVqYoDNPm1/ADpccde9p
+mntezm9v/xDXzVFD2qQTTve1mDpoy6YLZdY0mT6qUNElwZ+yZHXkBox1tpJ69Uw+
+pGanSMOy/Gj3W5RlX4qTSLLRcSePV8x65MzRwFoxgQKBgQDstP1/sfnS3JBWSW6V
+YAN3alXeyb0bH0uA+nfXRzy9GnwlFTAszzwnbKL9xK+hPjJRkDBf8XDyXKQ3FMyi
+OVf+H2IkhwErQL01qG4k8E3Hk9wQMvjdO00SaEiLD2uMxX9lRCs9vVlvtmSbGvTH
+/RXBFnqYDHeMJxnWZ8Y34chtoQKBgQDGt+cYtoXH79imuyOQ1SORtIQtSGEKcIcg
+20o5tCGJfCxLtrKs7n4Yph9IPvMtiA8idPACWU2Q8XV580RABzC7Am8ICGGJSwN8
+PLoWOadEpYYeFOV8Mzfxs/YhdQat6zvGy8sF0O+DER0b1ELfbA1I+FNOuz0y53AJ
+MXxOUvQ2wwKBgAFWHEBGTvTDzgTOsVMikaJw9T8mwGyQxqpZv6d1fYBLz/udnQID
+wYEvedQY8izk3v/a4osIH+0eXMb61RTtYfPLVZCDOpx15xuQcd6/hJDl4s4sm38U
+QKEj+ZTfZ2oKC2gU9HGKyiB5VSQTCOLAKQlICTUmjN47skelmlbibXFBAoGBAIHn
+UoELQGU1W3GTQGq7imcDlKxtdlJ2wT8vW1RhdtMDg4lzQ1ZdCb1fS2/VBu8q1In3
+27YNXvFzhxJTfrhEewylSKP9ppUznnGm2RcSVVBAzG35xxLsAJRWyn2QnO8wqYEJ
+VAzXSttpYpgAqD6Zyg17mCoNqLIQLWM1IEerXs41AoGAGdswRmzQ2oHF0f01yZaq
+rxGtLOuTyHzmwi8vA4qZj/9Baht9ihVJiqxTAg/CuA3sTM7DxAJ6P5h6mHsVM6bh
+tPVruBdPIOg4XchcXory1Pa8wSHsPkEnj2NnrZRhvcv86vFxDkhu601nv+AGHj1D
+szjDKeH4IP8fjbf/utRxo3w=
+-----END PRIVATE KEY-----
diff --git a/tests/fixtures/gearman/client.pem b/tests/fixtures/gearman/client.pem
new file mode 100644
index 0000000..aac9d8d
--- /dev/null
+++ b/tests/fixtures/gearman/client.pem
@@ -0,0 +1,20 @@
+-----BEGIN CERTIFICATE-----
+MIIDRDCCAiwCCQDnKP1tRJr+2DANBgkqhkiG9w0BAQsFADBiMQswCQYDVQQGEwJV
+UzEOMAwGA1UECAwFVGV4YXMxDzANBgNVBAcMBkF1c3RpbjEdMBsGA1UECgwUT3Bl
+blN0YWNrIEZvdW5kYXRpb24xEzARBgNVBAMMCmdlYXJtYW4tY2EwHhcNMTcwNjE0
+MTQwNzAwWhcNMjcwNjEyMTQwNzAwWjBmMQswCQYDVQQGEwJVUzEOMAwGA1UECAwF
+VGV4YXMxDzANBgNVBAcMBkF1c3RpbjEdMBsGA1UECgwUT3BlblN0YWNrIEZvdW5k
+YXRpb24xFzAVBgNVBAMMDmdlYXJtYW4tY2xpZW50MIIBIjANBgkqhkiG9w0BAQEF
+AAOCAQ8AMIIBCgKCAQEAt74HICRpz1xiN5XyVxZeEPF0Wpvs6b7pxZIn+GkzhnlD
+IYn1ccNbKGua3lZfaO8T9sDERZfFaPe/zFHkduNLpccEei5DHP4VINwbUZmM4srl
+/Cc7m0obl4t2sA1aA6onKckCwDQi7FREjguIFSfN2L/XRV7KDBIgc4EtVZ+tNk0q
+ITyyKJAhpQB6stNPwERQQBQjfxSFLg5PzdAZRtwI86SKgTn9DxVWm+W7d4ggSSRp
+yznb3cR9S9BFVIb4ZiNId8UFXsQJfuC7/ieixr9st9DDsX07ULYTpAShSpU0S1RF
+dPQAuUaRTKWrof1y1WOglL2JbxOgy1qNbY9lUj93owIDAQABMA0GCSqGSIb3DQEB
+CwUAA4IBAQBSYRP7DDGRBs1wwudH2HzaDRNZrhECUq6n45FY3YHkDU5xxi6CA3wD
+EA+fvvB95BvqNNCS4UxQMW3k7cgJQrUVBKXj9m5HqE/GVZuI15+bR9i7vc5USoen
+nfbVhDAvZcrzPhmj/pfnXKwgeE7PhG55mrJvJgSmxmK2wTcRRIQ6dfoj3OIJJHEY
+kW3oK8I+9r5Tufxbg+CIpZVIuENbRDNGhTPCtzDu3q6DHAEOBKHmwc64W/2c+2QV
+CpfPdutVF2reb6CJuGikM8WMh47mksNIyCW832bUvUCDgW4/tqanPqww4lTdU44b
+W8gkkWcUmOa6MVCXIzCy7tEbkEDJC2NE
+-----END CERTIFICATE-----
diff --git a/tests/fixtures/gearman/root-ca.key b/tests/fixtures/gearman/root-ca.key
new file mode 100644
index 0000000..3db94c3
--- /dev/null
+++ b/tests/fixtures/gearman/root-ca.key
@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDAglCCv7aQxXZg
+8wuLq0QuIQbZbK1Y0aFwMaEpOeVyZR/C42nws3hH0BivJZnr5w57fdT2OXFqkAyl
+Pw+sF8PcDlSi2wF33rnswz8qYszX5WUgvGnOtcJx8YJhqBqNCLb0wnneJqNQpXPs
+CmcsEeBMsCVN9Q1cRMgdjyMBpRfcq7WH5NN+o/n4zClHYZwa3wOyH2ipekl4XTEf
+Kz9aq88L3YE/N4dyUWH0UpS+lBem+D0GAarV2IXWqXeMrWce930mBONMhBrgw0X5
+QFrDa0KQn2QRcg9tqlEE9SlAbub/yHUsq7/7q7l6SWl7JBigj4jGw15w98WzSDkJ
+a0we1jexAgMBAAECggEAX/HS3IdeHyM7D7CyZWbzcSYmusBuWOEJ29fwYZKoZ248
++S3MhBl+bhQp6UkNQMSEtEmPlTQl8Z1foBAg6H1jsU43In+SaMLJ2VWqKp7ZRxTe
+ZQVimpJ+GbnraG6W5Qmd3bj7chvBs5TyhIbeytkR+EamIQdsJDtnnUvUf6JflSvl
+gUZxOvfB7UZQZ2PkMQFleZxlEAvgyk8e4k7AnY2IoTyvw1DIUdP7+7hPInBpWaUn
+jJPZzyWyrMDLB+BB7JcdqmO2N5bHudE4iEJwphmdIcHvOFhm/LHfJdZg6+X8lUCP
+lIfzp6Uk25nF5/dsoZQcrdBznhW4NfJsIviui+SSAQKBgQDrVI4pW/1fU4AYoOki
+jB+RaUaBXkRRV6MYJ/ZUPjAONDtFhDdBDEwunVUpTD8sgVKFnaMzNa/+Jcmy4jsE
+Ggj9ZupH05g9Y8w7dYFcfT6VeiPahyml8k/DWWe0qQk0+scH8vuiWcrqIjWRg4WD
+8CXJqSkgrCHFCjuJOvxM24d1UQKBgQDRaupcR/c9AGpUHmhFGexwOyXtIR2ObaVf
+lEZ9rhrpCRAl5RW0tUmd1vHMNDTdRidfYLFe29h6YQ1afgNcV8JdB51VfurJ+cOF
+jbc6FijDag31snIdBnDuV29mazejRm7PSfJjoBnBDNzh3kMed22DsQDlHQmudknH
+wUqUWnWEYQKBgG3bYSoJmXRgxJG6vFq2Ux5MqO9HlFjssmRac3HMPh7DX1AKcsjY
+9s9j/xdyUqNyE5Xwivki/O+FsGzjk21MwhmZa5DwREeUSQkQx7zncsnQ5N/k7Rpc
+zcOB/xmlN3kWAMfDNJkLleBK6/rsDO4Us286msp30KPtLPHZKWKvsMKhAoGAaiER
+5nR+Qsb8G+dRFnv9zB7dqKAYt36vyZF+a+EZODJkoZ/IcU1SopA0+DUY+W69M2Pw
+X89wlQysVMj58Ql0serS/GoWmQdf5EYermxeejI8IuEtXbJO9ysOhMwfZTqjm5+x
+HHYdty1Kn5khUMZblNrWRkaCCo1d9MLrheWWGuECgYEAy5kdeVE8lLliFL39Xrzl
+OCJ1rEIAhXrqr6E3PrMlUiQ75dAOiLEl3GGG7UkSHL/0dZv50RRee+4hxI63P2z0
+xPeH2nvrFzknmabOWxtOpw+H0qGOYto9VcvseFPNKTV2O5wxdfaYgLEOXt8ipaLD
+OVvm6yN1bP1Gxi6vdVppKwk=
+-----END PRIVATE KEY-----
diff --git a/tests/fixtures/gearman/root-ca.pem b/tests/fixtures/gearman/root-ca.pem
new file mode 100644
index 0000000..defedd6
--- /dev/null
+++ b/tests/fixtures/gearman/root-ca.pem
@@ -0,0 +1,22 @@
+-----BEGIN CERTIFICATE-----
+MIIDlzCCAn+gAwIBAgIJAPmWfgTknq1hMA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
+BAYTAlVTMQ4wDAYDVQQIDAVUZXhhczEPMA0GA1UEBwwGQXVzdGluMR0wGwYDVQQK
+DBRPcGVuU3RhY2sgRm91bmRhdGlvbjETMBEGA1UEAwwKZ2Vhcm1hbi1jYTAeFw0x
+NzA2MTQxNDA1NDNaFw0yNzA2MTIxNDA1NDNaMGIxCzAJBgNVBAYTAlVTMQ4wDAYD
+VQQIDAVUZXhhczEPMA0GA1UEBwwGQXVzdGluMR0wGwYDVQQKDBRPcGVuU3RhY2sg
+Rm91bmRhdGlvbjETMBEGA1UEAwwKZ2Vhcm1hbi1jYTCCASIwDQYJKoZIhvcNAQEB
+BQADggEPADCCAQoCggEBAMCCUIK/tpDFdmDzC4urRC4hBtlsrVjRoXAxoSk55XJl
+H8LjafCzeEfQGK8lmevnDnt91PY5cWqQDKU/D6wXw9wOVKLbAXfeuezDPypizNfl
+ZSC8ac61wnHxgmGoGo0ItvTCed4mo1Clc+wKZywR4EywJU31DVxEyB2PIwGlF9yr
+tYfk036j+fjMKUdhnBrfA7IfaKl6SXhdMR8rP1qrzwvdgT83h3JRYfRSlL6UF6b4
+PQYBqtXYhdapd4ytZx73fSYE40yEGuDDRflAWsNrQpCfZBFyD22qUQT1KUBu5v/I
+dSyrv/uruXpJaXskGKCPiMbDXnD3xbNIOQlrTB7WN7ECAwEAAaNQME4wHQYDVR0O
+BBYEFDIaceZ/LY42aNSV0hisgSEcnjlMMB8GA1UdIwQYMBaAFDIaceZ/LY42aNSV
+0hisgSEcnjlMMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAKN60Jnx
+NPSkDlqrKtcojX3+oVPC5MQctysZXmjjkGzHSAVKeonQ+gN/glfRc0qq/PuzvHej
+a2Mk9CirL2VzBgp1d/sGtOijqI0Otn706SBuQl1PEAzcmTyQt7TuhUnVcV22xBwy
+ONIuXVT5eh8MhUdrlqZKXX9U49sjmHCheJFFVqFmy0twlqf9YikC0CNxiWa/jDhj
+bxi73kxgZTb2RPjwYUWbESfyNCq0H+N2BmSz7Fgc2Ah/wvhXGdx1udaDVgzDqFIR
+lMGswkzmd76JpJdN0Rce7lmRoE8E6BqDShvoEGiGo3IbuOUwn5JRKFMUPhN6mv7N
+c49ykHzcCgc1wdY=
+-----END CERTIFICATE-----
diff --git a/tests/fixtures/gearman/root-ca.srl b/tests/fixtures/gearman/root-ca.srl
new file mode 100644
index 0000000..0ce584a
--- /dev/null
+++ b/tests/fixtures/gearman/root-ca.srl
@@ -0,0 +1 @@
+E728FD6D449AFED8
diff --git a/tests/fixtures/gearman/server.csr b/tests/fixtures/gearman/server.csr
new file mode 100644
index 0000000..bbb03d2
--- /dev/null
+++ b/tests/fixtures/gearman/server.csr
@@ -0,0 +1,17 @@
+-----BEGIN CERTIFICATE REQUEST-----
+MIICrDCCAZQCAQAwZzELMAkGA1UEBhMCVVMxDjAMBgNVBAgMBVRleGFzMQ8wDQYD
+VQQHDAZBdXN0aW4xHTAbBgNVBAoMFE9wZW5TdGFjayBGb3VuZGF0aW9uMRgwFgYD
+VQQDDA9ub2RlcG9vbC1zZXJ2ZXIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
+AoIBAQCzoKkaauTNBRry1Y5YCNG38IrxW0AH5TP5XdTF/q+Qu1p9onRsACiSZX8Y
+YAo/y6jVbZ3WKihVfVIQw9xrPTCoA0AwMtI8fiK70YwSuGg6gqBBCr8NXOaYsYFJ
+k2Vk+8utlNSmLYlcSTKZR0HbhWNmjH9lj5WngL0XPSbcoogtvet92111qGfBZrg+
+86B3XJh2/6PCru9YmufqlooFog7Q4Qo6Bnz7Dh+h2QjtDmGSFz0dQ9PqP8Jgh3LS
+fWRk5TrjGsthKszRTZCQDSXc1XcwAqfO21eufP9oTpfc0zTdAOC1tspdP/632q6B
+0Gf8sSEnMpKmwuGUH3z2ZCY6DSE1AgMBAAGgADANBgkqhkiG9w0BAQsFAAOCAQEA
+NPZ0BNt9vjNM9cNHCgL8rYdB9UnsnkcQ5R/XRV1W+tQlj9QjpvcGH5c3PJ6Ol1Qd
+x8o19aomLb/IMz8bnRmzLxWggKQHxLwU3UKjHBiV1aqI/ieka22IqKYkjeYUAyxC
+ZLytynIZRVt0MB/lo7Z2bjctGHSiZ9tkTsgjawE3hotnZ3BOEOkV42099bLLGdcz
+Jq433DsbwThKC0WijeHR4FZEj3H7Gj07PNAlfyM0KeyrZodtcIwvgA4NyBB8mPoV
+dARn5C8hOtDCWzRPba46h9mTzF8D87pdvmZce6k/bBGJfY+YvOpwBXsO3xhCDxqP
+p9gAs6m+qbxsrwvRRrtn6Q==
+-----END CERTIFICATE REQUEST-----
diff --git a/tests/fixtures/gearman/server.key b/tests/fixtures/gearman/server.key
new file mode 100644
index 0000000..c1707b0
--- /dev/null
+++ b/tests/fixtures/gearman/server.key
@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQCzoKkaauTNBRry
+1Y5YCNG38IrxW0AH5TP5XdTF/q+Qu1p9onRsACiSZX8YYAo/y6jVbZ3WKihVfVIQ
+w9xrPTCoA0AwMtI8fiK70YwSuGg6gqBBCr8NXOaYsYFJk2Vk+8utlNSmLYlcSTKZ
+R0HbhWNmjH9lj5WngL0XPSbcoogtvet92111qGfBZrg+86B3XJh2/6PCru9Ymufq
+looFog7Q4Qo6Bnz7Dh+h2QjtDmGSFz0dQ9PqP8Jgh3LSfWRk5TrjGsthKszRTZCQ
+DSXc1XcwAqfO21eufP9oTpfc0zTdAOC1tspdP/632q6B0Gf8sSEnMpKmwuGUH3z2
+ZCY6DSE1AgMBAAECggEAaG06YhVKtrYFGK92dU+LPHgnDnGSJATn1kzqacDKqEWD
+Mg7DyBW/gHxpCu6qhrQLjyiO3fbcQ/b7Qqva9K06IDLjmiGxf2GFJ9OGr0ttrLZM
+HAP3VflwRczL8M4z4CVSH7OqfIF0naYgOGPosYo2Y2PCnHSA+EQrqdrvQM1shcot
+8lW368VqlAm8ONgh8z4ZLSDswECgJzWleOSsTBIT0qJ6fXIwnN7akM8Bdyy/dPDD
+PnPvAu1N9KgwrzxKY9WthJ1alKiFQm4Po/TZZApALOtR8zCN4EmDG9izKdfU5FIL
+ZWpVDp0US7a8rbj2e0kf0loRg2bsR2eoJPL7JjJycQKBgQDiHjKnwximtfjqLTtg
+ZOHIL4tTlmeTLNq7ZW69BuJSfI7FTa20piHjny+g3mTvxnCQa/BTSpb6VRHPFcYV
+dVQzdAX6ZMvBZ3YMp9FkY+S9RrjEyimNU9kvJJQBnC1ujen3YuXj6ENFzcmGkvzR
+LZFx3dmFEzfDxOOqzdFTHscGuwKBgQDLXaVBH54yq1fDrXDLG/eEtQsNNyCujIV4
+gp1Z54L34htoDS98dx0L0qZGBEys8I0dGJd9kUBVNu53zDeiJSGW4tHYXQaUpxJH
+0wZDHo59mw3aGvVZ5YP+4uukuNHcX6cUYi2HAv0vwet46L3Kb/utDyyStp1QZw9s
+eucOLGkQzwKBgG3j0yZo0FAk28WjGdos7PWG9aU30TpbcCnmj7zZ3Z/M3O3SZHsI
+yit/L3x02IUW4Zmue2tfMqSSN0d3A39mN/eRiV45IjTp/RsFa+PoEEBUYHNy9GK0
+vzYEBtIJfqLd4TjTHXp3ZEpGSoxWXvuhs6+s64ua3V0NEL/vqq1EpeEFAoGAHa/i
+8tnJvz3SBwenoo7HmEDRhzFX/QMYbNosXDZ2oPcJ5yudlf7RZ6ttiGUSSGCpSOkR
+HEx65rWpJCXUrT/cYmlkFsCluEeXXJLKpDuus1lSMVekH2Zo2WmI2rf8Mr5n5ora
+eI4QJcuaM0FOi2HDjKTdbeFon5cb4ksitaf4AnMCgYB24KyMuOHBAuVlnuf3PSfr
+u3ZxqmcUX0D2BoK+1lw3lgzfQO26Qw5VtkjDBnIPL67IUYRZX2YvXsJPWaRRrF72
+yEqFXDWKbcE+Tl0LxLj6mLW5RKJP8LTybaIBgkyUaLtzTRr+TfK29CC8/FzWGiTf
+oJQozL3TAlvjoadEPrLnjg==
+-----END PRIVATE KEY-----
diff --git a/tests/fixtures/gearman/server.pem b/tests/fixtures/gearman/server.pem
new file mode 100644
index 0000000..1c85fad
--- /dev/null
+++ b/tests/fixtures/gearman/server.pem
@@ -0,0 +1,20 @@
+-----BEGIN CERTIFICATE-----
+MIIDRTCCAi0CCQDnKP1tRJr+1zANBgkqhkiG9w0BAQsFADBiMQswCQYDVQQGEwJV
+UzEOMAwGA1UECAwFVGV4YXMxDzANBgNVBAcMBkF1c3RpbjEdMBsGA1UECgwUT3Bl
+blN0YWNrIEZvdW5kYXRpb24xEzARBgNVBAMMCmdlYXJtYW4tY2EwHhcNMTcwNjE0
+MTQwNjM1WhcNMjcwNjEyMTQwNjM1WjBnMQswCQYDVQQGEwJVUzEOMAwGA1UECAwF
+VGV4YXMxDzANBgNVBAcMBkF1c3RpbjEdMBsGA1UECgwUT3BlblN0YWNrIEZvdW5k
+YXRpb24xGDAWBgNVBAMMD25vZGVwb29sLXNlcnZlcjCCASIwDQYJKoZIhvcNAQEB
+BQADggEPADCCAQoCggEBALOgqRpq5M0FGvLVjlgI0bfwivFbQAflM/ld1MX+r5C7
+Wn2idGwAKJJlfxhgCj/LqNVtndYqKFV9UhDD3Gs9MKgDQDAy0jx+IrvRjBK4aDqC
+oEEKvw1c5pixgUmTZWT7y62U1KYtiVxJMplHQduFY2aMf2WPlaeAvRc9JtyiiC29
+633bXXWoZ8FmuD7zoHdcmHb/o8Ku71ia5+qWigWiDtDhCjoGfPsOH6HZCO0OYZIX
+PR1D0+o/wmCHctJ9ZGTlOuMay2EqzNFNkJANJdzVdzACp87bV658/2hOl9zTNN0A
+4LW2yl0//rfaroHQZ/yxIScykqbC4ZQffPZkJjoNITUCAwEAATANBgkqhkiG9w0B
+AQsFAAOCAQEAlqcjSBG96JnKcSlw4ntxJiSGja5iuMi3yVpQS8G3ak6i8eGYlqMH
+SCWC96ZfXr/KjVyF3AsD554e54pEAywcFLH4QzZoceWc5L2etfTCa9cInQsiNpvV
+CfvVADRX4Ib7ozb4MJFJFy5OWnhPO6CcknA2KdTergKIichBmR0LvuUZEblwHOcg
+HAwxpZirNofs/i+aXnIgKAIC97WY1S+8SL5cEfdR0Sd9SpbCLVgSdyGhxm0NE2ls
+38jQhwYIVkpYYJd/jsyGtiHCDT4rkSEJlRWYfLXfSkyjtiERASqs/NEgrnbkgp/l
+Sa2wc5cjntNzls2ey7bkpZbgwOvGQVjS7w==
+-----END CERTIFICATE-----
diff --git a/tests/fixtures/layouts/requirements-github.yaml b/tests/fixtures/layouts/requirements-github.yaml
index 9933f27..891a366 100644
--- a/tests/fixtures/layouts/requirements-github.yaml
+++ b/tests/fixtures/layouts/requirements-github.yaml
@@ -168,6 +168,21 @@
github:
comment: true
+- pipeline:
+ name: require_label
+ manager: independent
+ require:
+ github:
+ label: approved
+ trigger:
+ github:
+ - event: pull_request
+ action: comment
+ comment: 'test me'
+ success:
+ github:
+ comment: true
+
- job:
name: project1-pipeline
- job:
@@ -186,6 +201,8 @@
name: project8-requireopen
- job:
name: project9-requirecurrent
+- job:
+ name: project10-label
- project:
name: org/project1
@@ -243,3 +260,9 @@
require_current:
jobs:
- project9-requirecurrent
+
+- project:
+ name: org/project10
+ require_label:
+ jobs:
+ - project10-label
diff --git a/tests/fixtures/zuul-connections-gerrit-and-github.conf b/tests/fixtures/zuul-connections-gerrit-and-github.conf
index bd05c75..69e7f8b 100644
--- a/tests/fixtures/zuul-connections-gerrit-and-github.conf
+++ b/tests/fixtures/zuul-connections-gerrit-and-github.conf
@@ -3,7 +3,6 @@
[zuul]
tenant_config=config/multi-driver/main.yaml
-job_name_in_report=true
[merger]
git_dir=/tmp/zuul-test/git
diff --git a/tests/fixtures/zuul-connections-merger.conf b/tests/fixtures/zuul-connections-merger.conf
index 7a1bc42..4499493 100644
--- a/tests/fixtures/zuul-connections-merger.conf
+++ b/tests/fixtures/zuul-connections-merger.conf
@@ -2,7 +2,6 @@
server=127.0.0.1
[zuul]
-job_name_in_report=true
status_url=http://zuul.example.com/status
[merger]
diff --git a/tests/fixtures/zuul-connections-multiple-gerrits.conf b/tests/fixtures/zuul-connections-multiple-gerrits.conf
index d1522ec..43b00ef 100644
--- a/tests/fixtures/zuul-connections-multiple-gerrits.conf
+++ b/tests/fixtures/zuul-connections-multiple-gerrits.conf
@@ -3,7 +3,6 @@
[zuul]
tenant_config=main.yaml
-job_name_in_report=true
[merger]
git_dir=/tmp/zuul-test/merger-git
diff --git a/tests/fixtures/zuul-connections-same-gerrit.conf b/tests/fixtures/zuul-connections-same-gerrit.conf
index 8ddd0f1..8a998cf 100644
--- a/tests/fixtures/zuul-connections-same-gerrit.conf
+++ b/tests/fixtures/zuul-connections-same-gerrit.conf
@@ -3,7 +3,6 @@
[zuul]
tenant_config=config/zuul-connections-same-gerrit/main.yaml
-job_name_in_report=true
[merger]
git_dir=/tmp/zuul-test/merger-git
diff --git a/tests/fixtures/zuul-git-driver.conf b/tests/fixtures/zuul-git-driver.conf
index 499b564..b6d3473 100644
--- a/tests/fixtures/zuul-git-driver.conf
+++ b/tests/fixtures/zuul-git-driver.conf
@@ -3,7 +3,6 @@
[zuul]
tenant_config=config/zuul-connections-same-gerrit/main.yaml
-job_name_in_report=true
[merger]
git_dir=/tmp/zuul-test/git
diff --git a/tests/fixtures/zuul-github-driver.conf b/tests/fixtures/zuul-github-driver.conf
index dfa813d..dc28f98 100644
--- a/tests/fixtures/zuul-github-driver.conf
+++ b/tests/fixtures/zuul-github-driver.conf
@@ -2,7 +2,6 @@
server=127.0.0.1
[zuul]
-job_name_in_report=true
status_url=http://zuul.example.com/status/#{change.number},{change.patchset}
[merger]
diff --git a/tests/fixtures/zuul-push-reqs.conf b/tests/fixtures/zuul-push-reqs.conf
index 661ac79..c5272aa 100644
--- a/tests/fixtures/zuul-push-reqs.conf
+++ b/tests/fixtures/zuul-push-reqs.conf
@@ -2,7 +2,6 @@
server=127.0.0.1
[zuul]
-job_name_in_report=true
status_url=http://zuul.example.com/status
[merger]
diff --git a/tests/fixtures/zuul-sql-driver-bad.conf b/tests/fixtures/zuul-sql-driver-bad.conf
index a4df735..1f1b75f 100644
--- a/tests/fixtures/zuul-sql-driver-bad.conf
+++ b/tests/fixtures/zuul-sql-driver-bad.conf
@@ -3,7 +3,6 @@
[zuul]
layout_config=layout-connections-multiple-voters.yaml
-job_name_in_report=true
[merger]
git_dir=/tmp/zuul-test/merger-git
diff --git a/tests/fixtures/zuul-sql-driver.conf b/tests/fixtures/zuul-sql-driver.conf
index 42ab1ba..6fdd081 100644
--- a/tests/fixtures/zuul-sql-driver.conf
+++ b/tests/fixtures/zuul-sql-driver.conf
@@ -4,7 +4,6 @@
[zuul]
tenant_config=main.yaml
url_pattern=http://logs.example.com/{change.number}/{change.patchset}/{pipeline.name}/{job.name}/{build.number}
-job_name_in_report=true
[merger]
git_dir=/tmp/zuul-test/merger-git
diff --git a/tests/fixtures/zuul.conf b/tests/fixtures/zuul.conf
index cd80a45..c4cfe70 100644
--- a/tests/fixtures/zuul.conf
+++ b/tests/fixtures/zuul.conf
@@ -3,7 +3,6 @@
[zuul]
tenant_config=main.yaml
-job_name_in_report=true
[merger]
git_dir=/tmp/zuul-test/merger-git
diff --git a/tests/unit/test_bubblewrap.py b/tests/unit/test_bubblewrap.py
index b274944..675221e 100644
--- a/tests/unit/test_bubblewrap.py
+++ b/tests/unit/test_bubblewrap.py
@@ -15,6 +15,7 @@
import subprocess
import tempfile
import testtools
+import os
from zuul.driver import bubblewrap
from zuul.executor.server import SshAgent
@@ -30,17 +31,14 @@
def test_bubblewrap_wraps(self):
bwrap = bubblewrap.BubblewrapDriver()
work_dir = tempfile.mkdtemp()
- ansible_dir = tempfile.mkdtemp()
ssh_agent = SshAgent()
self.addCleanup(ssh_agent.stop)
ssh_agent.start()
po = bwrap.getPopen(work_dir=work_dir,
- ansible_dir=ansible_dir,
ssh_auth_sock=ssh_agent.env['SSH_AUTH_SOCK'])
self.assertTrue(po.passwd_r > 2)
self.assertTrue(po.group_r > 2)
self.assertTrue(work_dir in po.command)
- self.assertTrue(ansible_dir in po.command)
# Now run /usr/bin/id to verify passwd/group entries made it in
true_proc = po(['/usr/bin/id'], stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
@@ -52,3 +50,23 @@
# Make sure the _r's are closed
self.assertIsNone(po.passwd_r)
self.assertIsNone(po.group_r)
+
+ def test_bubblewrap_leak(self):
+ bwrap = bubblewrap.BubblewrapDriver()
+ work_dir = tempfile.mkdtemp()
+ ansible_dir = tempfile.mkdtemp()
+ ssh_agent = SshAgent()
+ self.addCleanup(ssh_agent.stop)
+ ssh_agent.start()
+ po = bwrap.getPopen(work_dir=work_dir,
+ ansible_dir=ansible_dir,
+ ssh_auth_sock=ssh_agent.env['SSH_AUTH_SOCK'])
+ leak_time = 7
+ # Use hexadecimal notation to avoid false-positive
+ true_proc = po(['bash', '-c', 'sleep 0x%X & disown' % leak_time])
+ self.assertEqual(0, true_proc.wait())
+ cmdline = "sleep\x000x%X\x00" % leak_time
+ sleep_proc = [pid for pid in os.listdir("/proc") if
+ os.path.isfile("/proc/%s/cmdline" % pid) and
+ open("/proc/%s/cmdline" % pid).read() == cmdline]
+ self.assertEqual(len(sleep_proc), 0, "Processes leaked")
diff --git a/tests/unit/test_connection.py b/tests/unit/test_connection.py
index 142a248..fcfaf5d 100644
--- a/tests/unit/test_connection.py
+++ b/tests/unit/test_connection.py
@@ -120,9 +120,10 @@
# Check the first result, which should be the project-merge job
self.assertEqual('project-merge', buildset0_builds[0]['job_name'])
self.assertEqual("SUCCESS", buildset0_builds[0]['result'])
- self.assertEqual('https://server/job/project-merge/0/',
- buildset0_builds[0]['log_url'])
-
+ self.assertEqual(
+ 'finger://zl.example.com/{uuid}'.format(
+ uuid=buildset0_builds[0]['uuid']),
+ buildset0_builds[0]['log_url'])
self.assertEqual('check', buildset1['pipeline'])
self.assertEqual('org/project', buildset1['project'])
self.assertEqual(2, buildset1['change'])
@@ -142,8 +143,10 @@
# which failed
self.assertEqual('project-test1', buildset1_builds[-2]['job_name'])
self.assertEqual("FAILURE", buildset1_builds[-2]['result'])
- self.assertEqual('https://server/job/project-test1/0/',
- buildset1_builds[-2]['log_url'])
+ self.assertEqual(
+ 'finger://zl.example.com/{uuid}'.format(
+ uuid=buildset1_builds[-2]['uuid']),
+ buildset1_builds[-2]['log_url'])
def test_multiple_sql_connections(self):
"Test putting results in different databases"
diff --git a/tests/unit/test_github_requirements.py b/tests/unit/test_github_requirements.py
index 301ea2f..135f7ab 100644
--- a/tests/unit/test_github_requirements.py
+++ b/tests/unit/test_github_requirements.py
@@ -142,7 +142,7 @@
A = self.fake_github.openFakePullRequest('org/project4', 'master', 'A')
# Add derp to writers
- A.writers.append('derp')
+ A.writers.extend(('derp', 'werp'))
# A comment event that we will keep submitting to trigger
comment = A.getCommentAddedEvent('test me')
self.fake_github.emitEvent(comment)
@@ -156,16 +156,29 @@
self.waitUntilSettled()
self.assertEqual(len(self.history), 0)
+ # A negative review from werp should not cause it to be enqueued
+ A.addReview('werp', 'CHANGES_REQUESTED')
+ self.fake_github.emitEvent(comment)
+ self.waitUntilSettled()
+ self.assertEqual(len(self.history), 0)
+
# A positive from nobody should not cause it to be enqueued
A.addReview('nobody', 'APPROVED')
self.fake_github.emitEvent(comment)
self.waitUntilSettled()
self.assertEqual(len(self.history), 0)
- # A positive review from derp should cause it to be enqueued
+ # A positive review from derp should still be blocked by the
+ # negative review from werp
A.addReview('derp', 'APPROVED')
self.fake_github.emitEvent(comment)
self.waitUntilSettled()
+ self.assertEqual(len(self.history), 0)
+
+ # A positive review from werp should cause it to be enqueued
+ A.addReview('werp', 'APPROVED')
+ self.fake_github.emitEvent(comment)
+ self.waitUntilSettled()
self.assertEqual(len(self.history), 1)
self.assertEqual(self.history[0].name, 'project4-reviewreq')
@@ -337,3 +350,28 @@
self.waitUntilSettled()
# Event hash is not current, should not trigger
self.assertEqual(len(self.history), 1)
+
+ @simple_layout('layouts/requirements-github.yaml', driver='github')
+ def test_pipeline_require_label(self):
+ "Test pipeline requirement: label"
+ A = self.fake_github.openFakePullRequest('org/project10', 'master',
+ 'A')
+ # A comment event that we will keep submitting to trigger
+ comment = A.getCommentAddedEvent('test me')
+ self.fake_github.emitEvent(comment)
+ self.waitUntilSettled()
+ # No label so should not be enqueued
+ self.assertEqual(len(self.history), 0)
+
+ # A derp label should not cause it to be enqueued
+ A.addLabel('derp')
+ self.fake_github.emitEvent(comment)
+ self.waitUntilSettled()
+ self.assertEqual(len(self.history), 0)
+
+ # An approved label goes in
+ A.addLabel('approved')
+ self.fake_github.emitEvent(comment)
+ self.waitUntilSettled()
+ self.assertEqual(len(self.history), 1)
+ self.assertEqual(self.history[0].name, 'project10-label')
diff --git a/tests/unit/test_model.py b/tests/unit/test_model.py
index 7a4d53e..f4ca96f 100644
--- a/tests/unit/test_model.py
+++ b/tests/unit/test_model.py
@@ -266,11 +266,11 @@
self.assertEqual(len(nodes), 1)
self.assertEqual(nodes[0].label, 'new')
self.assertEqual([x.path for x in job.pre_run],
- ['playbooks/base-pre',
- 'playbooks/py27-pre'])
+ ['base-pre',
+ 'py27-pre'])
self.assertEqual([x.path for x in job.post_run],
- ['playbooks/py27-post',
- 'playbooks/base-post'])
+ ['py27-post',
+ 'base-post'])
self.assertEqual([x.path for x in job.run],
['playbooks/python27',
'playbooks/base'])
@@ -294,15 +294,15 @@
self.assertEqual(len(nodes), 1)
self.assertEqual(nodes[0].label, 'old')
self.assertEqual([x.path for x in job.pre_run],
- ['playbooks/base-pre',
- 'playbooks/py27-pre',
- 'playbooks/py27-diablo-pre'])
+ ['base-pre',
+ 'py27-pre',
+ 'py27-diablo-pre'])
self.assertEqual([x.path for x in job.post_run],
- ['playbooks/py27-diablo-post',
- 'playbooks/py27-post',
- 'playbooks/base-post'])
+ ['py27-diablo-post',
+ 'py27-post',
+ 'base-post'])
self.assertEqual([x.path for x in job.run],
- ['playbooks/py27-diablo']),
+ ['py27-diablo']),
# Test essex
change.branch = 'stable/essex'
@@ -319,13 +319,13 @@
job = item.getJobs()[0]
self.assertEqual(job.name, 'python27')
self.assertEqual([x.path for x in job.pre_run],
- ['playbooks/base-pre',
- 'playbooks/py27-pre',
- 'playbooks/py27-essex-pre'])
+ ['base-pre',
+ 'py27-pre',
+ 'py27-essex-pre'])
self.assertEqual([x.path for x in job.post_run],
- ['playbooks/py27-essex-post',
- 'playbooks/py27-post',
- 'playbooks/base-post'])
+ ['py27-essex-post',
+ 'py27-post',
+ 'base-post'])
self.assertEqual([x.path for x in job.run],
['playbooks/python27',
'playbooks/base'])
diff --git a/tests/unit/test_scheduler.py b/tests/unit/test_scheduler.py
index 839007d..eb17966 100755
--- a/tests/unit/test_scheduler.py
+++ b/tests/unit/test_scheduler.py
@@ -25,8 +25,8 @@
from unittest import skip
import git
-from six.moves import urllib
import testtools
+import urllib
import zuul.change_matcher
from zuul.driver.gerrit import gerritreporter
@@ -35,12 +35,36 @@
import zuul.model
from tests.base import (
+ SSLZuulTestCase,
ZuulTestCase,
repack_repo,
simple_layout,
)
+class TestSchedulerSSL(SSLZuulTestCase):
+ tenant_config_file = 'config/single-tenant/main.yaml'
+
+ def test_jobs_executed(self):
+ "Test that jobs are executed and a change is merged"
+
+ A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A')
+ A.addApproval('code-review', 2)
+ self.fake_gerrit.addEvent(A.addApproval('approved', 1))
+ self.waitUntilSettled()
+ self.assertEqual(self.getJobFromHistory('project-merge').result,
+ 'SUCCESS')
+ self.assertEqual(self.getJobFromHistory('project-test1').result,
+ 'SUCCESS')
+ self.assertEqual(self.getJobFromHistory('project-test2').result,
+ 'SUCCESS')
+ self.assertEqual(A.data['status'], 'MERGED')
+ self.assertEqual(A.reported, 2)
+ self.assertEqual(self.getJobFromHistory('project-test1').node,
+ 'label1')
+ self.assertIsNone(self.getJobFromHistory('project-test2').node)
+
+
class TestScheduler(ZuulTestCase):
tenant_config_file = 'config/single-tenant/main.yaml'
@@ -2264,20 +2288,22 @@
for job in change['jobs']:
status_jobs.append(job)
self.assertEqual('project-merge', status_jobs[0]['name'])
- self.assertEqual('https://server/job/project-merge/0/',
+ # TODO(mordred) pull uuids from self.builds
+ self.assertEqual('finger://zl.example.com/%s' % status_jobs[0]['uuid'],
status_jobs[0]['url'])
- self.assertEqual('https://server/job/project-merge/0/',
+ # TOOD(mordred) configure a success-url on the base job
+ self.assertEqual('finger://zl.example.com/%s' % status_jobs[0]['uuid'],
status_jobs[0]['report_url'])
self.assertEqual('project-test1', status_jobs[1]['name'])
- self.assertEqual('https://server/job/project-test1/0/',
+ self.assertEqual('finger://zl.example.com/%s' % status_jobs[1]['uuid'],
status_jobs[1]['url'])
- self.assertEqual('https://server/job/project-test1/0/',
+ self.assertEqual('finger://zl.example.com/%s' % status_jobs[1]['uuid'],
status_jobs[1]['report_url'])
self.assertEqual('project-test2', status_jobs[2]['name'])
- self.assertEqual('https://server/job/project-test2/0/',
+ self.assertEqual('finger://zl.example.com/%s' % status_jobs[2]['uuid'],
status_jobs[2]['url'])
- self.assertEqual('https://server/job/project-test2/0/',
+ self.assertEqual('finger://zl.example.com/%s' % status_jobs[2]['uuid'],
status_jobs[2]['report_url'])
def test_live_reconfiguration(self):
@@ -3551,7 +3577,7 @@
self.assertEqual('project-merge', job['name'])
self.assertEqual('gate', job['pipeline'])
self.assertEqual(False, job['retry'])
- self.assertEqual('https://server/job/project-merge/0/',
+ self.assertEqual('finger://zl.example.com/%s' % job['uuid'],
job['url'])
self.assertEqual(2, len(job['worker']))
self.assertEqual(False, job['canceled'])
@@ -4147,6 +4173,7 @@
self.fake_gerrit.addEvent(A.getPatchsetCreatedEvent(1))
self.waitUntilSettled()
+ self.assertEqual(A.reported, 1)
# Create B->A
B = self.fake_gerrit.addFakeChange('org/project1', 'master', 'B')
@@ -4155,41 +4182,33 @@
self.fake_gerrit.addEvent(B.getPatchsetCreatedEvent(1))
self.waitUntilSettled()
+ # Dep is there so zuul should have reported on B
+ self.assertEqual(B.reported, 1)
+
# Update A to add A->B (a cycle).
A.addPatchset()
A.data['commitMessage'] = '%s\n\nDepends-On: %s\n' % (
A.subject, B.data['id'])
- # Normally we would submit the patchset-created event for
- # processing here, however, we have no way of noting whether
- # the dependency cycle detection correctly raised an
- # exception, so instead, we reach into the source driver and
- # call the method that would ultimately be called by the event
- # processing.
+ self.fake_gerrit.addEvent(A.getPatchsetCreatedEvent(2))
+ self.waitUntilSettled()
- tenant = self.sched.abide.tenants.get('tenant-one')
- (trusted, project) = tenant.getProject('org/project')
- source = project.source
-
- # TODO(pabelanger): As we add more source / trigger APIs we should make
- # it easier for users to create events for testing.
- event = zuul.model.TriggerEvent()
- event.trigger_name = 'gerrit'
- event.change_number = '1'
- event.patch_number = '2'
- with testtools.ExpectedException(
- Exception, "Dependency cycle detected"):
- source.getChange(event, True)
- self.log.debug("Got expected dependency cycle exception")
+ # Dependency cycle injected so zuul should not have reported again on A
+ self.assertEqual(A.reported, 1)
# Now if we update B to remove the depends-on, everything
# should be okay. B; A->B
B.addPatchset()
B.data['commitMessage'] = '%s\n' % (B.subject,)
+ self.fake_gerrit.addEvent(A.getPatchsetCreatedEvent(2))
+ self.waitUntilSettled()
- source.getChange(event, True)
- event.change_number = '2'
- source.getChange(event, True)
+ # Cycle was removed so now zuul should have reported again on A
+ self.assertEqual(A.reported, 2)
+
+ self.fake_gerrit.addEvent(B.getPatchsetCreatedEvent(2))
+ self.waitUntilSettled()
+ self.assertEqual(B.reported, 2)
@simple_layout('layouts/disable_at.yaml')
def test_disable_at(self):
@@ -4279,15 +4298,15 @@
self.assertEqual(0, len(G.messages))
self.assertIn('Build failed.', self.smtp_messages[0]['body'])
self.assertIn(
- 'project-test1 https://server/job', self.smtp_messages[0]['body'])
+ 'project-test1 finger://', self.smtp_messages[0]['body'])
self.assertEqual(0, len(H.messages))
self.assertIn('Build failed.', self.smtp_messages[1]['body'])
self.assertIn(
- 'project-test1 https://server/job', self.smtp_messages[1]['body'])
+ 'project-test1 finger://', self.smtp_messages[1]['body'])
self.assertEqual(0, len(I.messages))
self.assertIn('Build succeeded.', self.smtp_messages[2]['body'])
self.assertIn(
- 'project-test1 https://server/job', self.smtp_messages[2]['body'])
+ 'project-test1 finger://', self.smtp_messages[2]['body'])
# Now reload the configuration (simulate a HUP) to check the pipeline
# comes out of disabled
@@ -4640,7 +4659,8 @@
for build in self.history:
if build.name == 'docs-draft-test':
uuid = build.uuid[:7]
- break
+ elif build.name == 'docs-draft-test2':
+ uuid_test2 = build.uuid
# Two msgs: 'Starting...' + results
self.assertEqual(len(self.smtp_messages), 2)
@@ -4654,7 +4674,8 @@
# NOTE: This default URL is currently hard-coded in executor/server.py
self.assertIn(
- '- docs-draft-test2 https://server/job',
+ '- docs-draft-test2 finger://zl.example.com/{uuid}'.format(
+ uuid=uuid_test2),
body[3])
diff --git a/tests/unit/test_webapp.py b/tests/unit/test_webapp.py
index b2836ae..da027c1 100644
--- a/tests/unit/test_webapp.py
+++ b/tests/unit/test_webapp.py
@@ -17,8 +17,8 @@
import os
import json
+import urllib
-from six.moves import urllib
import webob
from tests.base import ZuulTestCase, FIXTURE_DIR
diff --git a/tools/encrypt_secret.py b/tools/encrypt_secret.py
index 4865edd..e36b24e 100644
--- a/tools/encrypt_secret.py
+++ b/tools/encrypt_secret.py
@@ -17,7 +17,7 @@
import subprocess
import sys
import tempfile
-from six.moves import urllib
+import urllib
DESCRIPTION = """Encrypt a secret for Zuul.
diff --git a/tox.ini b/tox.ini
index 9b97eca..a3f018f 100644
--- a/tox.ini
+++ b/tox.ini
@@ -1,9 +1,10 @@
[tox]
minversion = 1.6
skipsdist = True
-envlist = pep8, py27
+envlist = pep8,py35
[testenv]
+basepython = python3
# Set STATSD env variables so that statsd code paths are tested.
setenv = STATSD_HOST=127.0.0.1
STATSD_PORT=8125
@@ -27,7 +28,6 @@
[testenv:pep8]
# streamer is python3 only, so we need to run flake8 in python3
-basepython = python3
commands = flake8 {posargs}
[testenv:cover]
diff --git a/zuul/ansible/callback/zuul_stream.py b/zuul/ansible/callback/zuul_stream.py
index 260f4ab..e3d1e14 100644
--- a/zuul/ansible/callback/zuul_stream.py
+++ b/zuul/ansible/callback/zuul_stream.py
@@ -14,10 +14,10 @@
# along with Ansible. If not, see <http://www.gnu.org/licenses/>.
import datetime
-import multiprocessing
import logging
import os
import socket
+import threading
import time
import uuid
@@ -62,11 +62,15 @@
of cmd it'll echo what the command was for folks.
"""
+ stdout = result.pop('stdout', '')
+ stdout_lines = result.pop('stdout_lines', [])
+ if not stdout_lines and stdout:
+ stdout_lines = stdout.split('\n')
+
for key in ('changed', 'cmd', 'zuul_log_id',
- 'stderr', 'stderr_lines',
- 'stdout', 'stdout_lines'):
+ 'stderr', 'stderr_lines'):
result.pop(key, None)
- return result
+ return stdout_lines
class CallbackModule(default.CallbackModule):
@@ -85,9 +89,8 @@
super(CallbackModule, self).__init__()
self._task = None
self._daemon_running = False
- self._host_dict = {}
self._play = None
- self._streamer = None
+ self._streamers = []
self.configure_logger()
def configure_logger(self):
@@ -100,17 +103,28 @@
else:
level = logging.INFO
logging.basicConfig(filename=path, level=level, format='%(message)s')
- self._log = logging.getLogger('zuul.executor.ansible')
+ self._logger = logging.getLogger('zuul.executor.ansible')
- def _read_log(self, host, ip, log_id, task_name):
- self._log.debug("[%s] Starting to log %s for task %s"
- % (host, log_id, task_name))
+ def _log(self, msg, ts=None, job=True, executor=False, debug=False):
+ if job:
+ now = ts or datetime.datetime.now()
+ self._logger.info("{now} | {msg}".format(now=now, msg=msg))
+ if executor:
+ if debug:
+ self._display.vvv(msg)
+ else:
+ self._display.display(msg)
+
+ def _read_log(self, host, ip, log_id, task_name, hosts):
+ self._log("[%s] Starting to log %s for task %s"
+ % (host, log_id, task_name), executor=True)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
while True:
try:
s.connect((ip, LOG_STREAM_PORT))
except Exception:
- self._log.debug("[%s] Waiting on logger" % host)
+ self._log("[%s] Waiting on logger" % host,
+ executor=True, debug=True)
time.sleep(0.1)
continue
msg = "%s\n" % log_id
@@ -119,9 +133,10 @@
if "[Zuul] Task exit code" in line:
return
else:
- ts, ln = line.strip().split(' | ', 1)
+ ts, ln = line.split(' | ', 1)
+ ln = ln.strip()
- self._log.info("%s | %s | %s " % (ts, host, ln))
+ self._log("%s | %s " % (host, ln), ts=ts)
def v2_playbook_on_start(self, playbook):
self._playbook_name = os.path.splitext(playbook._file_name)[0]
@@ -129,14 +144,13 @@
def v2_playbook_on_play_start(self, play):
self._play = play
name = play.get_name().strip()
- now = datetime.datetime.now()
if not name:
- msg = u"{now} | PLAY".format(now=now)
+ msg = u"PLAY"
else:
- msg = u"{now} | PLAY [{playbook} : {name}]".format(
- playbook=self._playbook_name, now=now, name=name)
+ msg = u"PLAY [{playbook} : {name}]".format(
+ playbook=self._playbook_name, name=name)
- self._log.info(msg)
+ self._log(msg)
def v2_playbook_on_task_start(self, task, is_conditional):
self._task = task
@@ -148,29 +162,50 @@
task.args['zuul_log_id'] = log_id
play_vars = self._play._variable_manager._hostvars
- hosts = self._play.hosts
- if 'all' in hosts:
- # NOTE(jamielennox): play.hosts is purely the list of hosts
- # that was provided not interpretted by inventory. We don't
- # have inventory access here but we can assume that 'all' is
- # everything in hostvars.
- hosts = play_vars.keys()
-
+ hosts = self._get_task_hosts(task)
for host in hosts:
+ if host in ('localhost', '127.0.0.1'):
+ # Don't try to stream from localhost
+ continue
ip = play_vars[host].get(
'ansible_host', play_vars[host].get(
'ansible_inventory_host'))
- self._host_dict[host] = ip
- self._streamer = multiprocessing.Process(
- target=self._read_log, args=(host, ip, log_id, task_name))
- self._streamer.daemon = True
- self._streamer.start()
+ streamer = threading.Thread(
+ target=self._read_log, args=(
+ host, ip, log_id, task_name, hosts))
+ streamer.daemon = True
+ streamer.start()
+ self._streamers.append(streamer)
+
+ def _stop_streamers(self):
+ while True:
+ if not self._streamers:
+ break
+ streamer = self._streamers.pop()
+ streamer.join(30)
+ if streamer.is_alive():
+ msg = "[Zuul] Log Stream did not terminate"
+ self._log(msg, job=True, executor=True)
+
+ def _process_result_for_localhost(self, result):
+ is_localhost = False
+ delegated_vars = result._result.get('_ansible_delegated_vars', None)
+ if delegated_vars:
+ delegated_host = delegated_vars['ansible_host']
+ if delegated_host in ('localhost', '127.0.0.1'):
+ is_localhost = True
+
+ if not is_localhost:
+ self._stop_streamers()
+ if result._task.action in ('command', 'shell'):
+ stdout_lines = zuul_filter_result(result._result)
+ if is_localhost:
+ for line in stdout_lines:
+ ts, ln = (x.strip() for x in line.split(' | ', 1))
+ self._log("localhost | %s " % ln, ts=ts)
def v2_runner_on_failed(self, result, ignore_errors=False):
- if self._streamer:
- self._streamer.join()
- if result._task.action in ('command', 'shell'):
- zuul_filter_result(result._result)
+ self._process_result_for_localhost(result)
self._handle_exception(result._result)
if result._task.loop and 'results' in result._result:
@@ -190,21 +225,16 @@
self._print_task_banner(result._task)
self._clean_results(result._result, result._task.action)
+ self._process_result_for_localhost(result)
if result._task.action in ('include', 'include_role'):
return
- if self._streamer:
- self._streamer.join()
-
if result._result.get('changed', False):
status = 'changed'
else:
status = 'ok'
- if result._task.action in ('command', 'shell'):
- zuul_filter_result(result._result)
-
if result._task.loop and 'results' in result._result:
self._process_items(result)
@@ -243,18 +273,32 @@
args = u', '.join(u'%s=%s' % a for a in task_args.items())
args = u' %s' % args
- msg = "{now} | TASK [{task}{args}]".format(
- now=datetime.datetime.now(),
+ msg = "TASK [{task}{args}]".format(
task=task_name,
args=args)
- self._log.info(msg)
+ self._log(msg)
return task
+ def _get_task_hosts(self, task):
+ # If this task has as delegate to, we don't care about the play hosts,
+ # we care about the task's delegate target.
+ delegate_to = task.delegate_to
+ if delegate_to:
+ return [delegate_to]
+ hosts = self._play.hosts
+ if 'all' in hosts:
+ # NOTE(jamielennox): play.hosts is purely the list of hosts
+ # that was provided not interpretted by inventory. We don't
+ # have inventory access here but we can assume that 'all' is
+ # everything in hostvars.
+ play_vars = self._play._variable_manager._hostvars
+ hosts = play_vars.keys()
+ return hosts
+
def _log_message(self, result, msg, status="ok"):
- now = datetime.datetime.now()
hostname = self._get_hostname(result)
- self._log.info("{now} | {host} | {status}: {msg}".format(
- host=hostname, now=now, status=status, msg=msg))
+ self._log("{host} | {status}: {msg}".format(
+ host=hostname, status=status, msg=msg))
def _get_hostname(self, result):
delegated_vars = result._result.get('_ansible_delegated_vars', None)
diff --git a/zuul/ansible/library/command.py b/zuul/ansible/library/command.py
index 4b3a30f..99392cc 100644
--- a/zuul/ansible/library/command.py
+++ b/zuul/ansible/library/command.py
@@ -19,6 +19,10 @@
# You should have received a copy of the GNU General Public License
# along with this software. If not, see <http://www.gnu.org/licenses/>.
+ANSIBLE_METADATA = {'metadata_version': '1.0',
+ 'status': ['stableinterface'],
+ 'supported_by': 'core'}
+
# flake8: noqa
# This file shares a significant chunk of code with an upstream ansible
# function, run_command. The goal is to not have to fork quite so much
@@ -34,7 +38,7 @@
short_description: Executes a command on a remote node
version_added: historical
description:
- - The M(command) module takes the command name followed by a list of space-delimited arguments.
+ - The C(command) module takes the command name followed by a list of space-delimited arguments.
- The given command will be executed on all selected nodes. It will not be
processed through the shell, so variables like C($HOME) and operations
like C("<"), C(">"), C("|"), C(";") and C("&") will not work (use the M(shell)
@@ -76,30 +80,33 @@
- if command warnings are on in ansible.cfg, do not warn about this particular line if set to no/false.
required: false
notes:
- - If you want to run a command through the shell (say you are using C(<),
- C(>), C(|), etc), you actually want the M(shell) module instead. The
- M(command) module is much more secure as it's not affected by the user's
- environment.
- - " C(creates), C(removes), and C(chdir) can be specified after the command. For instance, if you only want to run a command if a certain file does not exist, use this."
+ - If you want to run a command through the shell (say you are using C(<), C(>), C(|), etc), you actually want the M(shell) module instead.
+ The C(command) module is much more secure as it's not affected by the user's environment.
+ - " C(creates), C(removes), and C(chdir) can be specified after the command.
+ For instance, if you only want to run a command if a certain file does not exist, use this."
author:
- Ansible Core Team
- Michael DeHaan
'''
EXAMPLES = '''
-# Example from Ansible Playbooks.
-- command: /sbin/shutdown -t now
+- name: return motd to registered var
+ command: cat /etc/motd
+ register: mymotd
-# Run the command if the specified file does not exist.
-- command: /usr/bin/make_database.sh arg1 arg2 creates=/path/to/database
+- name: Run the command if the specified file does not exist.
+ command: /usr/bin/make_database.sh arg1 arg2 creates=/path/to/database
-# You can also use the 'args' form to provide the options. This command
-# will change the working directory to somedir/ and will only run when
-# /path/to/database doesn't exist.
-- command: /usr/bin/make_database.sh arg1 arg2
+# You can also use the 'args' form to provide the options.
+- name: This command will change the working directory to somedir/ and will only run when /path/to/database doesn't exist.
+ command: /usr/bin/make_database.sh arg1 arg2
args:
chdir: somedir/
creates: /path/to/database
+
+- name: safely use templated variable to run command. Always use the quote filter to avoid injection issues.
+ command: cat {{ myfile|quote }}
+ register: myoutput
'''
import datetime
@@ -116,10 +123,19 @@
import threading
from ansible.module_utils.basic import AnsibleModule, heuristic_log_sanitize
-from ansible.module_utils.basic import get_exception
-# ZUUL: Hardcode python2 until we're on ansible 2.2
-from ast import literal_eval
-
+from ansible.module_utils.pycompat24 import get_exception, literal_eval
+from ansible.module_utils.six import (
+ PY2,
+ PY3,
+ b,
+ binary_type,
+ integer_types,
+ iteritems,
+ string_types,
+ text_type,
+)
+from ansible.module_utils.six.moves import map, reduce
+from ansible.module_utils._text import to_native, to_bytes, to_text
LOG_STREAM_FILE = '/tmp/console-{log_uuid}.log'
PASSWD_ARG_RE = re.compile(r'^[-]{0,2}pass[-]?(word|wd)?')
@@ -166,7 +182,7 @@
# Taken from ansible/module_utils/basic.py ... forking the method for now
# so that we can dive in and figure out how to make appropriate hook points
-def zuul_run_command(self, args, zuul_log_id, check_rc=False, close_fds=True, executable=None, data=None, binary_data=False, path_prefix=None, cwd=None, use_unsafe_shell=False, prompt_regex=None, environ_update=None):
+def zuul_run_command(self, args, zuul_log_id, check_rc=False, close_fds=True, executable=None, data=None, binary_data=False, path_prefix=None, cwd=None, use_unsafe_shell=False, prompt_regex=None, environ_update=None, umask=None, encoding='utf-8', errors='surrogate_or_strict'):
'''
Execute a command, returns rc, stdout, and stderr.
@@ -188,7 +204,27 @@
:kw prompt_regex: Regex string (not a compiled regex) which can be
used to detect prompts in the stdout which would otherwise cause
the execution to hang (especially if no input data is specified)
- :kwarg environ_update: dictionary to *update* os.environ with
+ :kw environ_update: dictionary to *update* os.environ with
+ :kw umask: Umask to be used when running the command. Default None
+ :kw encoding: Since we return native strings, on python3 we need to
+ know the encoding to use to transform from bytes to text. If you
+ want to always get bytes back, use encoding=None. The default is
+ "utf-8". This does not affect transformation of strings given as
+ args.
+ :kw errors: Since we return native strings, on python3 we need to
+ transform stdout and stderr from bytes to text. If the bytes are
+ undecodable in the ``encoding`` specified, then use this error
+ handler to deal with them. The default is ``surrogate_or_strict``
+ which means that the bytes will be decoded using the
+ surrogateescape error handler if available (available on all
+ python3 versions we support) otherwise a UnicodeError traceback
+ will be raised. This does not affect transformations of strings
+ given as args.
+ :returns: A 3-tuple of return code (integer), stdout (native string),
+ and stderr (native string). On python2, stdout and stderr are both
+ byte strings. On python3, stdout and stderr are text strings converted
+ according to the encoding and errors parameters. If you want byte
+ strings on python3, use encoding=None to turn decoding to text off.
'''
shell = False
@@ -196,13 +232,15 @@
if use_unsafe_shell:
args = " ".join([pipes.quote(x) for x in args])
shell = True
- elif isinstance(args, (str, unicode)) and use_unsafe_shell:
+ elif isinstance(args, (binary_type, text_type)) and use_unsafe_shell:
shell = True
- elif isinstance(args, (str, unicode)):
+ elif isinstance(args, (binary_type, text_type)):
# On python2.6 and below, shlex has problems with text type
- # ZUUL: Hardcode python2 until we're on ansible 2.2
- if isinstance(args, unicode):
- args = args.encode('utf-8')
+ # On python3, shlex needs a text type.
+ if PY2:
+ args = to_bytes(args, errors='surrogate_or_strict')
+ elif PY3:
+ args = to_text(args, errors='surrogateescape')
args = shlex.split(args)
else:
msg = "Argument 'args' to run_command must be list or string"
@@ -210,6 +248,11 @@
prompt_re = None
if prompt_regex:
+ if isinstance(prompt_regex, text_type):
+ if PY3:
+ prompt_regex = to_bytes(prompt_regex, errors='surrogateescape')
+ elif PY2:
+ prompt_regex = to_bytes(prompt_regex, errors='surrogate_or_strict')
try:
prompt_re = re.compile(prompt_regex, re.MULTILINE)
except re.error:
@@ -217,7 +260,7 @@
# expand things like $HOME and ~
if not shell:
- args = [ os.path.expanduser(os.path.expandvars(x)) for x in args if x is not None ]
+ args = [os.path.expanduser(os.path.expandvars(x)) for x in args if x is not None]
rc = 0
msg = None
@@ -245,9 +288,9 @@
# Clean out python paths set by ansiballz
if 'PYTHONPATH' in os.environ:
pypaths = os.environ['PYTHONPATH'].split(':')
- pypaths = [x for x in pypaths \
- if not x.endswith('/ansible_modlib.zip') \
- and not x.endswith('/debug_dir')]
+ pypaths = [x for x in pypaths
+ if not x.endswith('/ansible_modlib.zip') and
+ not x.endswith('/debug_dir')]
os.environ['PYTHONPATH'] = ':'.join(pypaths)
if not os.environ['PYTHONPATH']:
del os.environ['PYTHONPATH']
@@ -256,8 +299,13 @@
# in reporting later, which strips out things like
# passwords from the args list
to_clean_args = args
- # ZUUL: Hardcode python2 until we're on ansible 2.2
- if isinstance(args, (unicode, str)):
+ if PY2:
+ if isinstance(args, text_type):
+ to_clean_args = to_bytes(args)
+ else:
+ if isinstance(args, binary_type):
+ to_clean_args = to_text(args)
+ if isinstance(args, (text_type, binary_type)):
to_clean_args = shlex.split(to_clean_args)
clean_args = []
@@ -291,34 +339,35 @@
stderr=subprocess.STDOUT,
)
- if cwd and os.path.isdir(cwd):
- kwargs['cwd'] = cwd
-
# store the pwd
prev_dir = os.getcwd()
# make sure we're in the right working directory
if cwd and os.path.isdir(cwd):
+ cwd = os.path.abspath(os.path.expanduser(cwd))
+ kwargs['cwd'] = cwd
try:
os.chdir(cwd)
except (OSError, IOError):
e = get_exception()
self.fail_json(rc=e.errno, msg="Could not open %s, %s" % (cwd, str(e)))
- try:
+ old_umask = None
+ if umask:
+ old_umask = os.umask(umask)
+ try:
if self._debug:
- if isinstance(args, list):
- running = ' '.join(args)
- else:
- running = args
- self.log('Executing: ' + running)
+ self.log('Executing: ' + clean_args)
+
# ZUUL: Replaced the excution loop with the zuul_runner run function
cmd = subprocess.Popen(args, **kwargs)
t = threading.Thread(target=follow, args=(cmd.stdout, zuul_log_id))
t.daemon = True
t.start()
+
ret = cmd.wait()
+
# Give the thread that is writing the console log up to 10 seconds
# to catch up and exit. If it hasn't done so by then, it is very
# likely stuck in readline() because it spawed a child that is
@@ -334,19 +383,21 @@
# we can't close stdout (attempting to do so raises an
# exception) , so this is disabled.
# cmd.stdout.close()
+ # cmd.stderr.close()
# ZUUL: stdout and stderr are in the console log file
# ZUUL: return the saved log lines so we can ship them back
- stdout = ''.join(_log_lines)
- stderr = ''
+ stdout = b('').join(_log_lines)
+ stderr = b('')
rc = cmd.returncode
except (OSError, IOError):
e = get_exception()
- self.fail_json(rc=e.errno, msg=str(e), cmd=clean_args)
+ self.log("Error Executing CMD:%s Exception:%s" % (clean_args, to_native(e)))
+ self.fail_json(rc=e.errno, msg=to_native(e), cmd=clean_args)
except Exception:
- e = get_exception()
- self.fail_json(rc=257, msg=str(e), exception=traceback.format_exc(), cmd=clean_args)
+ self.log("Error Executing CMD:%s Exception:%s" % (clean_args, to_native(traceback.format_exc())))
+ self.fail_json(rc=257, msg=to_native(e), exception=traceback.format_exc(), cmd=clean_args)
# Restore env settings
for key, val in old_env_vals.items():
@@ -355,6 +406,9 @@
else:
os.environ[key] = val
+ if old_umask:
+ os.umask(old_umask)
+
if rc != 0 and check_rc:
msg = heuristic_log_sanitize(stderr.rstrip(), self.no_log_values)
self.fail_json(cmd=clean_args, rc=rc, stdout=stdout, stderr=stderr, msg=msg)
@@ -362,6 +416,9 @@
# reset the pwd
os.chdir(prev_dir)
+ if encoding is not None:
+ return (rc, to_native(stdout, encoding=encoding, errors=errors),
+ to_native(stderr, encoding=encoding, errors=errors))
return (rc, stdout, stderr)
@@ -392,24 +449,24 @@
# hence don't copy this one if you are looking to build others!
module = AnsibleModule(
argument_spec=dict(
- _raw_params = dict(),
- _uses_shell = dict(type='bool', default=False),
- chdir = dict(type='path'),
- executable = dict(),
- creates = dict(type='path'),
- removes = dict(type='path'),
- warn = dict(type='bool', default=True),
- environ = dict(type='dict', default=None),
- zuul_log_id = dict(type='str'),
+ _raw_params = dict(),
+ _uses_shell = dict(type='bool', default=False),
+ chdir = dict(type='path'),
+ executable = dict(),
+ creates = dict(type='path'),
+ removes = dict(type='path'),
+ warn = dict(type='bool', default=True),
+ environ = dict(type='dict', default=None),
+ zuul_log_id = dict(type='str'),
)
)
shell = module.params['_uses_shell']
chdir = module.params['chdir']
executable = module.params['executable']
- args = module.params['_raw_params']
- creates = module.params['creates']
- removes = module.params['removes']
+ args = module.params['_raw_params']
+ creates = module.params['creates']
+ removes = module.params['removes']
warn = module.params['warn']
environ = module.params['environ']
zuul_log_id = module.params['zuul_log_id']
@@ -434,9 +491,9 @@
)
if removes:
- # do not run the command if the line contains removes=filename
- # and the filename does not exist. This allows idempotence
- # of command executions.
+ # do not run the command if the line contains removes=filename
+ # and the filename does not exist. This allows idempotence
+ # of command executions.
if not glob.glob(removes):
module.exit_json(
cmd=args,
@@ -453,20 +510,20 @@
args = shlex.split(args)
startd = datetime.datetime.now()
- rc, out, err = zuul_run_command(module, args, zuul_log_id, executable=executable, use_unsafe_shell=shell, environ_update=environ)
+ rc, out, err = zuul_run_command(module, args, zuul_log_id, executable=executable, use_unsafe_shell=shell, encoding=None, environ_update=environ)
endd = datetime.datetime.now()
delta = endd - startd
if out is None:
- out = ''
+ out = b('')
if err is None:
- err = ''
+ err = b('')
module.exit_json(
cmd = args,
- stdout = out.rstrip("\r\n"),
- stderr = err.rstrip("\r\n"),
+ stdout = out.rstrip(b("\r\n")),
+ stderr = err.rstrip(b("\r\n")),
rc = rc,
start = str(startd),
end = str(endd),
diff --git a/zuul/cmd/__init__.py b/zuul/cmd/__init__.py
index d31c5b8..8610114 100755
--- a/zuul/cmd/__init__.py
+++ b/zuul/cmd/__init__.py
@@ -14,9 +14,9 @@
# License for the specific language governing permissions and limitations
# under the License.
-import six
-from six.moves import configparser as ConfigParser
+import configparser
import extras
+import io
import logging
import logging.config
import os
@@ -48,7 +48,7 @@
yappi.start()
else:
yappi.stop()
- yappi_out = six.BytesIO()
+ yappi_out = io.BytesIO()
yappi.get_func_stats().print_all(out=yappi_out)
yappi.get_thread_stats().print_all(out=yappi_out)
log.debug(yappi_out.getvalue())
@@ -69,7 +69,7 @@
return "Zuul version: %s" % zuul_version_info.release_string()
def read_config(self):
- self.config = ConfigParser.ConfigParser()
+ self.config = configparser.ConfigParser()
if self.args.config:
locations = [self.args.config]
else:
diff --git a/zuul/cmd/client.py b/zuul/cmd/client.py
old mode 100644
new mode 100755
index 3f67a38..dec15e7
--- a/zuul/cmd/client.py
+++ b/zuul/cmd/client.py
@@ -25,6 +25,7 @@
import zuul.rpcclient
import zuul.cmd
+from zuul.lib.config import get_default
class Client(zuul.cmd.ZuulApp):
@@ -95,10 +96,11 @@
'running-jobs',
help='show the running jobs'
)
+ running_jobs_columns = list(self._show_running_jobs_columns().keys())
show_running_jobs.add_argument(
'--columns',
help="comma separated list of columns to display (or 'ALL')",
- choices=self._show_running_jobs_columns().keys().append('ALL'),
+ choices=running_jobs_columns.append('ALL'),
default='name, worker.name, start_time, result'
)
@@ -121,10 +123,10 @@
self.setup_logging()
self.server = self.config.get('gearman', 'server')
- if self.config.has_option('gearman', 'port'):
- self.port = self.config.get('gearman', 'port')
- else:
- self.port = 4730
+ self.port = get_default(self.config, 'gearman', 'port', 4730)
+ self.ssl_key = get_default(self.config, 'gearman', 'ssl_key')
+ self.ssl_cert = get_default(self.config, 'gearman', 'ssl_cert')
+ self.ssl_ca = get_default(self.config, 'gearman', 'ssl_ca')
if self.args.func():
sys.exit(0)
@@ -132,7 +134,8 @@
sys.exit(1)
def enqueue(self):
- client = zuul.rpcclient.RPCClient(self.server, self.port)
+ client = zuul.rpcclient.RPCClient(
+ self.server, self.port, self.ssl_key, self.ssl_cert, self.ssl_ca)
r = client.enqueue(tenant=self.args.tenant,
pipeline=self.args.pipeline,
project=self.args.project,
@@ -141,7 +144,8 @@
return r
def enqueue_ref(self):
- client = zuul.rpcclient.RPCClient(self.server, self.port)
+ client = zuul.rpcclient.RPCClient(
+ self.server, self.port, self.ssl_key, self.ssl_cert, self.ssl_ca)
r = client.enqueue_ref(tenant=self.args.tenant,
pipeline=self.args.pipeline,
project=self.args.project,
@@ -152,14 +156,16 @@
return r
def promote(self):
- client = zuul.rpcclient.RPCClient(self.server, self.port)
+ client = zuul.rpcclient.RPCClient(
+ self.server, self.port, self.ssl_key, self.ssl_cert, self.ssl_ca)
r = client.promote(tenant=self.args.tenant,
pipeline=self.args.pipeline,
change_ids=self.args.changes)
return r
def show_running_jobs(self):
- client = zuul.rpcclient.RPCClient(self.server, self.port)
+ client = zuul.rpcclient.RPCClient(
+ self.server, self.port, self.ssl_key, self.ssl_cert, self.ssl_ca)
running_items = client.get_running_jobs()
if len(running_items) == 0:
diff --git a/zuul/cmd/executor.py b/zuul/cmd/executor.py
index 7cc8dd8..57ecfa3 100755
--- a/zuul/cmd/executor.py
+++ b/zuul/cmd/executor.py
@@ -32,6 +32,7 @@
import zuul.cmd
import zuul.executor.server
+from zuul.lib.config import get_default
# No zuul imports that pull in paramiko here; it must not be
# imported until after the daemonization.
@@ -63,11 +64,8 @@
self.args = parser.parse_args()
def send_command(self, cmd):
- if self.config.has_option('zuul', 'state_dir'):
- state_dir = os.path.expanduser(
- self.config.get('zuul', 'state_dir'))
- else:
- state_dir = '/var/lib/zuul'
+ state_dir = get_default(self.config, 'zuul', 'state_dir',
+ '/var/lib/zuul', expand_user=True)
path = os.path.join(state_dir, 'executor.socket')
s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
s.connect(path)
@@ -114,10 +112,7 @@
def main(self, daemon=True):
# See comment at top of file about zuul imports
- if self.config.has_option('executor', 'user'):
- self.user = self.config.get('executor', 'user')
- else:
- self.user = 'zuul'
+ self.user = get_default(self.config, 'executor', 'user', 'zuul')
if self.config.has_option('zuul', 'jobroot_dir'):
self.jobroot_dir = os.path.expanduser(
@@ -132,10 +127,8 @@
self.setup_logging('executor', 'log_config')
self.log = logging.getLogger("zuul.Executor")
- if self.config.has_option('executor', 'finger_port'):
- self.finger_port = int(self.config.get('executor', 'finger_port'))
- else:
- self.finger_port = DEFAULT_FINGER_PORT
+ self.finger_port = int(get_default(self.config, 'executor',
+ 'finger_port', DEFAULT_FINGER_PORT))
self.start_log_streamer()
self.change_privs()
@@ -170,10 +163,9 @@
server.configure_connections(source_only=True)
- if server.config.has_option('executor', 'pidfile'):
- pid_fn = os.path.expanduser(server.config.get('executor', 'pidfile'))
- else:
- pid_fn = '/var/run/zuul-executor/zuul-executor.pid'
+ pid_fn = get_default(server.config, 'executor', 'pidfile',
+ '/var/run/zuul-executor/zuul-executor.pid',
+ expand_user=True)
pid = pid_file_module.TimeoutPIDLockFile(pid_fn, 10)
if server.args.nodaemon:
diff --git a/zuul/cmd/merger.py b/zuul/cmd/merger.py
index 686f34a..97f208c 100755
--- a/zuul/cmd/merger.py
+++ b/zuul/cmd/merger.py
@@ -27,6 +27,7 @@
import signal
import zuul.cmd
+from zuul.lib.config import get_default
# No zuul imports here because they pull in paramiko which must not be
# imported until after the daemonization.
@@ -79,10 +80,8 @@
server.read_config()
server.configure_connections(source_only=True)
- if server.config.has_option('zuul', 'state_dir'):
- state_dir = os.path.expanduser(server.config.get('zuul', 'state_dir'))
- else:
- state_dir = '/var/lib/zuul'
+ state_dir = get_default(server.config, 'zuul', 'state_dir',
+ '/var/lib/zuul', expand_user=True)
test_fn = os.path.join(state_dir, 'test')
try:
f = open(test_fn, 'w')
@@ -92,10 +91,9 @@
print("\nUnable to write to state directory: %s\n" % state_dir)
raise
- if server.config.has_option('merger', 'pidfile'):
- pid_fn = os.path.expanduser(server.config.get('merger', 'pidfile'))
- else:
- pid_fn = '/var/run/zuul-merger/zuul-merger.pid'
+ pid_fn = get_default(server.config, 'merger', 'pidfile',
+ '/var/run/zuul-merger/zuul-merger.pid',
+ expand_user=True)
pid = pid_file_module.TimeoutPIDLockFile(pid_fn, 10)
if server.args.nodaemon:
diff --git a/zuul/cmd/scheduler.py b/zuul/cmd/scheduler.py
index 5328bba..b32deaf 100755
--- a/zuul/cmd/scheduler.py
+++ b/zuul/cmd/scheduler.py
@@ -28,6 +28,7 @@
import signal
import zuul.cmd
+from zuul.lib.config import get_default
# No zuul imports here because they pull in paramiko which must not be
# imported until after the daemonization.
@@ -98,11 +99,14 @@
import zuul.lib.gearserver
statsd_host = os.environ.get('STATSD_HOST')
statsd_port = int(os.environ.get('STATSD_PORT', 8125))
- if self.config.has_option('gearman_server', 'listen_address'):
- host = self.config.get('gearman_server', 'listen_address')
- else:
- host = None
+ host = get_default(self.config, 'gearman_server', 'listen_address')
+ ssl_key = get_default(self.config, 'gearman_server', 'ssl_key')
+ ssl_cert = get_default(self.config, 'gearman_server', 'ssl_cert')
+ ssl_ca = get_default(self.config, 'gearman_server', 'ssl_ca')
zuul.lib.gearserver.GearServer(4730,
+ ssl_key=ssl_key,
+ ssl_cert=ssl_cert,
+ ssl_ca=ssl_ca,
host=host,
statsd_host=statsd_host,
statsd_port=statsd_port,
@@ -146,27 +150,16 @@
nodepool = zuul.nodepool.Nodepool(self.sched)
zookeeper = zuul.zk.ZooKeeper()
- if self.config.has_option('zuul', 'zookeeper_hosts'):
- zookeeper_hosts = self.config.get('zuul', 'zookeeper_hosts')
- else:
- zookeeper_hosts = '127.0.0.1:2181'
+ zookeeper_hosts = get_default(self.config, 'zuul', 'zookeeper_hosts',
+ '127.0.0.1:2181')
zookeeper.connect(zookeeper_hosts)
- if self.config.has_option('zuul', 'status_expiry'):
- cache_expiry = self.config.getint('zuul', 'status_expiry')
- else:
- cache_expiry = 1
+ cache_expiry = get_default(self.config, 'zuul', 'status_expiry', 1)
- if self.config.has_option('webapp', 'listen_address'):
- listen_address = self.config.get('webapp', 'listen_address')
- else:
- listen_address = '0.0.0.0'
-
- if self.config.has_option('webapp', 'port'):
- port = self.config.getint('webapp', 'port')
- else:
- port = 8001
+ listen_address = get_default(self.config, 'webapp', 'listen_address',
+ '0.0.0.0')
+ port = get_default(self.config, 'webapp', 'port', 8001)
webapp = zuul.webapp.WebApp(
self.sched, port=port, cache_expiry=cache_expiry,
@@ -215,10 +208,9 @@
if scheduler.args.validate:
sys.exit(scheduler.test_config())
- if scheduler.config.has_option('zuul', 'pidfile'):
- pid_fn = os.path.expanduser(scheduler.config.get('zuul', 'pidfile'))
- else:
- pid_fn = '/var/run/zuul-scheduler/zuul-scheduler.pid'
+ pid_fn = get_default(scheduler.config, 'zuul', 'pidfile',
+ '/var/run/zuul-scheduler/zuul-scheduler.pid',
+ expand_user=True)
pid = pid_file_module.TimeoutPIDLockFile(pid_fn, 10)
if scheduler.args.nodaemon:
diff --git a/zuul/configloader.py b/zuul/configloader.py
index 5e0fe65..84227f8 100644
--- a/zuul/configloader.py
+++ b/zuul/configloader.py
@@ -15,7 +15,6 @@
import copy
import os
import logging
-import six
import pprint
import textwrap
@@ -402,18 +401,15 @@
job.inheritFrom(parent)
for pre_run_name in as_list(conf.get('pre-run')):
- full_pre_run_name = os.path.join('playbooks', pre_run_name)
pre_run = model.PlaybookContext(job.source_context,
- full_pre_run_name)
+ pre_run_name)
job.pre_run = job.pre_run + (pre_run,)
for post_run_name in as_list(conf.get('post-run')):
- full_post_run_name = os.path.join('playbooks', post_run_name)
post_run = model.PlaybookContext(job.source_context,
- full_post_run_name)
+ post_run_name)
job.post_run = (post_run,) + job.post_run
if 'run' in conf:
- run_name = os.path.join('playbooks', conf['run'])
- run = model.PlaybookContext(job.source_context, run_name)
+ run = model.PlaybookContext(job.source_context, conf['run'])
job.run = (run,)
else:
if not project_pipeline:
@@ -427,7 +423,7 @@
setattr(job, a, conf[k])
if 'nodes' in conf:
conf_nodes = conf['nodes']
- if isinstance(conf_nodes, six.string_types):
+ if isinstance(conf_nodes, str):
# This references an existing named nodeset in the layout.
ns = layout.nodesets[conf_nodes]
else:
@@ -576,7 +572,7 @@
def _parseJobList(tenant, layout, conf, source_context,
start_mark, job_list):
for conf_job in conf:
- if isinstance(conf_job, six.string_types):
+ if isinstance(conf_job, str):
attrs = dict(name=conf_job)
elif isinstance(conf_job, dict):
# A dictionary in a job tree may override params
@@ -1007,7 +1003,7 @@
@staticmethod
def _getProject(source, conf, current_include):
- if isinstance(conf, six.string_types):
+ if isinstance(conf, str):
# Return a project object whether conf is a dict or a str
project = source.getProject(conf)
project_include = current_include
@@ -1031,7 +1027,7 @@
def _getProjects(source, conf, current_include):
# Return a project object whether conf is a dict or a str
projects = []
- if isinstance(conf, six.string_types):
+ if isinstance(conf, str):
# A simple project name string
projects.append(TenantParser._getProject(
source, conf, current_include))
diff --git a/zuul/connection/__init__.py b/zuul/connection/__init__.py
index 90ab39c..3655115 100644
--- a/zuul/connection/__init__.py
+++ b/zuul/connection/__init__.py
@@ -15,11 +15,9 @@
import abc
import extras
-import six
-@six.add_metaclass(abc.ABCMeta)
-class BaseConnection(object):
+class BaseConnection(object, metaclass=abc.ABCMeta):
"""Base class for connections.
A connection is a shared object that sources, triggers and reporters can
diff --git a/zuul/driver/__init__.py b/zuul/driver/__init__.py
index 0c3105d..c78283d 100644
--- a/zuul/driver/__init__.py
+++ b/zuul/driver/__init__.py
@@ -14,11 +14,8 @@
import abc
-import six
-
-@six.add_metaclass(abc.ABCMeta)
-class Driver(object):
+class Driver(object, metaclass=abc.ABCMeta):
"""A Driver is an extension component of Zuul that supports
interfacing with a remote system. It can support any of the following
interfaces (but must support at least one to be useful):
@@ -80,8 +77,7 @@
pass
-@six.add_metaclass(abc.ABCMeta)
-class ConnectionInterface(object):
+class ConnectionInterface(object, metaclass=abc.ABCMeta):
"""The Connection interface.
A driver which is able to supply a Connection should implement
@@ -124,8 +120,7 @@
pass
-@six.add_metaclass(abc.ABCMeta)
-class TriggerInterface(object):
+class TriggerInterface(object, metaclass=abc.ABCMeta):
"""The trigger interface.
A driver which is able to supply a trigger should implement this
@@ -167,8 +162,7 @@
pass
-@six.add_metaclass(abc.ABCMeta)
-class SourceInterface(object):
+class SourceInterface(object, metaclass=abc.ABCMeta):
"""The source interface to be implemented by a driver.
A driver which is able to supply a Source should implement this
@@ -216,8 +210,7 @@
pass
-@six.add_metaclass(abc.ABCMeta)
-class ReporterInterface(object):
+class ReporterInterface(object, metaclass=abc.ABCMeta):
"""The reporter interface to be implemented by a driver.
A driver which is able to supply a Reporter should implement this
@@ -256,8 +249,7 @@
pass
-@six.add_metaclass(abc.ABCMeta)
-class WrapperInterface(object):
+class WrapperInterface(object, metaclass=abc.ABCMeta):
"""The wrapper interface to be implmeneted by a driver.
A driver which wraps execution of commands executed by Zuul should
@@ -278,3 +270,13 @@
:rtype: Callable
"""
pass
+
+ @abc.abstractmethod
+ def setMountsMap(self, state_dir, ro_dirs=[], rw_dirs=[]):
+ """Add additional mount point to the execution environment.
+
+ :arg str state_dir: the state directory to be read write
+ :arg list ro_dirs: read only directories paths
+ :arg list rw_dirs: read write directories paths
+ """
+ pass
diff --git a/zuul/driver/bubblewrap/__init__.py b/zuul/driver/bubblewrap/__init__.py
index c93e912..9e9a26e 100644
--- a/zuul/driver/bubblewrap/__init__.py
+++ b/zuul/driver/bubblewrap/__init__.py
@@ -19,11 +19,10 @@
import logging
import os
import pwd
+import shlex
import subprocess
import sys
-from six.moves import shlex_quote
-
from zuul.driver import (Driver, WrapperInterface)
@@ -84,20 +83,21 @@
'--ro-bind', '/bin', '/bin',
'--ro-bind', '/sbin', '/sbin',
'--ro-bind', '/etc/resolv.conf', '/etc/resolv.conf',
- '--ro-bind', '{ansible_dir}', '{ansible_dir}',
'--ro-bind', '{ssh_auth_sock}', '{ssh_auth_sock}',
'--dir', '{work_dir}',
'--bind', '{work_dir}', '{work_dir}',
'--dev', '/dev',
'--dir', '{user_home}',
- '--chdir', '/',
+ '--chdir', '{work_dir}',
'--unshare-all',
'--share-net',
+ '--die-with-parent',
'--uid', '{uid}',
'--gid', '{gid}',
'--file', '{uid_fd}', '/etc/passwd',
'--file', '{gid_fd}', '/etc/group',
]
+ mounts_map = {'rw': [], 'ro': []}
def reconfigure(self, tenant):
pass
@@ -105,6 +105,9 @@
def stop(self):
pass
+ def setMountsMap(self, state_dir, ro_dirs=[], rw_dirs=[]):
+ self.mounts_map = {'ro': ro_dirs, 'rw': [state_dir] + rw_dirs}
+
def getPopen(self, **kwargs):
# Set zuul_dir if it was not passed in
if 'zuul_dir' in kwargs:
@@ -118,6 +121,11 @@
if not zuul_dir.startswith('/usr'):
bwrap_command.extend(['--ro-bind', zuul_dir, zuul_dir])
+ for mount_type in ('ro', 'rw'):
+ bind_arg = '--ro-bind' if mount_type == 'ro' else '--bind'
+ for bind in self.mounts_map[mount_type]:
+ bwrap_command.extend([bind_arg, bind, bind])
+
# Need users and groups
uid = os.getuid()
passwd = pwd.getpwuid(uid)
@@ -125,6 +133,7 @@
['{}'.format(x).encode('utf8') for x in passwd])
(passwd_r, passwd_w) = os.pipe()
os.write(passwd_w, passwd_bytes)
+ os.write(passwd_w, b'\n')
os.close(passwd_w)
gid = os.getgid()
@@ -133,6 +142,7 @@
['{}'.format(x).encode('utf8') for x in group])
group_r, group_w = os.pipe()
os.write(group_w, group_bytes)
+ os.write(group_w, b'\n')
os.close(group_w)
kwargs = dict(kwargs) # Don't update passed in dict
@@ -144,7 +154,7 @@
command = [x.format(**kwargs) for x in bwrap_command]
self.log.debug("Bubblewrap command: %s",
- " ".join(shlex_quote(c) for c in command))
+ " ".join(shlex.quote(c) for c in command))
wrapped_popen = WrappedPopen(command, passwd_r, group_r)
@@ -152,18 +162,18 @@
def main(args=None):
+ logging.basicConfig(level=logging.DEBUG)
+
driver = BubblewrapDriver()
parser = argparse.ArgumentParser()
parser.add_argument('work_dir')
- parser.add_argument('ansible_dir')
parser.add_argument('run_args', nargs='+')
cli_args = parser.parse_args()
ssh_auth_sock = os.environ.get('SSH_AUTH_SOCK')
popen = driver.getPopen(work_dir=cli_args.work_dir,
- ansible_dir=cli_args.ansible_dir,
ssh_auth_sock=ssh_auth_sock)
x = popen(cli_args.run_args)
x.wait()
diff --git a/zuul/driver/gerrit/gerritconnection.py b/zuul/driver/gerrit/gerritconnection.py
index fa43e66..39a81bc 100644
--- a/zuul/driver/gerrit/gerritconnection.py
+++ b/zuul/driver/gerrit/gerritconnection.py
@@ -18,11 +18,11 @@
import select
import threading
import time
-from six.moves import queue as Queue
-from six.moves import shlex_quote
import paramiko
import logging
import pprint
+import shlex
+import queue
import voluptuous as v
from zuul.connection import BaseConnection
@@ -31,20 +31,6 @@
from zuul.driver.gerrit.gerritmodel import GerritChange, GerritTriggerEvent
-# Walk the change dependency tree to find a cycle
-def detect_cycle(change, history=None):
- if history is None:
- history = []
- else:
- history = history[:]
- history.append(change.number)
- for dep in change.needs_changes:
- if dep.number in history:
- raise Exception("Dependency cycle detected: %s in %s" % (
- dep.number, history))
- detect_cycle(dep, history)
-
-
class GerritEventConnector(threading.Thread):
"""Move events from Gerrit to the scheduler."""
@@ -274,7 +260,7 @@
self.keyfile = self.connection_config.get('sshkey', None)
self.keepalive = int(self.connection_config.get('keepalive', 60))
self.watcher_thread = None
- self.event_queue = Queue.Queue()
+ self.event_queue = queue.Queue()
self.client = None
self.baseurl = self.connection_config.get('baseurl',
@@ -383,6 +369,13 @@
return records
def _updateChange(self, change, history=None):
+
+ # In case this change is already in the history we have a cyclic
+ # dependency and don't need to update ourselves again as this gets
+ # done in a previous frame of the call stack.
+ if history and change.number in history:
+ return change
+
self.log.info("Updating %s" % (change,))
data = self.query(change.number)
change._data = data
@@ -432,18 +425,9 @@
if 'dependsOn' in data:
parts = data['dependsOn'][0]['ref'].split('/')
dep_num, dep_ps = parts[3], parts[4]
- if dep_num in history:
- raise Exception("Dependency cycle detected: %s in %s" % (
- dep_num, history))
self.log.debug("Updating %s: Getting git-dependent change %s,%s" %
(change, dep_num, dep_ps))
dep = self._getChange(dep_num, dep_ps, history=history)
- # Because we are not forcing a refresh in _getChange, it
- # may return without executing this code, so if we are
- # updating our change to add ourselves to a dependency
- # cycle, we won't detect it. By explicitly performing a
- # walk of the dependency tree, we will.
- detect_cycle(dep, history)
if (not dep.is_merged) and dep not in needs_changes:
needs_changes.append(dep)
@@ -451,19 +435,10 @@
change):
dep_num = record['number']
dep_ps = record['currentPatchSet']['number']
- if dep_num in history:
- raise Exception("Dependency cycle detected: %s in %s" % (
- dep_num, history))
self.log.debug("Updating %s: Getting commit-dependent "
"change %s,%s" %
(change, dep_num, dep_ps))
dep = self._getChange(dep_num, dep_ps, history=history)
- # Because we are not forcing a refresh in _getChange, it
- # may return without executing this code, so if we are
- # updating our change to add ourselves to a dependency
- # cycle, we won't detect it. By explicitly performing a
- # walk of the dependency tree, we will.
- detect_cycle(dep, history)
if (not dep.is_merged) and dep not in needs_changes:
needs_changes.append(dep)
change.needs_changes = needs_changes
@@ -475,7 +450,7 @@
dep_num, dep_ps = parts[3], parts[4]
self.log.debug("Updating %s: Getting git-needed change %s,%s" %
(change, dep_num, dep_ps))
- dep = self._getChange(dep_num, dep_ps)
+ dep = self._getChange(dep_num, dep_ps, history=history)
if (not dep.is_merged) and dep.is_current_patchset:
needed_by_changes.append(dep)
@@ -487,8 +462,11 @@
# Because a commit needed-by may be a cross-repo
# dependency, cause that change to refresh so that it will
# reference the latest patchset of its Depends-On (this
- # change).
- dep = self._getChange(dep_num, dep_ps, refresh=True)
+ # change). In case the dep is already in history we already
+ # refreshed this change so refresh is not needed in this case.
+ refresh = dep_num not in history
+ dep = self._getChange(
+ dep_num, dep_ps, refresh=refresh, history=history)
if (not dep.is_merged) and dep.is_current_patchset:
needed_by_changes.append(dep)
change.needed_by_changes = needed_by_changes
@@ -628,7 +606,7 @@
def review(self, project, change, message, action={}):
cmd = 'gerrit review --project %s' % project
if message:
- cmd += ' --message %s' % shlex_quote(message)
+ cmd += ' --message %s' % shlex.quote(message)
for key, val in action.items():
if val is True:
cmd += ' --%s' % key
diff --git a/zuul/driver/git/gitconnection.py b/zuul/driver/git/gitconnection.py
index ca88d3f..f4fe7e5 100644
--- a/zuul/driver/git/gitconnection.py
+++ b/zuul/driver/git/gitconnection.py
@@ -14,7 +14,7 @@
# under the License.
import logging
-from six.moves import urllib
+import urllib
import voluptuous as v
diff --git a/zuul/driver/github/githubconnection.py b/zuul/driver/github/githubconnection.py
index 4910e51..7a3491e 100644
--- a/zuul/driver/github/githubconnection.py
+++ b/zuul/driver/github/githubconnection.py
@@ -253,7 +253,7 @@
raise webob.exc.HTTPUnauthorized(
'Please specify a X-Hub-Signature header with secret.')
- payload_signature = 'sha1=' + hmac.new(secret,
+ payload_signature = 'sha1=' + hmac.new(secret.encode('utf-8'),
body,
hashlib.sha1).hexdigest()
@@ -524,6 +524,7 @@
change.patchset)
change.reviews = self.getPullReviews(change.project,
change.number)
+ change.labels = change.pr.get('labels')
return change
@@ -565,9 +566,18 @@
def getPull(self, project_name, number):
github = self.getGithubClient(project_name)
owner, proj = project_name.split('/')
- probj = github.pull_request(owner, proj, number)
+ for retry in range(5):
+ probj = github.pull_request(owner, proj, number)
+ if probj is not None:
+ break
+ self.log.warning("Pull request #%s of %s/%s returned None!" % (
+ number, owner, proj))
+ time.sleep(1)
+ # Get the issue obj so we can get the labels (this is silly)
+ issueobj = probj.issue()
pr = probj.as_dict()
pr['files'] = [f.filename for f in probj.files()]
+ pr['labels'] = [l.name for l in issueobj.labels()]
log_rate_limit(self.log, github)
return pr
@@ -593,7 +603,7 @@
if not pr_url:
continue
# the issue provides no good description of the project :\
- owner, project, _, number = pr_url.split('/')[4:]
+ owner, project, _, number = pr_url.split('/')[-4:]
github = self.getGithubClient("%s/%s" % (owner, project))
pr = github.pull_request(owner, project, number)
if pr.head.sha != sha:
diff --git a/zuul/driver/github/githubmodel.py b/zuul/driver/github/githubmodel.py
index 9516097..db119f0 100644
--- a/zuul/driver/github/githubmodel.py
+++ b/zuul/driver/github/githubmodel.py
@@ -28,9 +28,13 @@
class PullRequest(Change):
def __init__(self, project):
super(PullRequest, self).__init__(project)
+ self.project = None
+ self.pr = None
self.updated_at = None
self.title = None
self.reviews = []
+ self.files = []
+ self.labels = []
def isUpdateOf(self, other):
if (hasattr(other, 'number') and self.number == other.number and
@@ -60,9 +64,12 @@
class GithubCommonFilter(object):
- def __init__(self, required_reviews=[], required_statuses=[]):
+ def __init__(self, required_reviews=[], required_statuses=[],
+ reject_reviews=[]):
self._required_reviews = copy.deepcopy(required_reviews)
+ self._reject_reviews = copy.deepcopy(reject_reviews)
self.required_reviews = self._tidy_reviews(required_reviews)
+ self.reject_reviews = self._tidy_reviews(reject_reviews)
self.required_statuses = required_statuses
def _tidy_reviews(self, reviews):
@@ -109,15 +116,17 @@
return True
def matchesReviews(self, change):
- if self.required_reviews:
+ if self.required_reviews or self.reject_reviews:
if not hasattr(change, 'number'):
# not a PR, no reviews
return False
- if not change.reviews:
- # No reviews means no matching
+ if self.required_reviews and not change.reviews:
+ # No reviews means no matching of required bits
+ # having reject reviews but no reviews on the change is okay
return False
- return self.matchesRequiredReviews(change)
+ return (self.matchesRequiredReviews(change) and
+ self.matchesNoRejectReviews(change))
def matchesRequiredReviews(self, change):
for rreview in self.required_reviews:
@@ -131,6 +140,14 @@
return False
return True
+ def matchesNoRejectReviews(self, change):
+ for rreview in self.reject_reviews:
+ for review in change.reviews:
+ if self._match_review_required_review(rreview, review):
+ # A review matched, we can reject right away
+ return False
+ return True
+
def matchesRequiredStatuses(self, change):
# statuses are ORed
# A PR head can have multiple statuses on it. If the change
@@ -271,14 +288,17 @@
class GithubRefFilter(RefFilter, GithubCommonFilter):
def __init__(self, connection_name, statuses=[], required_reviews=[],
- open=None, current_patchset=None):
+ reject_reviews=[], open=None, current_patchset=None,
+ labels=[]):
RefFilter.__init__(self, connection_name)
GithubCommonFilter.__init__(self, required_reviews=required_reviews,
+ reject_reviews=reject_reviews,
required_statuses=statuses)
self.statuses = statuses
self.open = open
self.current_patchset = current_patchset
+ self.labels = labels
def __repr__(self):
ret = '<GithubRefFilter'
@@ -289,10 +309,15 @@
if self.required_reviews:
ret += (' required-reviews: %s' %
str(self.required_reviews))
+ if self.reject_reviews:
+ ret += (' reject-reviews: %s' %
+ str(self.reject_reviews))
if self.open:
ret += ' open: %s' % self.open
if self.current_patchset:
ret += ' current-patchset: %s' % self.current_patchset
+ if self.labels:
+ ret += ' labels: %s' % self.labels
ret += '>'
@@ -320,8 +345,13 @@
else:
return False
- # required reviews are ANDed
+ # required reviews are ANDed (reject reviews are ORed)
if not self.matchesReviews(change):
return False
+ # required labels are ANDed
+ for label in self.labels:
+ if label not in change.labels:
+ return False
+
return True
diff --git a/zuul/driver/github/githubreporter.py b/zuul/driver/github/githubreporter.py
index 29edb8a..37cbe61 100644
--- a/zuul/driver/github/githubreporter.py
+++ b/zuul/driver/github/githubreporter.py
@@ -40,21 +40,25 @@
self._unlabels = [self._unlabels]
def report(self, item):
- """Comment on PR and set commit status."""
- if self._create_comment:
- self.addPullComment(item)
+ """Report on an event."""
+ # order is important for github branch protection.
+ # A status should be set before a merge attempt
if (self._commit_status is not None and
hasattr(item.change, 'patchset') and
item.change.patchset is not None):
- self.setPullStatus(item)
- if (self._merge and
- hasattr(item.change, 'number')):
- self.mergePull(item)
- if not item.change.is_merged:
- msg = self._formatItemReportMergeFailure(item)
- self.addPullComment(item, msg)
- if self._labels or self._unlabels:
- self.setLabels(item)
+ self.setCommitStatus(item)
+ # Comments, labels, and merges can only be performed on pull requests.
+ # If the change is not a pull request (e.g. a push) skip them.
+ if hasattr(item.change, 'number'):
+ if self._create_comment:
+ self.addPullComment(item)
+ if self._labels or self._unlabels:
+ self.setLabels(item)
+ if (self._merge):
+ self.mergePull(item)
+ if not item.change.is_merged:
+ msg = self._formatItemReportMergeFailure(item)
+ self.addPullComment(item, msg)
def addPullComment(self, item, comment=None):
message = comment or self._formatItemReport(item)
@@ -65,7 +69,7 @@
(item.change, self.config, message))
self.connection.commentPull(project, pr_number, message)
- def setPullStatus(self, item):
+ def setCommitStatus(self, item):
project = item.change.project.name
sha = item.change.patchset
context = '%s/%s' % (item.pipeline.layout.tenant.name,
diff --git a/zuul/driver/github/githubsource.py b/zuul/driver/github/githubsource.py
index 1c2f727..1bd280f 100644
--- a/zuul/driver/github/githubsource.py
+++ b/zuul/driver/github/githubsource.py
@@ -97,11 +97,16 @@
required_reviews=to_list(config.get('review')),
open=config.get('open'),
current_patchset=config.get('current-patchset'),
+ labels=to_list(config.get('label')),
)
return [f]
def getRejectFilters(self, config):
- return []
+ f = GithubRefFilter(
+ connection_name=self.connection.connection_name,
+ reject_reviews=to_list(config.get('review'))
+ )
+ return [f]
review = v.Schema({'username': str,
@@ -117,7 +122,8 @@
require = {'status': scalar_or_list(str),
'review': scalar_or_list(review),
'open': bool,
- 'current-patchset': bool}
+ 'current-patchset': bool,
+ 'label': scalar_or_list(str)}
return require
diff --git a/zuul/driver/nullwrap/__init__.py b/zuul/driver/nullwrap/__init__.py
index ebcd1da..50fea27 100644
--- a/zuul/driver/nullwrap/__init__.py
+++ b/zuul/driver/nullwrap/__init__.py
@@ -26,3 +26,6 @@
def getPopen(self, **kwargs):
return subprocess.Popen
+
+ def setMountsMap(self, **kwargs):
+ pass
diff --git a/zuul/executor/client.py b/zuul/executor/client.py
index f6961f3..2e17b3e 100644
--- a/zuul/executor/client.py
+++ b/zuul/executor/client.py
@@ -21,6 +21,7 @@
from uuid import uuid4
import zuul.model
+from zuul.lib.config import get_default
from zuul.model import Build
@@ -115,13 +116,12 @@
self.meta_jobs = {} # A list of meta-jobs like stop or describe
server = config.get('gearman', 'server')
- if config.has_option('gearman', 'port'):
- port = config.get('gearman', 'port')
- else:
- port = 4730
-
+ port = get_default(self.config, 'gearman', 'port', 4730)
+ ssl_key = get_default(self.config, 'gearman', 'ssl_key')
+ ssl_cert = get_default(self.config, 'gearman', 'ssl_cert')
+ ssl_ca = get_default(self.config, 'gearman', 'ssl_ca')
self.gearman = ZuulGearmanClient(self)
- self.gearman.addServer(server, port)
+ self.gearman.addServer(server, port, ssl_key, ssl_cert, ssl_ca)
self.cleanup_thread = GearmanCleanup(self)
self.cleanup_thread.start()
diff --git a/zuul/executor/server.py b/zuul/executor/server.py
index c498fa4..657a063 100644
--- a/zuul/executor/server.py
+++ b/zuul/executor/server.py
@@ -18,6 +18,7 @@
import os
import shutil
import signal
+import shlex
import socket
import subprocess
import tempfile
@@ -25,9 +26,9 @@
import time
import traceback
from zuul.lib.yamlutil import yaml
+from zuul.lib.config import get_default
import gear
-from six.moves import shlex_quote
import zuul.merger.merger
import zuul.ansible
@@ -181,7 +182,9 @@
# Ansible
self.ansible_root = os.path.join(self.root, 'ansible')
os.makedirs(self.ansible_root)
- self.known_hosts = os.path.join(self.ansible_root, 'known_hosts')
+ ssh_dir = os.path.join(self.work_root, '.ssh')
+ os.mkdir(ssh_dir, 0o700)
+ self.known_hosts = os.path.join(ssh_dir, 'known_hosts')
self.inventory = os.path.join(self.ansible_root, 'inventory.yaml')
self.playbooks = [] # The list of candidate playbooks
self.playbook = None # A pointer to the candidate we have chosen
@@ -373,33 +376,15 @@
unverbose=self.verboseOff,
)
- if self.config.has_option('executor', 'git_dir'):
- self.merge_root = self.config.get('executor', 'git_dir')
- else:
- self.merge_root = '/var/lib/zuul/executor-git'
-
- if self.config.has_option('executor', 'default_username'):
- self.default_username = self.config.get('executor',
- 'default_username')
- else:
- self.default_username = 'zuul'
-
- if self.config.has_option('merger', 'git_user_email'):
- self.merge_email = self.config.get('merger', 'git_user_email')
- else:
- self.merge_email = None
-
- if self.config.has_option('merger', 'git_user_name'):
- self.merge_name = self.config.get('merger', 'git_user_name')
- else:
- self.merge_name = None
-
- if self.config.has_option('executor', 'untrusted_wrapper'):
- untrusted_wrapper_name = self.config.get(
- 'executor', 'untrusted_wrapper').strip()
- else:
- untrusted_wrapper_name = 'bubblewrap'
- self.untrusted_wrapper = connections.drivers[untrusted_wrapper_name]
+ self.merge_root = get_default(self.config, 'executor', 'git_dir',
+ '/var/lib/zuul/executor-git')
+ self.default_username = get_default(self.config, 'executor',
+ 'default_username', 'zuul')
+ self.merge_email = get_default(self.config, 'merger', 'git_user_email')
+ self.merge_name = get_default(self.config, 'merger', 'git_user_name')
+ execution_wrapper_name = get_default(self.config, 'executor',
+ 'execution_wrapper', 'bubblewrap')
+ self.execution_wrapper = connections.drivers[execution_wrapper_name]
self.connections = connections
# This merger and its git repos are used to maintain
@@ -409,11 +394,8 @@
self.merger = self._getMerger(self.merge_root)
self.update_queue = DeduplicateQueue()
- if self.config.has_option('zuul', 'state_dir'):
- state_dir = os.path.expanduser(
- self.config.get('zuul', 'state_dir'))
- else:
- state_dir = '/var/lib/zuul'
+ state_dir = get_default(self.config, 'zuul', 'state_dir',
+ '/var/lib/zuul', expand_user=True)
path = os.path.join(state_dir, 'executor.socket')
self.command_socket = commandsocket.CommandSocket(path)
ansible_dir = os.path.join(state_dir, 'ansible')
@@ -455,14 +437,14 @@
self._running = True
self._command_running = True
server = self.config.get('gearman', 'server')
- if self.config.has_option('gearman', 'port'):
- port = self.config.get('gearman', 'port')
- else:
- port = 4730
+ port = get_default(self.config, 'gearman', 'port', 4730)
+ ssl_key = get_default(self.config, 'gearman', 'ssl_key')
+ ssl_cert = get_default(self.config, 'gearman', 'ssl_cert')
+ ssl_ca = get_default(self.config, 'gearman', 'ssl_ca')
self.merger_worker = ExecutorMergeWorker(self, 'Zuul Executor Merger')
- self.merger_worker.addServer(server, port)
+ self.merger_worker.addServer(server, port, ssl_key, ssl_cert, ssl_ca)
self.executor_worker = gear.TextWorker('Zuul Executor Server')
- self.executor_worker.addServer(server, port)
+ self.executor_worker.addServer(server, port, ssl_key, ssl_cert, ssl_ca)
self.log.debug("Waiting for server")
self.merger_worker.waitForServer()
self.executor_worker.waitForServer()
@@ -681,6 +663,13 @@
RESULT_UNREACHABLE = 3
RESULT_ABORTED = 4
+ RESULT_MAP = {
+ RESULT_NORMAL: 'RESULT_NORMAL',
+ RESULT_TIMED_OUT: 'RESULT_TIMED_OUT',
+ RESULT_UNREACHABLE: 'RESULT_UNREACHABLE',
+ RESULT_ABORTED: 'RESULT_ABORTED',
+ }
+
def __init__(self, executor_server, job):
logger = logging.getLogger("zuul.AnsibleJob")
self.log = AnsibleJobLogAdapter(logger, {'job': job.unique})
@@ -694,12 +683,9 @@
self.thread = None
self.ssh_agent = None
- if self.executor_server.config.has_option(
- 'executor', 'private_key_file'):
- self.private_key_file = self.executor_server.config.get(
- 'executor', 'private_key_file')
- else:
- self.private_key_file = '~/.ssh/id_rsa'
+ self.private_key_file = get_default(self.executor_server.config,
+ 'executor', 'private_key_file',
+ '~/.ssh/id_rsa')
self.ssh_agent = SshAgent()
def run(self):
@@ -799,7 +785,9 @@
data = {
'manager': self.executor_server.hostname,
- 'url': 'https://server/job/{}/0/'.format(args['job']),
+ 'url': 'finger://{server}/{unique}'.format(
+ unique=self.job.unique,
+ server=self.executor_server.hostname),
'worker_name': 'My Worker',
}
@@ -903,9 +891,10 @@
host_vars = dict(
ansible_host=ip,
ansible_user=self.executor_server.default_username,
- nodepool_az=node.get('az'),
- nodepool_provider=node.get('provider'),
- nodepool_region=node.get('region'))
+ nodepool=dict(
+ az=node.get('az'),
+ provider=node.get('provider'),
+ region=node.get('region')))
host_keys = []
for key in node.get('host_keys'):
@@ -1142,6 +1131,8 @@
def prepareAnsibleFiles(self, args):
all_vars = dict(args['vars'])
+ # TODO(mordred) Hack to work around running things with python3
+ all_vars['ansible_python_interpreter'] = '/usr/bin/python2'
all_vars['zuul']['executor'] = dict(
hostname=self.executor_server.hostname,
src_root=self.jobdir.src_root,
@@ -1170,7 +1161,6 @@
self.jobdir.root)
config.write('remote_tmp = %s/.ansible/remote_tmp\n' %
self.jobdir.root)
- config.write('private_key_file = %s\n' % self.private_key_file)
config.write('retry_files_enabled = False\n')
config.write('gathering = explicit\n')
config.write('library = %s\n'
@@ -1248,14 +1238,24 @@
if trusted:
config_file = self.jobdir.trusted_config
- popen = subprocess.Popen
+ opt_prefix = 'trusted'
else:
config_file = self.jobdir.untrusted_config
- driver = self.executor_server.untrusted_wrapper
- popen = driver.getPopen(
- work_dir=self.jobdir.root,
- ansible_dir=self.executor_server.ansible_dir,
- ssh_auth_sock=env_copy.get('SSH_AUTH_SOCK'))
+ opt_prefix = 'untrusted'
+ ro_dirs = get_default(self.executor_server.config, 'executor',
+ '%s_ro_dirs' % opt_prefix)
+ rw_dirs = get_default(self.executor_server.config, 'executor',
+ '%s_rw_dirs' % opt_prefix)
+ state_dir = get_default(self.executor_server.config, 'zuul',
+ 'state_dir', '/var/lib/zuul', expand_user=True)
+ ro_dirs = ro_dirs.split(":") if ro_dirs else []
+ rw_dirs = rw_dirs.split(":") if rw_dirs else []
+ self.executor_server.execution_wrapper.setMountsMap(state_dir, ro_dirs,
+ rw_dirs)
+
+ popen = self.executor_server.execution_wrapper.getPopen(
+ work_dir=self.jobdir.root,
+ ssh_auth_sock=env_copy.get('SSH_AUTH_SOCK'))
env_copy['ANSIBLE_CONFIG'] = config_file
@@ -1263,7 +1263,7 @@
if self.aborted:
return (self.RESULT_ABORTED, None)
self.log.debug("Ansible command: ANSIBLE_CONFIG=%s %s",
- config_file, " ".join(shlex_quote(c) for c in cmd))
+ config_file, " ".join(shlex.quote(c) for c in cmd))
self.proc = popen(
cmd,
cwd=self.jobdir.work_root,
@@ -1282,11 +1282,13 @@
for line in iter(self.proc.stdout.readline, b''):
line = line[:1024].rstrip()
self.log.debug("Ansible output: %s" % (line,))
+ self.log.debug("Ansible output terminated")
ret = self.proc.wait()
+ self.log.debug("Ansible exit code: %s" % (ret,))
finally:
if timeout:
watchdog.stop()
- self.log.debug("Ansible exit code: %s" % (ret,))
+ self.log.debug("Stopped watchdog")
with self.proc_lock:
self.proc = None
@@ -1317,5 +1319,8 @@
if success is not None:
cmd.extend(['-e', 'success=%s' % str(bool(success))])
- return self.runAnsible(
+ result, code = self.runAnsible(
cmd=cmd, timeout=timeout, trusted=playbook.trusted)
+ self.log.debug("Ansible complete, result %s code %s" % (
+ self.RESULT_MAP[result], code))
+ return result, code
diff --git a/zuul/lib/clonemapper.py b/zuul/lib/clonemapper.py
index 57ac177..7423308 100644
--- a/zuul/lib/clonemapper.py
+++ b/zuul/lib/clonemapper.py
@@ -14,17 +14,11 @@
# under the License.
from collections import defaultdict
-import extras
+from collections import OrderedDict
import logging
import os
import re
-import six
-
-
-OrderedDict = extras.try_imports(['collections.OrderedDict',
- 'ordereddict.OrderedDict'])
-
class CloneMapper(object):
log = logging.getLogger("zuul.CloneMapper")
@@ -62,17 +56,17 @@
raise Exception("Expansion error. Check error messages above")
self.log.info("Mapping projects to workspace...")
- for project, dest in six.iteritems(ret):
+ for project, dest in ret.items():
dest = os.path.normpath(os.path.join(workspace, dest[0]))
ret[project] = dest
self.log.info(" %s -> %s", project, dest)
self.log.debug("Checking overlap in destination directories...")
check = defaultdict(list)
- for project, dest in six.iteritems(ret):
+ for project, dest in ret.items():
check[dest].append(project)
- dupes = dict((d, p) for (d, p) in six.iteritems(check) if len(p) > 1)
+ dupes = dict((d, p) for (d, p) in check.items() if len(p) > 1)
if dupes:
raise Exception("Some projects share the same destination: %s",
dupes)
diff --git a/zuul/lib/cloner.py b/zuul/lib/cloner.py
index 3070be6..3fcffbe 100644
--- a/zuul/lib/cloner.py
+++ b/zuul/lib/cloner.py
@@ -18,8 +18,6 @@
import os
import re
-import six
-
from git import GitCommandError
from zuul import exceptions
from zuul.lib.clonemapper import CloneMapper
@@ -72,7 +70,7 @@
dests = mapper.expand(workspace=self.workspace)
self.log.info("Preparing %s repositories", len(dests))
- for project, dest in six.iteritems(dests):
+ for project, dest in dests.items():
self.prepareRepo(project, dest)
self.log.info("Prepared all repositories")
diff --git a/zuul/lib/commandsocket.py b/zuul/lib/commandsocket.py
index ae62204..901291a 100644
--- a/zuul/lib/commandsocket.py
+++ b/zuul/lib/commandsocket.py
@@ -18,7 +18,7 @@
import os
import socket
import threading
-from six.moves import queue
+import queue
class CommandSocket(object):
diff --git a/zuul/lib/config.py b/zuul/lib/config.py
new file mode 100644
index 0000000..9cdf66e
--- /dev/null
+++ b/zuul/lib/config.py
@@ -0,0 +1,23 @@
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+
+import os
+
+
+def get_default(config, section, option, default=None, expand_user=False):
+ if config.has_option(section, option):
+ value = config.get(section, option)
+ else:
+ value = default
+ if expand_user and value:
+ return os.path.expanduser(value)
+ return value
diff --git a/zuul/manager/__init__.py b/zuul/manager/__init__.py
index c3958d7..20bc459 100644
--- a/zuul/manager/__init__.py
+++ b/zuul/manager/__init__.py
@@ -176,7 +176,7 @@
return True
def enqueueChangesAhead(self, change, quiet, ignore_requirements,
- change_queue):
+ change_queue, history=None):
return True
def enqueueChangesBehind(self, change, quiet, ignore_requirements,
@@ -264,7 +264,7 @@
def addChange(self, change, quiet=False, enqueue_time=None,
ignore_requirements=False, live=True,
- change_queue=None):
+ change_queue=None, history=None):
self.log.debug("Considering adding change %s" % change)
# If we are adding a live change, check if it's a live item
@@ -299,7 +299,7 @@
return False
if not self.enqueueChangesAhead(change, quiet, ignore_requirements,
- change_queue):
+ change_queue, history=history):
self.log.debug("Failed to enqueue changes "
"ahead of %s" % change)
return False
@@ -738,7 +738,8 @@
# pipeline, use the dynamic layout if available, otherwise,
# fall back to the current static layout as a best
# approximation.
- layout = item.layout or self.pipeline.layout
+ layout = (item.current_build_set.layout or
+ self.pipeline.layout)
if not layout.hasProject(item.change.project):
self.log.debug("Project %s not in pipeline %s for change %s" % (
diff --git a/zuul/manager/dependent.py b/zuul/manager/dependent.py
index 6c56a30..ada3491 100644
--- a/zuul/manager/dependent.py
+++ b/zuul/manager/dependent.py
@@ -115,7 +115,15 @@
change_queue=change_queue)
def enqueueChangesAhead(self, change, quiet, ignore_requirements,
- change_queue):
+ change_queue, history=None):
+ if history and change.number in history:
+ # detected dependency cycle
+ self.log.warn("Dependency cycle detected")
+ return False
+ if hasattr(change, 'number'):
+ history = history or []
+ history.append(change.number)
+
ret = self.checkForChangesNeededBy(change, change_queue)
if ret in [True, False]:
return ret
@@ -124,7 +132,7 @@
for needed_change in ret:
r = self.addChange(needed_change, quiet=quiet,
ignore_requirements=ignore_requirements,
- change_queue=change_queue)
+ change_queue=change_queue, history=history)
if not r:
return False
return True
diff --git a/zuul/manager/independent.py b/zuul/manager/independent.py
index 9e2a7d6..06c9a01 100644
--- a/zuul/manager/independent.py
+++ b/zuul/manager/independent.py
@@ -36,7 +36,15 @@
return DynamicChangeQueueContextManager(change_queue)
def enqueueChangesAhead(self, change, quiet, ignore_requirements,
- change_queue):
+ change_queue, history=None):
+ if history and change.number in history:
+ # detected dependency cycle
+ self.log.warn("Dependency cycle detected")
+ return False
+ if hasattr(change, 'number'):
+ history = history or []
+ history.append(change.number)
+
ret = self.checkForChangesNeededBy(change, change_queue)
if ret in [True, False]:
return ret
@@ -50,7 +58,8 @@
# live).
r = self.addChange(needed_change, quiet=True,
ignore_requirements=True,
- live=False, change_queue=change_queue)
+ live=False, change_queue=change_queue,
+ history=history)
if not r:
return False
return True
diff --git a/zuul/merger/client.py b/zuul/merger/client.py
index c98f20e..e92d9fd 100644
--- a/zuul/merger/client.py
+++ b/zuul/merger/client.py
@@ -20,6 +20,7 @@
import gear
import zuul.model
+from zuul.lib.config import get_default
def getJobData(job):
@@ -75,13 +76,13 @@
self.config = config
self.sched = sched
server = self.config.get('gearman', 'server')
- if self.config.has_option('gearman', 'port'):
- port = self.config.get('gearman', 'port')
- else:
- port = 4730
+ port = get_default(self.config, 'gearman', 'port', 4730)
+ ssl_key = get_default(self.config, 'gearman', 'ssl_key')
+ ssl_cert = get_default(self.config, 'gearman', 'ssl_cert')
+ ssl_ca = get_default(self.config, 'gearman', 'ssl_ca')
self.log.debug("Connecting to gearman at %s:%s" % (server, port))
self.gearman = MergeGearmanClient(self)
- self.gearman.addServer(server, port)
+ self.gearman.addServer(server, port, ssl_key, ssl_cert, ssl_ca)
self.log.debug("Waiting for gearman")
self.gearman.waitForServer()
self.jobs = set()
diff --git a/zuul/merger/server.py b/zuul/merger/server.py
index 1a32f96..cbc4cb8 100644
--- a/zuul/merger/server.py
+++ b/zuul/merger/server.py
@@ -19,6 +19,7 @@
import gear
+from zuul.lib.config import get_default
from zuul.merger import merger
@@ -29,20 +30,10 @@
self.config = config
self.zuul_url = config.get('merger', 'zuul_url')
- if self.config.has_option('merger', 'git_dir'):
- merge_root = self.config.get('merger', 'git_dir')
- else:
- merge_root = '/var/lib/zuul/merger-git'
-
- if self.config.has_option('merger', 'git_user_email'):
- merge_email = self.config.get('merger', 'git_user_email')
- else:
- merge_email = None
-
- if self.config.has_option('merger', 'git_user_name'):
- merge_name = self.config.get('merger', 'git_user_name')
- else:
- merge_name = None
+ merge_root = get_default(self.config, 'merger', 'git_dir',
+ '/var/lib/zuul/merger-git')
+ merge_email = get_default(self.config, 'merger', 'git_user_email')
+ merge_name = get_default(self.config, 'merger', 'git_user_name')
self.merger = merger.Merger(merge_root, connections, merge_email,
merge_name)
@@ -50,12 +41,12 @@
def start(self):
self._running = True
server = self.config.get('gearman', 'server')
- if self.config.has_option('gearman', 'port'):
- port = self.config.get('gearman', 'port')
- else:
- port = 4730
+ port = get_default(self.config, 'gearman', 'port', 4730)
+ ssl_key = get_default(self.config, 'gearman', 'ssl_key')
+ ssl_cert = get_default(self.config, 'gearman', 'ssl_cert')
+ ssl_ca = get_default(self.config, 'gearman', 'ssl_ca')
self.worker = gear.TextWorker('Zuul Merger')
- self.worker.addServer(server, port)
+ self.worker.addServer(server, port, ssl_key, ssl_cert, ssl_ca)
self.log.debug("Waiting for server")
self.worker.waitForServer()
self.log.debug("Registering")
diff --git a/zuul/model.py b/zuul/model.py
index 5eedc75..a89c6d1 100644
--- a/zuul/model.py
+++ b/zuul/model.py
@@ -13,19 +13,13 @@
# under the License.
import abc
+from collections import OrderedDict
import copy
import logging
import os
import struct
import time
from uuid import uuid4
-import extras
-
-import six
-
-OrderedDict = extras.try_imports(['collections.OrderedDict',
- 'ordereddict.OrderedDict'])
-
MERGER_MERGE = 1 # "git merge"
MERGER_MERGE_RESOLVE = 2 # "git merge -s resolve"
@@ -154,7 +148,8 @@
return None
def removeQueue(self, queue):
- self.queues.remove(queue)
+ if queue in self.queues:
+ self.queues.remove(queue)
def getChangesInQueue(self):
changes = []
@@ -668,8 +663,7 @@
path=self.path)
-@six.add_metaclass(abc.ABCMeta)
-class Role(object):
+class Role(object, metaclass=abc.ABCMeta):
"""A reference to an ansible role."""
def __init__(self, target_name):
@@ -1339,7 +1333,6 @@
self.quiet = False
self.active = False # Whether an item is within an active window
self.live = True # Whether an item is intended to be processed at all
- self.layout = None # This item's shadow layout
self.job_graph = None
def __repr__(self):
diff --git a/zuul/reporter/__init__.py b/zuul/reporter/__init__.py
index dc99c8b..0ac5766 100644
--- a/zuul/reporter/__init__.py
+++ b/zuul/reporter/__init__.py
@@ -15,11 +15,8 @@
import abc
import logging
-import six
-
-@six.add_metaclass(abc.ABCMeta)
-class BaseReporter(object):
+class BaseReporter(object, metaclass=abc.ABCMeta):
"""Base class for reporters.
Defines the exact public methods that must be supplied.
@@ -142,11 +139,7 @@
elapsed = ' in %ds' % (s)
else:
elapsed = ''
- name = ''
- if config.has_option('zuul', 'job_name_in_report'):
- if config.getboolean('zuul',
- 'job_name_in_report'):
- name = job.name + ' '
+ name = job.name + ' '
ret += '- %s%s : %s%s%s\n' % (name, url, result, elapsed,
voting)
return ret
diff --git a/zuul/rpcclient.py b/zuul/rpcclient.py
index d980992..6f0d34b 100644
--- a/zuul/rpcclient.py
+++ b/zuul/rpcclient.py
@@ -26,10 +26,10 @@
class RPCClient(object):
log = logging.getLogger("zuul.RPCClient")
- def __init__(self, server, port):
+ def __init__(self, server, port, ssl_key=None, ssl_cert=None, ssl_ca=None):
self.log.debug("Connecting to gearman at %s:%s" % (server, port))
self.gearman = gear.Client()
- self.gearman.addServer(server, port)
+ self.gearman.addServer(server, port, ssl_key, ssl_cert, ssl_ca)
self.log.debug("Waiting for gearman")
self.gearman.waitForServer()
diff --git a/zuul/rpclistener.py b/zuul/rpclistener.py
index 6508e84..be3b7d1 100644
--- a/zuul/rpclistener.py
+++ b/zuul/rpclistener.py
@@ -19,9 +19,9 @@
import traceback
import gear
-import six
from zuul import model
+from zuul.lib.config import get_default
class RPCListener(object):
@@ -34,13 +34,15 @@
def start(self):
self._running = True
server = self.config.get('gearman', 'server')
- if self.config.has_option('gearman', 'port'):
- port = self.config.get('gearman', 'port')
- else:
- port = 4730
+ port = get_default(self.config, 'gearman', 'port', 4730)
+ ssl_key = get_default(self.config, 'gearman', 'ssl_key')
+ ssl_cert = get_default(self.config, 'gearman', 'ssl_cert')
+ ssl_ca = get_default(self.config, 'gearman', 'ssl_ca')
self.worker = gear.TextWorker('Zuul RPC Listener')
- self.worker.addServer(server, port)
+ self.worker.addServer(server, port, ssl_key, ssl_cert, ssl_ca)
+ self.log.debug("Waiting for server")
self.worker.waitForServer()
+ self.log.debug("Registering")
self.register()
self.thread = threading.Thread(target=self.run)
self.thread.daemon = True
@@ -165,8 +167,7 @@
# TODO: use args to filter by pipeline etc
running_items = []
for tenant in self.sched.abide.tenants.values():
- for pipeline_name, pipeline in six.iteritems(
- tenant.layout.pipelines):
+ for pipeline_name, pipeline in tenant.layout.pipelines.items():
for queue in pipeline.queues:
for item in queue.queue:
running_items.append(item.formatJSON())
diff --git a/zuul/scheduler.py b/zuul/scheduler.py
index a63d270..c762309 100644
--- a/zuul/scheduler.py
+++ b/zuul/scheduler.py
@@ -20,8 +20,7 @@
import logging
import os
import pickle
-import six
-from six.moves import queue as Queue
+import queue
import socket
import sys
import threading
@@ -31,6 +30,7 @@
from zuul import model
from zuul import exceptions
from zuul import version as zuul_version
+from zuul.lib.config import get_default
class ManagementEvent(object):
@@ -49,7 +49,9 @@
def wait(self, timeout=None):
self._wait_event.wait(timeout)
if self._exc_info:
- six.reraise(*self._exc_info)
+ # http://python3porting.com/differences.html#raise
+ e, v, t = self._exc_info
+ raise e(v).with_traceback(t)
return self._wait_event.is_set()
@@ -217,9 +219,9 @@
self.triggers = dict()
self.config = config
- self.trigger_event_queue = Queue.Queue()
- self.result_event_queue = Queue.Queue()
- self.management_event_queue = Queue.Queue()
+ self.trigger_event_queue = queue.Queue()
+ self.result_event_queue = queue.Queue()
+ self.management_event_queue = queue.Queue()
self.abide = model.Abide()
if not testonly:
@@ -370,30 +372,21 @@
self.log.debug("Waiting for exit")
def _get_queue_pickle_file(self):
- if self.config.has_option('zuul', 'state_dir'):
- state_dir = os.path.expanduser(self.config.get('zuul',
- 'state_dir'))
- else:
- state_dir = '/var/lib/zuul'
+ state_dir = get_default(self.config, 'zuul', 'state_dir',
+ '/var/lib/zuul', expand_user=True)
return os.path.join(state_dir, 'queue.pickle')
def _get_time_database_dir(self):
- if self.config.has_option('zuul', 'state_dir'):
- state_dir = os.path.expanduser(self.config.get('zuul',
- 'state_dir'))
- else:
- state_dir = '/var/lib/zuul'
+ state_dir = get_default(self.config, 'zuul', 'state_dir',
+ '/var/lib/zuul', expand_user=True)
d = os.path.join(state_dir, 'times')
if not os.path.exists(d):
os.mkdir(d)
return d
def _get_project_key_dir(self):
- if self.config.has_option('zuul', 'state_dir'):
- state_dir = os.path.expanduser(self.config.get('zuul',
- 'state_dir'))
- else:
- state_dir = '/var/lib/zuul'
+ state_dir = get_default(self.config, 'zuul', 'state_dir',
+ '/var/lib/zuul', expand_user=True)
key_dir = os.path.join(state_dir, 'keys')
if not os.path.exists(key_dir):
os.mkdir(key_dir, 0o700)
diff --git a/zuul/source/__init__.py b/zuul/source/__init__.py
index 68baf0e..b37aeb4 100644
--- a/zuul/source/__init__.py
+++ b/zuul/source/__init__.py
@@ -14,11 +14,8 @@
import abc
-import six
-
-@six.add_metaclass(abc.ABCMeta)
-class BaseSource(object):
+class BaseSource(object, metaclass=abc.ABCMeta):
"""Base class for sources.
A source class gives methods for fetching and updating changes. Each
diff --git a/zuul/trigger/__init__.py b/zuul/trigger/__init__.py
index a5406d6..a67c99b 100644
--- a/zuul/trigger/__init__.py
+++ b/zuul/trigger/__init__.py
@@ -14,11 +14,8 @@
import abc
-import six
-
-@six.add_metaclass(abc.ABCMeta)
-class BaseTrigger(object):
+class BaseTrigger(object, metaclass=abc.ABCMeta):
"""Base class for triggers.
Defines the exact public methods that must be supplied."""