Merge "Build status page more jquery like"
diff --git a/doc/source/launchers.rst b/doc/source/launchers.rst
index f3e45db..accd80b 100644
--- a/doc/source/launchers.rst
+++ b/doc/source/launchers.rst
@@ -11,6 +11,8 @@
.. _`Turbo-Hipster Documentation`:
http://turbo-hipster.rtfd.org/
+.. _FormPost: http://docs.openstack.org/developer/swift/misc.html#module-swift.common.middleware.formpost
+
.. _launchers:
Launchers
@@ -117,6 +119,34 @@
Your jobs can check whether the parameters are ``000000`` to act
differently on each kind of event.
+Swift parameters
+~~~~~~~~~~~~~~~~
+
+If swift information has been configured for the job zuul will also
+provide signed credentials for the builder to upload results and
+assets into containers using the `FormPost`_ middleware.
+
+Each zuul container/instruction set will contain each of the following
+parameters where $NAME is the ``name`` defined in the layout.
+
+*SWIFT_$NAME_URL*
+ The swift destination URL. This will be the entire URL including
+ the AUTH, container and path prefix (folder).
+*SWIFT_$NAME_HMAC_BODY*
+ The information signed in the HMAC body. The body is as follows::
+
+ PATH TO OBJECT PREFIX (excluding domain)
+ BLANK LINE (zuul implements no form redirect)
+ MAX FILE SIZE
+ MAX FILE COUNT
+ SIGNATURE EXPIRY
+
+*SWIFT_$NAME_SIGNATURE*
+ The HMAC body signed with the configured key.
+*SWIFT_$NAME_LOGSERVER_PREFIX*
+ The URL to prepend to the object path when returning the results
+ from a build.
+
Gearman
-------
diff --git a/doc/source/zuul.rst b/doc/source/zuul.rst
index 7274342..408b0ac 100644
--- a/doc/source/zuul.rst
+++ b/doc/source/zuul.rst
@@ -67,6 +67,10 @@
FQDN of Gerrit server.
``server=review.example.com``
+**port**
+ Optional: Gerrit server port.
+ ``port=29418``
+
**baseurl**
Optional: path to Gerrit web interface. Defaults to ``https://<value
of server>/``. ``baseurl=https://review.example.com/review_site/``
@@ -158,6 +162,10 @@
SMTP server hostname or address to use.
``server=localhost``
+**port**
+ Optional: SMTP server port.
+ ``port=25``
+
**default_from**
Who the email should appear to be sent from when emailing the report.
This can be overridden by individual pipelines.
@@ -168,6 +176,59 @@
This can be overridden by individual pipelines.
``default_to=you@example.com``
+.. _swift:
+
+swift
+"""""
+
+To send (optional) swift upload instructions this section must be
+present. Multiple destinations can be defined in the :ref:`jobs`
+section of the layout.
+
+**authurl**
+ The (keystone) Auth URL for swift
+ ``For example, https://identity.api.rackspacecloud.com/v2.0/``
+
+Any of the `swiftclient connection parameters`_ can also be defined
+here by the same name. Including the os_options by their key name (
+``for example tenant_id``)
+
+.. _swiftclient connection parameters: http://docs.openstack.org/developer/python-swiftclient/swiftclient.html#module-swiftclient.client
+
+**X-Account-Meta-Temp-Url-Key** (optional)
+ This is the key used to sign the HMAC message. zuul will send the
+ key to swift for you so you only need to define it here. If you do
+ not set a key zuul will generate one automatically.
+
+**region_name** (optional)
+ The region name holding the swift container
+ ``For example, SYD``
+
+Each destination defined by the :ref:`jobs` will have the following
+default values that it may overwrite.
+
+**default_container** (optional)
+ Container name to place the log into
+ ``For example, logs``
+
+**default_expiry** (optional)
+ How long the signed destination should be available for
+ ``default: 7200 (2hrs)``
+
+**default_max_file_size** (optional)
+ The maximum size of an individual file
+ ``default: 104857600 (100MB)``
+
+**default_max_file_count** (optional)
+ The maximum number of separate files to allow
+ ``default: 10``
+
+**default_logserver_prefix**
+ Provide a URL to the CDN or logserver app so that a worker knows
+ what URL to return. The worker should return the logserver_prefix
+ url and the object path.
+ ``For example: http://logs.example.org/server.app?obj=``
+
layout.yaml
~~~~~~~~~~~
@@ -567,6 +628,9 @@
the ``ref-updated`` event which does include the commit sha1 (but lacks the
Gerrit change number).
+
+.. _jobs:
+
Jobs
""""
@@ -649,6 +713,39 @@
be used to specify on what node (or class of node) the job should be
run.
+**swift**
+ If :ref:`swift` is configured then each job can define a destination
+ container for the builder to place logs and/or assets into. Multiple
+ containers can be listed for each job by providing a unique ``name``.
+
+ *name*
+ Set an identifying name for the container. This is used in the
+ parameter key sent to the builder. For example if it ``logs`` then
+ one of the parameters sent will be ``SWIFT_logs_CONTAINER``
+ (case-sensitive).
+
+ Each of the defaults defined in :ref:`swift` can be overwritten as:
+
+ *container* (optional)
+ Container name to place the log into
+ ``For example, logs``
+
+ *expiry* (optional)
+ How long the signed destination should be available for
+
+ *max_file_size** (optional)
+ The maximum size of an individual file
+
+ *max_file_count* (optional)
+ The maximum number of separate files to allow
+
+ *logserver_prefix*
+ Provide a URL to the CDN or logserver app so that a worker knows
+ what URL to return.
+ ``For example: http://logs.example.org/server.app?obj=``
+ The worker should return the logserver_prefix url and the object
+ path as the URL in the results data packet.
+
Here is an example of setting the failure message for jobs that check
whether a change merges cleanly::
@@ -772,8 +869,8 @@
Note that if multiple templates are used for a project and one
template specifies a job that is also specified in another template,
-or specified in the project itself, those jobs will be duplicated in
-the resulting project configuration.
+or specified in the project itself, the configuration defined by
+either the last template or the project itself will take priority.
logging.conf
~~~~~~~~~~~~
diff --git a/etc/zuul.conf-sample b/etc/zuul.conf-sample
index 75c84e4..ac8021b 100644
--- a/etc/zuul.conf-sample
+++ b/etc/zuul.conf-sample
@@ -23,6 +23,15 @@
;git_user_name=zuul
zuul_url=http://zuul.example.com/p
+[swift]
+authurl=https://identity.api.example.org/v2.0/
+user=username
+key=password
+
+default_container=logs
+region_name=EXP
+logserver_prefix=http://logs.example.org/server.app/
+
[smtp]
server=localhost
port=25
diff --git a/requirements.txt b/requirements.txt
index f14441b..bb48290 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -14,3 +14,5 @@
voluptuous>=0.7
gear>=0.5.4,<1.0.0
apscheduler>=2.1.1,<3.0
+python-swiftclient>=1.6
+python-keystoneclient>=0.4.2
diff --git a/tests/fixtures/layout-swift.yaml b/tests/fixtures/layout-swift.yaml
new file mode 100644
index 0000000..acaaad8
--- /dev/null
+++ b/tests/fixtures/layout-swift.yaml
@@ -0,0 +1,59 @@
+pipelines:
+ - name: check
+ manager: IndependentPipelineManager
+ trigger:
+ gerrit:
+ - event: patchset-created
+ success:
+ gerrit:
+ verified: 1
+ failure:
+ gerrit:
+ verified: -1
+
+ - name: post
+ manager: IndependentPipelineManager
+ trigger:
+ gerrit:
+ - event: ref-updated
+ ref: ^(?!refs/).*$
+
+ - name: gate
+ manager: DependentPipelineManager
+ failure-message: Build failed. For information on how to proceed, see http://wiki.example.org/Test_Failures
+ trigger:
+ gerrit:
+ - event: comment-added
+ approval:
+ - approved: 1
+ success:
+ gerrit:
+ verified: 2
+ submit: true
+ failure:
+ gerrit:
+ verified: -2
+ start:
+ gerrit:
+ verified: 0
+ precedence: high
+
+jobs:
+ - name: ^.*$
+ swift:
+ - name: logs
+ - name: ^.*-merge$
+ swift:
+ - name: logs
+ container: merge_logs
+ failure-message: Unable to merge change
+ - name: test-test
+ swift:
+ - name: MOSTLY
+ container: stash
+
+projects:
+ - name: org/project
+ gate:
+ - test-merge
+ - test-test
diff --git a/tests/fixtures/layout.yaml b/tests/fixtures/layout.yaml
index b1c94de..b02c782 100644
--- a/tests/fixtures/layout.yaml
+++ b/tests/fixtures/layout.yaml
@@ -117,9 +117,6 @@
- name: test-five
check:
- '{name}-{something}-test5'
- - name: test-five-also
- check:
- - '{name}-{something}-test5'
projects:
- name: org/project
@@ -215,8 +212,6 @@
- name: test-three-and-four
- name: test-five
something: foo
- - name: test-five-also
- something: foo
check:
- project-test6
diff --git a/tests/fixtures/layouts/bad_swift.yaml b/tests/fixtures/layouts/bad_swift.yaml
new file mode 100644
index 0000000..d8a8c3f
--- /dev/null
+++ b/tests/fixtures/layouts/bad_swift.yaml
@@ -0,0 +1,29 @@
+pipelines:
+ - name: check
+ manager: IndependentPipelineManager
+ trigger:
+ gerrit:
+ - event: patchset-created
+ success:
+ gerrit:
+ verified: 1
+ failure:
+ gerrit:
+ verified: -1
+
+jobs:
+ - name: ^.*$
+ swift:
+ - name: logs
+ - name: ^.*-merge$
+ swift:
+ container: merge_assets
+ failure-message: Unable to merge change
+ - name: test-test
+ swift:
+
+projects:
+ - name: test-org/test
+ check:
+ - test-merge
+ - test-test
diff --git a/tests/fixtures/layouts/good_swift.yaml b/tests/fixtures/layouts/good_swift.yaml
new file mode 100644
index 0000000..913c268
--- /dev/null
+++ b/tests/fixtures/layouts/good_swift.yaml
@@ -0,0 +1,32 @@
+pipelines:
+ - name: check
+ manager: IndependentPipelineManager
+ trigger:
+ gerrit:
+ - event: patchset-created
+ success:
+ gerrit:
+ verified: 1
+ failure:
+ gerrit:
+ verified: -1
+
+jobs:
+ - name: ^.*$
+ swift:
+ - name: logs
+ - name: ^.*-merge$
+ swift:
+ - name: assets
+ container: merge_assets
+ failure-message: Unable to merge change
+ - name: test-test
+ swift:
+ - name: mostly
+ container: stash
+
+projects:
+ - name: test-org/test
+ check:
+ - test-merge
+ - test-test
diff --git a/tests/fixtures/zuul.conf b/tests/fixtures/zuul.conf
index bee06e4..ec76cd0 100644
--- a/tests/fixtures/zuul.conf
+++ b/tests/fixtures/zuul.conf
@@ -21,4 +21,14 @@
server=localhost
port=25
default_from=zuul@example.com
-default_to=you@example.com
\ No newline at end of file
+default_to=you@example.com
+
+[swift]
+authurl=https://identity.api.example.org/v2.0/
+user=username
+key=password
+tenant_name=" "
+
+default_container=logs
+region_name=EXP
+logserver_prefix=http://logs.example.org/server.app/
diff --git a/tests/test_scheduler.py b/tests/test_scheduler.py
index 351854d..7e1416f 100755
--- a/tests/test_scheduler.py
+++ b/tests/test_scheduler.py
@@ -30,6 +30,7 @@
import socket
import string
import subprocess
+import swiftclient
import threading
import time
import urllib
@@ -47,6 +48,7 @@
import zuul.rpclistener
import zuul.rpcclient
import zuul.launcher.gearman
+import zuul.lib.swift
import zuul.merger.server
import zuul.merger.client
import zuul.reporter.gerrit
@@ -743,6 +745,18 @@
return True
+class FakeSwiftClientConnection(swiftclient.client.Connection):
+ def post_account(self, headers):
+ # Do nothing
+ pass
+
+ def get_auth(self):
+ # Returns endpoint and (unused) auth token
+ endpoint = os.path.join('https://storage.example.org', 'V1',
+ 'AUTH_account')
+ return endpoint, ''
+
+
class TestScheduler(testtools.TestCase):
log = logging.getLogger("zuul.test")
@@ -823,12 +837,18 @@
self.sched = zuul.scheduler.Scheduler()
+ self.useFixture(fixtures.MonkeyPatch('swiftclient.client.Connection',
+ FakeSwiftClientConnection))
+ self.swift = zuul.lib.swift.Swift(self.config)
+
def URLOpenerFactory(*args, **kw):
args = [self.fake_gerrit] + list(args)
return FakeURLOpener(self.upstream_root, *args, **kw)
urllib2.urlopen = URLOpenerFactory
- self.launcher = zuul.launcher.gearman.Gearman(self.config, self.sched)
+
+ self.launcher = zuul.launcher.gearman.Gearman(self.config, self.sched,
+ self.swift)
self.merge_client = zuul.merger.client.MergeClient(
self.config, self.sched)
@@ -2061,13 +2081,8 @@
).result, 'SUCCESS')
self.assertEqual(self.getJobFromHistory('layered-project-test4'
).result, 'SUCCESS')
- # test5 should run twice because two templates define it
- test5_count = 0
- for job in self.worker.build_history:
- if job.name == 'layered-project-foo-test5':
- test5_count += 1
- self.assertEqual(job.result, 'SUCCESS')
- self.assertEqual(test5_count, 2)
+ self.assertEqual(self.getJobFromHistory('layered-project-foo-test5'
+ ).result, 'SUCCESS')
self.assertEqual(self.getJobFromHistory('project-test6').result,
'SUCCESS')
@@ -3810,3 +3825,48 @@
self.assertEqual(1, len(self.smtp_messages))
self.assertEqual('The merge failed! For more information...',
self.smtp_messages[0]['body'])
+
+ def test_swift_instructions(self):
+ "Test that the correct swift instructions are sent to the workers"
+ self.config.set('zuul', 'layout_config',
+ 'tests/fixtures/layout-swift.yaml')
+ self.sched.reconfigure(self.config)
+ self.registerJobs()
+
+ self.worker.hold_jobs_in_build = True
+ A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A')
+
+ A.addApproval('CRVW', 2)
+ self.fake_gerrit.addEvent(A.addApproval('APRV', 1))
+ self.waitUntilSettled()
+
+ self.assertEqual(
+ "https://storage.example.org/V1/AUTH_account/merge_logs/1/1/1/"
+ "gate/test-merge/",
+ self.builds[0].parameters['SWIFT_logs_URL'][:-32])
+ self.assertEqual(5,
+ len(self.builds[0].parameters['SWIFT_logs_HMAC_BODY'].
+ split('\n')))
+ self.assertIn('SWIFT_logs_SIGNATURE', self.builds[0].parameters)
+
+ self.assertEqual(
+ "https://storage.example.org/V1/AUTH_account/logs/1/1/1/"
+ "gate/test-test/",
+ self.builds[1].parameters['SWIFT_logs_URL'][:-32])
+ self.assertEqual(5,
+ len(self.builds[1].parameters['SWIFT_logs_HMAC_BODY'].
+ split('\n')))
+ self.assertIn('SWIFT_logs_SIGNATURE', self.builds[1].parameters)
+
+ self.assertEqual(
+ "https://storage.example.org/V1/AUTH_account/stash/1/1/1/"
+ "gate/test-test/",
+ self.builds[1].parameters['SWIFT_MOSTLY_URL'][:-32])
+ self.assertEqual(5,
+ len(self.builds[1].
+ parameters['SWIFT_MOSTLY_HMAC_BODY'].split('\n')))
+ self.assertIn('SWIFT_MOSTLY_SIGNATURE', self.builds[1].parameters)
+
+ self.worker.hold_jobs_in_build = False
+ self.worker.release()
+ self.waitUntilSettled()
diff --git a/zuul/cmd/server.py b/zuul/cmd/server.py
index 13e6283..8caa1fd 100755
--- a/zuul/cmd/server.py
+++ b/zuul/cmd/server.py
@@ -180,6 +180,7 @@
import zuul.scheduler
import zuul.launcher.gearman
import zuul.merger.client
+ import zuul.lib.swift
import zuul.reporter.gerrit
import zuul.reporter.smtp
import zuul.trigger.gerrit
@@ -195,8 +196,10 @@
self.log = logging.getLogger("zuul.Server")
self.sched = zuul.scheduler.Scheduler()
+ self.swift = zuul.lib.swift.Swift(self.config)
- gearman = zuul.launcher.gearman.Gearman(self.config, self.sched)
+ gearman = zuul.launcher.gearman.Gearman(self.config, self.sched,
+ self.swift)
merger = zuul.merger.client.MergeClient(self.config, self.sched)
gerrit = zuul.trigger.gerrit.Gerrit(self.config, self.sched)
timer = zuul.trigger.timer.Timer(self.config, self.sched)
diff --git a/zuul/launcher/gearman.py b/zuul/launcher/gearman.py
index 9ab1f61..b0d8546 100644
--- a/zuul/launcher/gearman.py
+++ b/zuul/launcher/gearman.py
@@ -16,6 +16,7 @@
import inspect
import json
import logging
+import os
import time
import threading
from uuid import uuid4
@@ -151,8 +152,10 @@
log = logging.getLogger("zuul.Gearman")
negative_function_cache_ttl = 5
- def __init__(self, config, sched):
+ def __init__(self, config, sched, swift):
+ self.config = config
self.sched = sched
+ self.swift = swift
self.builds = {}
self.meta_jobs = {} # A list of meta-jobs like stop or describe
@@ -215,6 +218,50 @@
self.log.debug("Function %s is not registered" % name)
return False
+ def updateBuildParams(self, job, item, params):
+ """Allow the job to modify and add build parameters"""
+
+ # NOTE(jhesketh): The params need to stay in a key=value data pair
+ # as workers cannot necessarily handle lists.
+
+ if job.swift and self.swift.connection:
+
+ for name, s in job.swift.items():
+ swift_instructions = {}
+ s_config = {}
+ s_config.update((k, v.format(item=item, job=job,
+ change=item.change))
+ for k, v in s.items())
+
+ (swift_instructions['URL'],
+ swift_instructions['HMAC_BODY'],
+ swift_instructions['SIGNATURE']) = \
+ self.swift.generate_form_post_middleware_params(
+ params['LOG_PATH'], **s_config)
+
+ if 'logserver_prefix' in s_config:
+ swift_instructions['LOGSERVER_PREFIX'] = \
+ s_config['logserver_prefix']
+ elif self.config.has_option('swift',
+ 'default_logserver_prefix'):
+ swift_instructions['LOGSERVER_PREFIX'] = \
+ s_config['logserver_prefix']
+
+ # Create a set of zuul instructions for each instruction-set
+ # given in the form of NAME_PARAMETER=VALUE
+ for key, value in swift_instructions.items():
+ params['_'.join(['SWIFT', name, key])] = value
+
+ if callable(job.parameter_function):
+ pargs = inspect.getargspec(job.parameter_function)
+ if len(pargs.args) == 2:
+ job.parameter_function(item, params)
+ else:
+ job.parameter_function(item, job, params)
+ self.log.debug("Custom parameter function used for job %s, "
+ "change: %s, params: %s" % (job, item.change,
+ params))
+
def launch(self, job, item, pipeline, dependent_items=[]):
self.log.info("Launch job %s for change %s with dependent changes %s" %
(job, item.change,
@@ -252,6 +299,16 @@
params['ZUUL_REF'] = item.change.ref
params['ZUUL_COMMIT'] = item.change.newrev
+ # The destination_path is a unqiue path for this build request
+ # and generally where the logs are expected to be placed
+ destination_path = os.path.join(item.change.getBasePath(),
+ pipeline.name, job.name, uuid)
+ params['BASE_LOG_PATH'] = item.change.getBasePath()
+ params['LOG_PATH'] = destination_path
+
+ # Allow the job to update the params
+ self.updateBuildParams(job, item, params)
+
# This is what we should be heading toward for parameters:
# required:
@@ -273,16 +330,6 @@
# ZUUL_OLDREV
# ZUUL_NEWREV
- if callable(job.parameter_function):
- pargs = inspect.getargspec(job.parameter_function)
- if len(pargs.args) == 2:
- job.parameter_function(item, params)
- else:
- job.parameter_function(item, job, params)
- self.log.debug("Custom parameter function used for job %s, "
- "change: %s, params: %s" % (job, item.change,
- params))
-
if 'ZUUL_NODE' in params:
name = "build:%s:%s" % (job.name, params['ZUUL_NODE'])
else:
diff --git a/zuul/layoutvalidator.py b/zuul/layoutvalidator.py
index 18b532d..15aa687 100644
--- a/zuul/layoutvalidator.py
+++ b/zuul/layoutvalidator.py
@@ -99,6 +99,14 @@
project_template = {v.Required('name'): str}
project_templates = [project_template]
+ swift = {v.Required('name'): str,
+ 'container': str,
+ 'expiry': int,
+ 'max_file_size': int,
+ 'max_file_count': int,
+ 'logserver_prefix': int,
+ }
+
job = {v.Required('name'): str,
'failure-message': str,
'success-message': str,
@@ -109,6 +117,7 @@
'parameter-function': str,
'branch': toList(str),
'files': toList(str),
+ 'swift': toList(swift),
}
jobs = [job]
diff --git a/zuul/lib/swift.py b/zuul/lib/swift.py
new file mode 100644
index 0000000..8f926ad
--- /dev/null
+++ b/zuul/lib/swift.py
@@ -0,0 +1,140 @@
+# Copyright 2014 Rackspace Australia
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+
+import hmac
+from hashlib import sha1
+from time import time
+import os
+import random
+import string
+import swiftclient
+import urlparse
+
+
+class Swift(object):
+ def __init__(self, config):
+ self.config = config
+ self.connection = False
+ if self.config.has_option('swift', 'X-Account-Meta-Temp-Url-Key'):
+ self.secure_key = self.config.get('swift',
+ 'X-Account-Meta-Temp-Url-Key')
+ else:
+ self.secure_key = ''.join(
+ random.choice(string.ascii_uppercase + string.digits)
+ for x in range(20)
+ )
+
+ self.connect()
+
+ def connect(self):
+ if self.config.has_section('swift'):
+ # required
+ authurl = self.config.get('swift', 'authurl')
+
+ user = (self.config.get('swift', 'user')
+ if self.config.has_option('swift', 'user') else None)
+ key = (self.config.get('swift', 'key')
+ if self.config.has_option('swift', 'key') else None)
+ retries = (self.config.get('swift', 'retries')
+ if self.config.has_option('swift', 'retries') else 5)
+ preauthurl = (self.config.get('swift', 'preauthurl')
+ if self.config.has_option('swift', 'preauthurl')
+ else None)
+ preauthtoken = (self.config.get('swift', 'preauthtoken')
+ if self.config.has_option('swift', 'preauthtoken')
+ else None)
+ snet = (self.config.get('swift', 'snet')
+ if self.config.has_option('swift', 'snet') else False)
+ starting_backoff = (self.config.get('swift', 'starting_backoff')
+ if self.config.has_option('swift',
+ 'starting_backoff')
+ else 1)
+ max_backoff = (self.config.get('swift', 'max_backoff')
+ if self.config.has_option('swift', 'max_backoff')
+ else 64)
+ tenant_name = (self.config.get('swift', 'tenant_name')
+ if self.config.has_option('swift', 'tenant_name')
+ else None)
+ auth_version = (self.config.get('swift', 'auth_version')
+ if self.config.has_option('swift', 'auth_version')
+ else 2.0)
+ cacert = (self.config.get('swift', 'cacert')
+ if self.config.has_option('swift', 'cacert') else None)
+ insecure = (self.config.get('swift', 'insecure')
+ if self.config.has_option('swift', 'insecure')
+ else False)
+ ssl_compression = (self.config.get('swift', 'ssl_compression')
+ if self.config.has_option('swift',
+ 'ssl_compression')
+ else True)
+
+ available_os_options = ['tenant_id', 'auth_token', 'service_type',
+ 'endpoint_type', 'tenant_name',
+ 'object_storage_url', 'region_name']
+
+ os_options = {}
+ for os_option in available_os_options:
+ if self.config.has_option('swift', os_option):
+ os_options[os_option] = self.config.get('swift', os_option)
+
+ self.connection = swiftclient.client.Connection(
+ authurl=authurl, user=user, key=key, retries=retries,
+ preauthurl=preauthurl, preauthtoken=preauthtoken, snet=snet,
+ starting_backoff=starting_backoff, max_backoff=max_backoff,
+ tenant_name=tenant_name, os_options=os_options,
+ auth_version=auth_version, cacert=cacert, insecure=insecure,
+ ssl_compression=ssl_compression)
+
+ # Tell swift of our key
+ headers = {}
+ headers['X-Account-Meta-Temp-Url-Key'] = self.secure_key
+ self.connection.post_account(headers)
+
+ self.storage_url, self.auth_token = self.connection.get_auth()
+
+ def generate_form_post_middleware_params(self, destination_prefix='',
+ **kwargs):
+ """Generate the FormPost middleware params for the given settings"""
+
+ # Define the available settings and their defaults
+ settings = {
+ 'container': '',
+ 'expiry': 7200,
+ 'max_file_size': 104857600,
+ 'max_file_count': 10,
+ 'file_path_prefix': ''
+ }
+
+ for key, default in settings.iteritems():
+ if key in kwargs:
+ settings[key] = kwargs[key]
+ elif self.config.has_option('swift', 'default_' + key):
+ settings[key] = self.config.get('swift', 'default_' + key)
+
+ expires = int(time() + settings['expiry'])
+ redirect = ''
+
+ url = os.path.join(self.storage_url, settings['container'],
+ settings['file_path_prefix'],
+ destination_prefix)
+ u = urlparse.urlparse(url)
+
+ hmac_body = '%s\n%s\n%s\n%s\n%s' % (u.path, redirect,
+ settings['max_file_size'],
+ settings['max_file_count'],
+ expires)
+
+ signature = hmac.new(self.secure_key, hmac_body, sha1).hexdigest()
+
+ return url, hmac_body, signature
diff --git a/zuul/model.py b/zuul/model.py
index 1a7c421..9028577 100644
--- a/zuul/model.py
+++ b/zuul/model.py
@@ -537,6 +537,7 @@
self._branches = []
self.files = []
self._files = []
+ self.swift = {}
def __str__(self):
return self.name
@@ -561,6 +562,8 @@
if other.files:
self.files = other.files[:]
self._files = other._files[:]
+ if other.swift:
+ self.swift.update(other.swift)
self.hold_following_changes = other.hold_following_changes
self.voting = other.voting
@@ -596,9 +599,10 @@
self.job_trees = []
def addJob(self, job):
- t = JobTree(job)
- self.job_trees.append(t)
- return t
+ if job not in [x.job for x in self.job_trees]:
+ t = JobTree(job)
+ self.job_trees.append(t)
+ return t
def getJobs(self):
jobs = []
@@ -763,6 +767,16 @@
def __init__(self, project):
self.project = project
+ def getBasePath(self):
+ base_path = ''
+ if hasattr(self, 'refspec'):
+ base_path = "%s/%s/%s" % (
+ self.number[-2:], self.number, self.patchset)
+ elif hasattr(self, 'ref'):
+ base_path = "%s/%s" % (self.newrev[:2], self.newrev)
+
+ return base_path
+
def equals(self, other):
raise NotImplementedError()
diff --git a/zuul/scheduler.py b/zuul/scheduler.py
index 1a4474c..18f44db 100644
--- a/zuul/scheduler.py
+++ b/zuul/scheduler.py
@@ -347,6 +347,10 @@
if files:
job._files = files
job.files = [re.compile(x) for x in files]
+ swift = toList(config_job.get('swift'))
+ if swift:
+ for s in swift:
+ job.swift[s['name']] = s
def add_jobs(job_tree, config_jobs):
for job in config_jobs:
@@ -384,11 +388,10 @@
config_project.update(
{pipeline.name: expanded[pipeline.name] +
config_project.get(pipeline.name, [])})
- # TODO: future enhancement -- add an option to the
- # template block to indicate that duplicate jobs should be
- # merged (especially to handle the case where they have
- # children and you want all of the children to run after a
- # single run of the parent).
+ # TODO: future enhancement -- handle the case where
+ # duplicate jobs have different children and you want all
+ # of the children to run after a single run of the
+ # parent).
layout.projects[config_project['name']] = project
mode = config_project.get('merge-mode', 'merge-resolve')