Remove internal Swift support
The zuul v3 architecture does not require that this be supported
as an internal special case. Remove it for now in the expectation
that we will implement similar functionality using the auth system
and Ansible roles.
Change-Id: I9c1e05a716ce8cfb5f66c4688db110c320d33b4b
diff --git a/doc/source/launchers.rst b/doc/source/launchers.rst
index 8a8c932..c9dbd99 100644
--- a/doc/source/launchers.rst
+++ b/doc/source/launchers.rst
@@ -11,8 +11,6 @@
.. _`Turbo-Hipster Documentation`:
http://turbo-hipster.rtfd.org/
-.. _FormPost: http://docs.openstack.org/developer/swift/misc.html#module-swift.common.middleware.formpost
-
.. _launchers:
Launchers
@@ -138,34 +136,6 @@
Your jobs can check whether the parameters are ``000000`` to act
differently on each kind of event.
-Swift parameters
-~~~~~~~~~~~~~~~~
-
-If swift information has been configured for the job zuul will also
-provide signed credentials for the builder to upload results and
-assets into containers using the `FormPost`_ middleware.
-
-Each zuul container/instruction set will contain each of the following
-parameters where $NAME is the ``name`` defined in the layout.
-
-*SWIFT_$NAME_URL*
- The swift destination URL. This will be the entire URL including
- the AUTH, container and path prefix (folder).
-*SWIFT_$NAME_HMAC_BODY*
- The information signed in the HMAC body. The body is as follows::
-
- PATH TO OBJECT PREFIX (excluding domain)
- BLANK LINE (zuul implements no form redirect)
- MAX FILE SIZE
- MAX FILE COUNT
- SIGNATURE EXPIRY
-
-*SWIFT_$NAME_SIGNATURE*
- The HMAC body signed with the configured key.
-*SWIFT_$NAME_LOGSERVER_PREFIX*
- The URL to prepend to the object path when returning the results
- from a build.
-
Gearman
-------
diff --git a/doc/source/zuul.rst b/doc/source/zuul.rst
index 8906dac..4f43596 100644
--- a/doc/source/zuul.rst
+++ b/doc/source/zuul.rst
@@ -172,76 +172,6 @@
Path to PID lock file for the merger process.
``pidfile=/var/run/zuul-merger/merger.pid``
-.. _swift:
-
-swift
-"""""
-
-To send (optional) swift upload instructions this section must be
-present. Multiple destinations can be defined in the :ref:`jobs` section
-of the layout.
-
-If you are sending the temp-url-key or fetching the x-storage-url, you
-will need the python-swiftclient module installed.
-
-**X-Account-Meta-Temp-Url-Key** (optional)
- This is the key used to sign the HMAC message. If you do not set a
- key Zuul will generate one automatically.
-
-**Send-Temp-Url-Key** (optional)
- Zuul can send the X-Account-Meta-Temp-Url-Key to swift for you if
- you have set up the appropriate credentials in ``authurl`` below.
- This isn't necessary if you know and have set your
- X-Account-Meta-Temp-Url-Key.
- If set, Zuul requires the python-swiftclient module.
- ``default: true``
-
-**X-Storage-Url** (optional)
- The storage URL is the destination to upload files into. If you do
- not set this the ``authurl`` credentials are used to fetch the url
- from swift and Zuul will requires the python-swiftclient module.
-
-**authurl** (optional)
- The (keystone) Auth URL for swift.
- ``For example, https://identity.api.rackspacecloud.com/v2.0/``
- This is required if you have Send-Temp-Url-Key set to ``True`` or
- if you have not supplied the X-Storage-Url.
-
-Any of the `swiftclient connection parameters`_ can also be defined
-here by the same name. Including the os_options by their key name (
-``for example tenant_id``)
-
-.. _swiftclient connection parameters: http://docs.openstack.org/developer/python-swiftclient/swiftclient.html#module-swiftclient.client
-
-**region_name** (optional)
- The region name holding the swift container
- ``For example, SYD``
-
-Each destination defined by the :ref:`jobs` will have the following
-default values that it may overwrite.
-
-**default_container** (optional)
- Container name to place the log into
- ``For example, logs``
-
-**default_expiry** (optional)
- How long the signed destination should be available for
- ``default: 7200 (2hrs)``
-
-**default_max_file_size** (optional)
- The maximum size of an individual file
- ``default: 104857600 (100MB)``
-
-**default_max_file_count** (optional)
- The maximum number of separate files to allow
- ``default: 10``
-
-**default_logserver_prefix**
- Provide a URL to the CDN or logserver app so that a worker knows
- what URL to return. The worker should return the logserver_prefix
- url and the object path.
- ``For example: http://logs.example.org/server.app?obj=``
-
.. _connection:
connection ArbitraryName
@@ -786,48 +716,6 @@
**tags (optional)**
A list of arbitrary strings which will be associated with the job.
-**swift**
- If :ref:`swift` is configured then each job can define a destination
- container for the builder to place logs and/or assets into. Multiple
- containers can be listed for each job by providing a unique ``name``.
-
- *name*
- Set an identifying name for the container. This is used in the
- parameter key sent to the builder. For example if it ``logs`` then
- one of the parameters sent will be ``SWIFT_logs_CONTAINER``
- (case-sensitive).
-
- Each of the defaults defined in :ref:`swift` can be overwritten as:
-
- *container* (optional)
- Container name to place the log into
- ``For example, logs``
-
- *expiry* (optional)
- How long the signed destination should be available for
-
- *max-file-size** (optional)
- The maximum size of an individual file
-
- *max_file_size** (optional, deprecated)
- A deprecated alternate spelling of *max-file-size*.
-
- *max-file-count* (optional)
- The maximum number of separate files to allow
-
- *max_file_count* (optional, deprecated)
- A deprecated alternate spelling of *max-file-count*.
-
- *logserver-prefix*
- Provide a URL to the CDN or logserver app so that a worker knows
- what URL to return.
- ``For example: http://logs.example.org/server.app?obj=``
- The worker should return the logserver-prefix url and the object
- path as the URL in the results data packet.
-
- *logserver_prefix* (deprecated)
- A deprecated alternate spelling of *logserver-prefix*.
-
Here is an example of setting the failure message for jobs that check
whether a change merges cleanly::
diff --git a/test-requirements.txt b/test-requirements.txt
index e43b7a1..b99c803 100644
--- a/test-requirements.txt
+++ b/test-requirements.txt
@@ -6,7 +6,6 @@
fixtures>=0.3.14
python-keystoneclient>=0.4.2
python-subunit
-python-swiftclient>=1.6
testrepository>=0.0.17
testtools>=0.9.32
sphinxcontrib-programoutput
diff --git a/tests/base.py b/tests/base.py
index 0221a9e..f26863e 100755
--- a/tests/base.py
+++ b/tests/base.py
@@ -32,7 +32,6 @@
import socket
import string
import subprocess
-import swiftclient
import sys
import tempfile
import threading
@@ -60,7 +59,6 @@
import zuul.rpclistener
import zuul.launcher.server
import zuul.launcher.client
-import zuul.lib.swift
import zuul.lib.connections
import zuul.merger.client
import zuul.merger.merger
@@ -904,18 +902,6 @@
return True
-class FakeSwiftClientConnection(swiftclient.client.Connection):
- def post_account(self, headers):
- # Do nothing
- pass
-
- def get_auth(self):
- # Returns endpoint and (unused) auth token
- endpoint = os.path.join('https://storage.example.org', 'V1',
- 'AUTH_account')
- return endpoint, ''
-
-
class FakeNodepool(object):
REQUEST_ROOT = '/nodepool/requests'
NODE_ROOT = '/nodepool/nodes'
@@ -1332,11 +1318,6 @@
self.sched = zuul.scheduler.Scheduler(self.config)
- self.useFixture(fixtures.MonkeyPatch('swiftclient.client.Connection',
- FakeSwiftClientConnection))
-
- self.swift = zuul.lib.swift.Swift(self.config)
-
self.event_queues = [
self.sched.result_event_queue,
self.sched.trigger_event_queue,
@@ -1367,7 +1348,7 @@
self.builds = self.launch_server.running_builds
self.launch_client = zuul.launcher.client.LaunchClient(
- self.config, self.sched, self.swift)
+ self.config, self.sched)
self.merge_client = zuul.merger.client.MergeClient(
self.config, self.sched)
self.nodepool = zuul.nodepool.Nodepool(self.sched)
diff --git a/tests/fixtures/layout-swift.yaml b/tests/fixtures/layout-swift.yaml
deleted file mode 100644
index acaaad8..0000000
--- a/tests/fixtures/layout-swift.yaml
+++ /dev/null
@@ -1,59 +0,0 @@
-pipelines:
- - name: check
- manager: IndependentPipelineManager
- trigger:
- gerrit:
- - event: patchset-created
- success:
- gerrit:
- verified: 1
- failure:
- gerrit:
- verified: -1
-
- - name: post
- manager: IndependentPipelineManager
- trigger:
- gerrit:
- - event: ref-updated
- ref: ^(?!refs/).*$
-
- - name: gate
- manager: DependentPipelineManager
- failure-message: Build failed. For information on how to proceed, see http://wiki.example.org/Test_Failures
- trigger:
- gerrit:
- - event: comment-added
- approval:
- - approved: 1
- success:
- gerrit:
- verified: 2
- submit: true
- failure:
- gerrit:
- verified: -2
- start:
- gerrit:
- verified: 0
- precedence: high
-
-jobs:
- - name: ^.*$
- swift:
- - name: logs
- - name: ^.*-merge$
- swift:
- - name: logs
- container: merge_logs
- failure-message: Unable to merge change
- - name: test-test
- swift:
- - name: MOSTLY
- container: stash
-
-projects:
- - name: org/project
- gate:
- - test-merge
- - test-test
diff --git a/tests/fixtures/zuul-connections-bad-sql.conf b/tests/fixtures/zuul-connections-bad-sql.conf
index 150643d..2d1e804 100644
--- a/tests/fixtures/zuul-connections-bad-sql.conf
+++ b/tests/fixtures/zuul-connections-bad-sql.conf
@@ -12,16 +12,6 @@
git_user_name=zuul
zuul_url=http://zuul.example.com/p
-[swift]
-authurl=https://identity.api.example.org/v2.0/
-user=username
-key=password
-tenant_name=" "
-
-default_container=logs
-region_name=EXP
-logserver_prefix=http://logs.example.org/server.app/
-
[connection review_gerrit]
driver=gerrit
server=review.example.com
diff --git a/tests/fixtures/zuul-connections-multiple-gerrits.conf b/tests/fixtures/zuul-connections-multiple-gerrits.conf
index 3e6850d..c1a335d 100644
--- a/tests/fixtures/zuul-connections-multiple-gerrits.conf
+++ b/tests/fixtures/zuul-connections-multiple-gerrits.conf
@@ -15,16 +15,6 @@
[launcher]
git_dir=/tmp/zuul-test/launcher-git
-[swift]
-authurl=https://identity.api.example.org/v2.0/
-user=username
-key=password
-tenant_name=" "
-
-default_container=logs
-region_name=EXP
-logserver_prefix=http://logs.example.org/server.app/
-
[connection review_gerrit]
driver=gerrit
server=review.example.com
diff --git a/tests/fixtures/zuul-connections-same-gerrit.conf b/tests/fixtures/zuul-connections-same-gerrit.conf
index 30564de..5c10444 100644
--- a/tests/fixtures/zuul-connections-same-gerrit.conf
+++ b/tests/fixtures/zuul-connections-same-gerrit.conf
@@ -15,16 +15,6 @@
[launcher]
git_dir=/tmp/zuul-test/launcher-git
-[swift]
-authurl=https://identity.api.example.org/v2.0/
-user=username
-key=password
-tenant_name=" "
-
-default_container=logs
-region_name=EXP
-logserver_prefix=http://logs.example.org/server.app/
-
[connection review_gerrit]
driver=gerrit
server=review.example.com
diff --git a/tests/unit/test_scheduler.py b/tests/unit/test_scheduler.py
index eb8fccc..02e648e 100755
--- a/tests/unit/test_scheduler.py
+++ b/tests/unit/test_scheduler.py
@@ -3583,52 +3583,6 @@
self.assertNotIn('logs.example.com', B.messages[1])
self.assertNotIn('SKIPPED', B.messages[1])
- @skip("Disabled for early v3 development")
- def test_swift_instructions(self):
- "Test that the correct swift instructions are sent to the workers"
- self.updateConfigLayout(
- 'tests/fixtures/layout-swift.yaml')
- self.sched.reconfigure(self.config)
- self.registerJobs()
-
- self.launch_server.hold_jobs_in_build = True
- A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A')
-
- A.addApproval('code-review', 2)
- self.fake_gerrit.addEvent(A.addApproval('approved', 1))
- self.waitUntilSettled()
-
- self.assertEqual(
- "https://storage.example.org/V1/AUTH_account/merge_logs/1/1/1/"
- "gate/test-merge/",
- self.builds[0].parameters['SWIFT_logs_URL'][:-7])
- self.assertEqual(5,
- len(self.builds[0].parameters['SWIFT_logs_HMAC_BODY'].
- split('\n')))
- self.assertIn('SWIFT_logs_SIGNATURE', self.builds[0].parameters)
-
- self.assertEqual(
- "https://storage.example.org/V1/AUTH_account/logs/1/1/1/"
- "gate/test-test/",
- self.builds[1].parameters['SWIFT_logs_URL'][:-7])
- self.assertEqual(5,
- len(self.builds[1].parameters['SWIFT_logs_HMAC_BODY'].
- split('\n')))
- self.assertIn('SWIFT_logs_SIGNATURE', self.builds[1].parameters)
-
- self.assertEqual(
- "https://storage.example.org/V1/AUTH_account/stash/1/1/1/"
- "gate/test-test/",
- self.builds[1].parameters['SWIFT_MOSTLY_URL'][:-7])
- self.assertEqual(5,
- len(self.builds[1].
- parameters['SWIFT_MOSTLY_HMAC_BODY'].split('\n')))
- self.assertIn('SWIFT_MOSTLY_SIGNATURE', self.builds[1].parameters)
-
- self.launch_server.hold_jobs_in_build = False
- self.launch_server.release()
- self.waitUntilSettled()
-
def test_client_get_running_jobs(self):
"Test that the RPC client can get a list of running jobs"
self.launch_server.hold_jobs_in_build = True
diff --git a/zuul/cmd/scheduler.py b/zuul/cmd/scheduler.py
index 9a8b24f..8b8fb57 100755
--- a/zuul/cmd/scheduler.py
+++ b/zuul/cmd/scheduler.py
@@ -127,7 +127,6 @@
import zuul.launcher.client
import zuul.merger.client
import zuul.nodepool
- import zuul.lib.swift
import zuul.webapp
import zuul.rpclistener
import zuul.zk
@@ -141,11 +140,8 @@
self.log = logging.getLogger("zuul.Scheduler")
self.sched = zuul.scheduler.Scheduler(self.config)
- # TODO(jhesketh): Move swift into a connection?
- self.swift = zuul.lib.swift.Swift(self.config)
- gearman = zuul.launcher.client.LaunchClient(self.config, self.sched,
- self.swift)
+ gearman = zuul.launcher.client.LaunchClient(self.config, self.sched)
merger = zuul.merger.client.MergeClient(self.config, self.sched)
nodepool = zuul.nodepool.Nodepool(self.sched)
diff --git a/zuul/configloader.py b/zuul/configloader.py
index 3f72520..2c31341 100644
--- a/zuul/configloader.py
+++ b/zuul/configloader.py
@@ -122,20 +122,8 @@
class JobParser(object):
@staticmethod
def getSchema():
- swift_tmpurl = {vs.Required('name'): str,
- 'container': str,
- 'expiry': int,
- 'max_file_size': int,
- 'max-file-size': int,
- 'max_file_count': int,
- 'max-file-count': int,
- 'logserver_prefix': str,
- 'logserver-prefix': str,
- }
-
auth = {'secrets': to_list(str),
'inherit': bool,
- 'swift-tmpurl': to_list(swift_tmpurl),
}
node = {vs.Required('name'): str,
diff --git a/zuul/launcher/client.py b/zuul/launcher/client.py
index 6abd6f4..63c9c86 100644
--- a/zuul/launcher/client.py
+++ b/zuul/launcher/client.py
@@ -17,7 +17,6 @@
import json
import logging
import os
-import six
import time
import threading
from uuid import uuid4
@@ -149,10 +148,9 @@
log = logging.getLogger("zuul.LaunchClient")
negative_function_cache_ttl = 5
- def __init__(self, config, sched, swift):
+ def __init__(self, config, sched):
self.config = config
self.sched = sched
- self.swift = swift
self.builds = {}
self.meta_jobs = {} # A list of meta-jobs like stop or describe
@@ -211,42 +209,6 @@
self.log.debug("Function %s is not registered" % name)
return False
- def updateBuildParams(self, job, item, params):
- """Allow the job to modify and add build parameters"""
-
- # NOTE(jhesketh): The params need to stay in a key=value data pair
- # as workers cannot necessarily handle lists.
-
- if 'swift' in job.auth and self.swift.connection:
-
- for name, s in job.swift.items():
- swift_instructions = {}
- s_config = {}
- s_config.update((k, v.format(item=item, job=job,
- change=item.change))
- if isinstance(v, six.string_types)
- else (k, v)
- for k, v in s.items())
-
- (swift_instructions['URL'],
- swift_instructions['HMAC_BODY'],
- swift_instructions['SIGNATURE']) = \
- self.swift.generate_form_post_middleware_params(
- params['LOG_PATH'], **s_config)
-
- if 'logserver_prefix' in s_config:
- swift_instructions['LOGSERVER_PREFIX'] = \
- s_config['logserver_prefix']
- elif self.config.has_option('swift',
- 'default_logserver_prefix'):
- swift_instructions['LOGSERVER_PREFIX'] = \
- self.config.get('swift', 'default_logserver_prefix')
-
- # Create a set of zuul instructions for each instruction-set
- # given in the form of NAME_PARAMETER=VALUE
- for key, value in swift_instructions.items():
- params['_'.join(['SWIFT', name, key])] = value
-
def launch(self, job, item, pipeline, dependent_items=[]):
uuid = str(uuid4().hex)
self.log.info(
@@ -309,9 +271,6 @@
params['BASE_LOG_PATH'] = item.change.getBasePath()
params['LOG_PATH'] = destination_path
- # Allow the job to update the params
- self.updateBuildParams(job, item, params)
-
# This is what we should be heading toward for parameters:
# required:
diff --git a/zuul/lib/swift.py b/zuul/lib/swift.py
deleted file mode 100644
index 5660819..0000000
--- a/zuul/lib/swift.py
+++ /dev/null
@@ -1,168 +0,0 @@
-# Copyright 2014 Rackspace Australia
-#
-# Licensed under the Apache License, Version 2.0 (the "License"); you may
-# not use this file except in compliance with the License. You may obtain
-# a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
-# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
-# License for the specific language governing permissions and limitations
-# under the License.
-
-import hmac
-from hashlib import sha1
-import logging
-from time import time
-import os
-import random
-import six
-from six.moves import urllib
-import string
-
-
-class Swift(object):
- log = logging.getLogger("zuul.Swift")
-
- def __init__(self, config):
- self.config = config
- self.connection = False
- if self.config.has_option('swift', 'X-Account-Meta-Temp-Url-Key'):
- self.secure_key = self.config.get('swift',
- 'X-Account-Meta-Temp-Url-Key')
- else:
- self.secure_key = ''.join(
- random.choice(string.ascii_uppercase + string.digits)
- for x in range(20)
- )
-
- self.storage_url = ''
- if self.config.has_option('swift', 'X-Storage-Url'):
- self.storage_url = self.config.get('swift', 'X-Storage-Url')
-
- try:
- if self.config.has_section('swift'):
- if (not self.config.has_option('swift', 'Send-Temp-Url-Key')
- or self.config.getboolean('swift',
- 'Send-Temp-Url-Key')):
- self.connect()
-
- # Tell swift of our key
- headers = {}
- headers['X-Account-Meta-Temp-Url-Key'] = self.secure_key
- self.connection.post_account(headers)
-
- if not self.config.has_option('swift', 'X-Storage-Url'):
- self.connect()
- self.storage_url = self.connection.get_auth()[0]
- except Exception as e:
- self.log.warning("Unable to set up swift. Signed storage URL is "
- "likely to be wrong. %s" % e)
-
- def connect(self):
- if not self.connection:
- authurl = self.config.get('swift', 'authurl')
-
- user = (self.config.get('swift', 'user')
- if self.config.has_option('swift', 'user') else None)
- key = (self.config.get('swift', 'key')
- if self.config.has_option('swift', 'key') else None)
- retries = (self.config.get('swift', 'retries')
- if self.config.has_option('swift', 'retries') else 5)
- preauthurl = (self.config.get('swift', 'preauthurl')
- if self.config.has_option('swift', 'preauthurl')
- else None)
- preauthtoken = (self.config.get('swift', 'preauthtoken')
- if self.config.has_option('swift', 'preauthtoken')
- else None)
- snet = (self.config.get('swift', 'snet')
- if self.config.has_option('swift', 'snet') else False)
- starting_backoff = (self.config.get('swift', 'starting_backoff')
- if self.config.has_option('swift',
- 'starting_backoff')
- else 1)
- max_backoff = (self.config.get('swift', 'max_backoff')
- if self.config.has_option('swift', 'max_backoff')
- else 64)
- tenant_name = (self.config.get('swift', 'tenant_name')
- if self.config.has_option('swift', 'tenant_name')
- else None)
- auth_version = (self.config.get('swift', 'auth_version')
- if self.config.has_option('swift', 'auth_version')
- else 2.0)
- cacert = (self.config.get('swift', 'cacert')
- if self.config.has_option('swift', 'cacert') else None)
- insecure = (self.config.get('swift', 'insecure')
- if self.config.has_option('swift', 'insecure')
- else False)
- ssl_compression = (self.config.get('swift', 'ssl_compression')
- if self.config.has_option('swift',
- 'ssl_compression')
- else True)
-
- available_os_options = ['tenant_id', 'auth_token', 'service_type',
- 'endpoint_type', 'tenant_name',
- 'object_storage_url', 'region_name']
-
- os_options = {}
- for os_option in available_os_options:
- if self.config.has_option('swift', os_option):
- os_options[os_option] = self.config.get('swift', os_option)
-
- import swiftclient
- self.connection = swiftclient.client.Connection(
- authurl=authurl, user=user, key=key, retries=retries,
- preauthurl=preauthurl, preauthtoken=preauthtoken, snet=snet,
- starting_backoff=starting_backoff, max_backoff=max_backoff,
- tenant_name=tenant_name, os_options=os_options,
- auth_version=auth_version, cacert=cacert, insecure=insecure,
- ssl_compression=ssl_compression)
-
- def generate_form_post_middleware_params(self, destination_prefix='',
- **kwargs):
- """Generate the FormPost middleware params for the given settings"""
-
- # Define the available settings and their defaults
- settings = {
- 'container': '',
- 'expiry': 7200,
- 'max_file_size': 104857600,
- 'max_file_count': 10,
- 'file_path_prefix': ''
- }
-
- for key, default in six.iteritems(settings):
- # TODO(jeblair): Remove the following two lines after a
- # deprecation period for the underscore variants of the
- # settings in YAML.
- if key in kwargs:
- settings[key] = kwargs[key]
- # Since we prefer '-' rather than '_' in YAML, look up
- # keys there using hyphens. Continue to use underscores
- # everywhere else.
- altkey = key.replace('_', '-')
- if altkey in kwargs:
- settings[key] = kwargs[altkey]
- elif self.config.has_option('swift', 'default_' + key):
- settings[key] = self.config.get('swift', 'default_' + key)
- # TODO: these are always strings; some should be converted
- # to ints.
-
- expires = int(time() + int(settings['expiry']))
- redirect = ''
-
- url = os.path.join(self.storage_url, settings['container'],
- settings['file_path_prefix'],
- destination_prefix)
- u = urllib.parse.urlparse(url)
-
- hmac_body = '%s\n%s\n%s\n%s\n%s' % (u.path, redirect,
- settings['max_file_size'],
- settings['max_file_count'],
- expires)
-
- signature = hmac.new(self.secure_key, hmac_body, sha1).hexdigest()
-
- return url, hmac_body, signature