Restructured documentation
Moved information out of intro.rst and into more specific doc files.
Reorganised index.rst accordingly, and deleted unused files.
Change-Id: I660c5e5e597baddbd1d2d5f60871687b5306834c
diff --git a/doc/source/configuration.rst b/doc/source/configuration.rst
deleted file mode 100644
index 5bba790..0000000
--- a/doc/source/configuration.rst
+++ /dev/null
@@ -1,10 +0,0 @@
-:title: Configuration
-
-
-Configuration
-=============
-
-It is required that debug logging be configured for Turbo Hispter, as we use
-the python logging framework to capture log messages from the task plugin
-code. Configuration is as simple as setting the debug_log configuration
-setting in config.json.
diff --git a/doc/source/index.rst b/doc/source/index.rst
index 25c0f30..c9ebbd0 100644
--- a/doc/source/index.rst
+++ b/doc/source/index.rst
@@ -12,9 +12,7 @@
:maxdepth: 2
intro
- structure
installation
- configuration
running
diff --git a/doc/source/installation.rst b/doc/source/installation.rst
index a0a0732..7c3de7f 100644
--- a/doc/source/installation.rst
+++ b/doc/source/installation.rst
@@ -6,142 +6,111 @@
Installation
============
-Download
---------
+Turbo-hipster is installed directly into your Python ``site-packages``
+directory, and is then run as a service. It is managed using a configuration
+file, which is in .json format.
-Grab a copy from https://github.com/rcbau/turbo-hipster
+Installing turbo-hipster
+------------------------
+1. Turbo-Hipster can be installed directly to your Python ``site-packages``
+directory::
-Install
--------
+ $ sudo python setup.py install
-turbo-hipster is configured to use setup tools for installation if
-you would like to install it to your site-packages use::
+2. Copy the configuration file to a convenient location. By default,
+turbo-hipster will look in ``/etc/turbo-hipster/config.json`` ::
- sudo python setup.py install
+ $ cp -R etc/turbo-hipster /etc/
+3. The Turbo-Hipster configuration file is in .json format. Open the
+``config.json`` configuration file in your preferred editor and modify it
+for your environment::
-Copy config
------------
+ **zuul_server**
+ A dictionary containing details about how to communicate
+ with zuul
+ **git_url**
+ The publicly accessible protocol and URI from where
+ to clone projects and zuul_ references from. For
+ example::
+ http://review.openstack.org/p/
+ or::
+ git://review.example.org
+ **gearman_host**
+ The host of gearman_. zuul talks to its workers via
+ the gearman protocol and while it comes with a built-
+ in gearman server you can use a separate one.
+ **gearman_port**
+ The port that gearman listens on.
+ **debug_log**
+ A path to the debug log. Turbo-hipster will attempt to create
+ the file but must have write permissions.
+ **jobs_working_dir**
+ Each job will likely need to write out log and debug
+ information. This defines where turbo-hipster will do that.
+ **git_working_dir**
+ turbo-hipster needs to take a copy of the git tree of a
+ project to work from. This is the path it'll clone into and
+ work from (if needed).
+ **pip_download_cache**
+ Some of turbo-hipsters task plugins download requirements
+ for projects. This is the cache directory used by pip.
+ **plugins**
+ A list of enabled plugins and their settings in a dictionary.
+ The only required parameters are *name*, which should be the
+ same as the folder containing the plugin module, and
+ *function*, which is the function registered with zuul.
+ Any other parameters are specified by the plugin themselves
+ as required.
+ **publish_logs**
+ Log results from plugins can be published using multiple
+ methods. Currently only a local copy is fully implemented.
+ **type**
+ The type of protocol to copy the log to. eg 'local'
+ **path**
+ A type specific parameter defining the local location
+ destination.
+ **prepend_url**
+ What to prepend to the path when sending the result
+ URL back to zuul. This can be useful as you may want
+ to use a script to authenticate against a swift
+ account or to use *laughing_spice* to format the logs
+ etc.
-Place the configuration where you are comfortable managing it. For
-example::
+4. Create a turbo-hipster user:
- cp -R etc/turbo-hipster /etc/
+ $ useradd turbo-hipster
+5. Create the directories listed in the configuration file, and give the
+``turbo-hipster`` user write access::
-Edit config
------------
+ $ mkdir -p /var/log/turbo-hipster/
+ $ chown turbo-hipster:turbo-hipster /var/log/turbo-hipster/
-Turbo-hipsters configuration is currently stored in json format.
-Modify the config.json appropriately::
+ $ mkdir -p /var/lib/turbo-hipster/jobs
+ $ chown turbo-hipster:turbo-hipster /var/lib/turbo-hipster/jobs
- **zuul_server**
- A dictionary containing details about how to communicate
- with zuul
- **git_url**
- The publicly accessible protocol and URI from where
- to clone projects and zuul_ references from. For
- example::
- http://review.openstack.org/p/
- or::
- git://review.example.org
- **gearman_host**
- The host of gearman_. zuul talks to its workers via
- the gearman protocol and while it comes with a built-
- in gearman server you can use a separate one.
- **gearman_port**
- The port that gearman listens on.
- **debug_log**
- A path to the debug log. Turbo-hipster will attempt to create
- the file but must have write permissions.
- **jobs_working_dir**
- Each job will likely need to write out log and debug
- information. This defines where turbo-hipster will do that.
- **git_working_dir**
- turbo-hipster needs to take a copy of the git tree of a
- project to work from. This is the path it'll clone into and
- work from (if needed).
- **pip_download_cache**
- Some of turbo-hipsters task plugins download requirements
- for projects. This is the cache directory used by pip.
- **plugins**
- A list of enabled plugins and their settings in a dictionary.
- The only required parameters are *name*, which should be the
- same as the folder containing the plugin module, and
- *function*, which is the function registered with zuul.
- Any other parameters are specified by the plugin themselves
- as required.
- **publish_logs**
- Log results from plugins can be published using multiple
- methods. Currently only a local copy is fully implemented.
- **type**
- The type of protocol to copy the log to. eg 'local'
- **path**
- A type specific parameter defining the local location
- destination.
- **prepend_url**
- What to prepend to the path when sending the result
- URL back to zuul. This can be useful as you may want
- to use a script to authenticate against a swift
- account or to use *laughing_spice* to format the logs
- etc.
+ $ mkdir -p /var/lib/turbo-hipster/git
+ $ chown turbo-hipster:turbo-hipster /var/lib/turbo-hipster/git
+ $ mkdir -p /var/cache/pip
+ $ chown turbo-hipster:turbo-hipster /var/cache/pip
-Set up
-------
+6. Open the MySQL log rotation configuration file in your preferred text
+editor, and edit it to ensure it is writable by ``other``::
-You probably want to create a user to run turbo-hipster as. This user
-will need to write to all of the directories specified in
-config.json.
+ $ vim /etc/logrotate.d/mysql-server
+ # edit create 640 to 644.
-Make sure the required directories as defined by the config.json
-exist and are writeable by your turbo-hipster user::
+.. note::
+ The turbo-hipster source code is also available for download from
+ the `turbo-hipster github page <https://github.com/rcbau/turbo-hipster/>`_
- mkdir -p /var/log/turbo-hipster/
- chown turbo-hipster:turbo-hipster /var/log/turbo-hipster/
+ $ git clone https://github.com/rcbau/turbo-hipster
- mkdir -p /var/lib/turbo-hipster/jobs
- chown turbo-hipster:turbo-hipster /var/lib/turbo-hipster/jobs
-
- mkdir -p /var/lib/turbo-hipster/git
- chown turbo-hipster:turbo-hipster /var/lib/turbo-hipster/git
-
- mkdir -p /var/cache/pip
- chown turbo-hipster:turbo-hipster /var/cache/pip
-
-Edit MySQL's log rotate to ensure it is other writable::
-
- vim /etc/logrotate.d/mysql-server
- # edit create 640 to 644.
-
-
-Start turbo-hipster
--------------------
-
-turbo hipster can be ran by executing::
-
- ./turbo-hipster/worker_server.py
-
-and optionally takes the following parameters::
-
- ./turbo_hipster/worker_server.py --help
- usage: worker_server.py [-h] [-c CONFIG] [-b] [-p PIDFILE]
-
- optional arguments:
- -h, --help show this help message and exit
- -c CONFIG, --config CONFIG
- Path to json config file.
- -b, --background Run as a daemon in the background.
- -p PIDFILE, --pidfile PIDFILE
- PID file to lock during daemonization.
-
-By default turbo-hipster will look for
-*/etc/turbo-hipster/config.json*
-
-Alternatively turbo-hipster can be launched by init.d using the
-included etc/init.d/turbo-hipster script::
-
- sudo cp etc/init.d/turbo-hipster /etc/init.d/
- sudo update-rc.d turbo-hipster defaults
- sudo service turbo-hipster start
+.. note::
+ Debug logging must be configured for turbo-hipster, as it uses the Python
+ logging framework to capture log messages from the task plugin code.
+ To configure debug logging, set the ``debug_log`` configuration
+ setting in the ``config.json`` configuration file.
\ No newline at end of file
diff --git a/doc/source/intro.rst b/doc/source/intro.rst
index 5ec60bc..d2c0045 100644
--- a/doc/source/intro.rst
+++ b/doc/source/intro.rst
@@ -92,260 +92,4 @@
}
-Installation
-============
-
-Turbo-hipster is installed directly into your Python ``site-packages``
-directory, and is then run as a service. It is managed using a configuration
-file, which is in .json format.
-
-Installing turbo-hipster
-------------------------
-
-1. Turbo-Hipster can be installed directly to your Python ``site-packages``
-directory::
-
- $ sudo python setup.py install
-
-2. Copy the configuration file to a convenient location. By default,
-turbo-hipster will look in ``/etc/turbo-hipster/config.json`` ::
-
- $ cp -R etc/turbo-hipster /etc/
-
-3. The Turbo-Hipster configuration file is in .json format. Open the
-``config.json`` configuration file in your preferred editor and modify it
-for your environment::
-
- **zuul_server**
- A dictionary containing details about how to communicate
- with zuul
- **git_url**
- The publicly accessible protocol and URI from where
- to clone projects and zuul_ references from. For
- example::
- http://review.openstack.org/p/
- or::
- git://review.example.org
- **gearman_host**
- The host of gearman_. zuul talks to its workers via
- the gearman protocol and while it comes with a built-
- in gearman server you can use a separate one.
- **gearman_port**
- The port that gearman listens on.
- **debug_log**
- A path to the debug log. Turbo-hipster will attempt to create
- the file but must have write permissions.
- **jobs_working_dir**
- Each job will likely need to write out log and debug
- information. This defines where turbo-hipster will do that.
- **git_working_dir**
- turbo-hipster needs to take a copy of the git tree of a
- project to work from. This is the path it'll clone into and
- work from (if needed).
- **pip_download_cache**
- Some of turbo-hipsters task plugins download requirements
- for projects. This is the cache directory used by pip.
- **plugins**
- A list of enabled plugins and their settings in a dictionary.
- The only required parameters are *name*, which should be the
- same as the folder containing the plugin module, and
- *function*, which is the function registered with zuul.
- Any other parameters are specified by the plugin themselves
- as required.
- **publish_logs**
- Log results from plugins can be published using multiple
- methods. Currently only a local copy is fully implemented.
- **type**
- The type of protocol to copy the log to. eg 'local'
- **path**
- A type specific parameter defining the local location
- destination.
- **prepend_url**
- What to prepend to the path when sending the result
- URL back to zuul. This can be useful as you may want
- to use a script to authenticate against a swift
- account or to use *laughing_spice* to format the logs
- etc.
-
-4. Create a turbo-hipster user:
-
- $ useradd turbo-hipster
-
-5. Create the directories listed in the configuration file, and give the
-``turbo-hipster`` user write access::
-
- $ mkdir -p /var/log/turbo-hipster/
- $ chown turbo-hipster:turbo-hipster /var/log/turbo-hipster/
-
- $ mkdir -p /var/lib/turbo-hipster/jobs
- $ chown turbo-hipster:turbo-hipster /var/lib/turbo-hipster/jobs
-
- $ mkdir -p /var/lib/turbo-hipster/git
- $ chown turbo-hipster:turbo-hipster /var/lib/turbo-hipster/git
-
- $ mkdir -p /var/cache/pip
- $ chown turbo-hipster:turbo-hipster /var/cache/pip
-
-6. Open the MySQL log rotation configuration file in your preferred text
-editor, and edit it to ensure it is writable by ``other``::
-
- $ vim /etc/logrotate.d/mysql-server
- # edit create 640 to 644.
-
-.. note::
- The turbo-hipster source code is also available for download from
- the `turbo-hipster github page <https://github.com/rcbau/turbo-hipster/>`_
-
- $ git clone https://github.com/rcbau/turbo-hipster
-
-Starting turbo-hipster
-----------------------
-
-Turbo-hipster can be run from the command line::
-
- $ ./turbo-hipster/worker_server.py
-
-This option allows you to pass parameters to turbo-hipster. Use the --help
-parameter to see a full list.
-
-+-------+--------------+--------------------------------------------------------+
-| Short | Long | Description |
-+=======+==============+========================================================+
-| -c | --config | Print the path to the configuration file and exit |
-+-------+--------------+--------------------------------------------------------+
-| -b | --background | Run as a daemon in the background |
-+-------+--------------+--------------------------------------------------------+
-| -p | --pidfile | Specify the PID file to lock while running as a daemon |
-+-------+--------------+--------------------------------------------------------+
-
-Alternatively, you can start turbo-hipster as a service.
-
-1. Copy the turbo-hipster init.d script to /etc/init.d/:
-
- $ sudo cp etc/init.d/turbo-hipster /etc/init.d/
-
-2. Reload the script with the default configuration:
-
- $ sudo update-rc.d turbo-hipster defaults
-
-3. Start the service:
-
- $ sudo service turbo-hipster start
-
-Plugins
-=======
-
-Plugins can be used to extend turbo-hipster's capabilities.
-
-.. note::
- Currently, the only available plugin for turbo-hipster is the
- Database Migration plugin, ``gate_real_db_upgrade``, which tests code
- against a variety of real-world databases.
-
-Installing plugins
-------------------
-
-Turbo-hipster plugins are responsible for handling the jobs that are passed
-to it. They must successfully build reports and publish them according to
-their configuration. They must also be able to communicate test results back
-to Zuul using Gearman.
-
-Plugins must take a standard format in order to be able to work correctly
-with turbo-hipster. They must contain a ``task.py`` file with a ``Runner``
-class.
-
-Once you have created a turbo-hipster plugin, you need to configure it in
-the ``config.json`` configuration file.
-
-.. FIXME More config information required here
-
-Plugin: Database migration with ``gate_real_db_upgrade``
---------------------------------------------------------
-
-The database migration plugin, ``gate_real_db_upgrade``, is used to test
-datasets against real-world, anonymized, databases.
-
-Migrating a database
---------------------
-
-In order to use turbo-hipster with the ``gate_real_db_upgrade`` plugin, you
-need to set up the databases to test against, and point to the plugin in
-turbo-hipster's configuration file.
-
-1. Create a directory for the datasets:
-
- $ mkdir -p /var/lib/turbo-hipster/datasets
-
-2. Copy the json dataset to the directory you created:
-
- $ cp /my/dataset.json /var/lib/turbo-hipster/datasets/
-
-3. Open the ``/etc/turbo-hipster/config.json`` file in your preferred
-editor, locate the plugins section, and add this line::
-
- **plugins**
- gate_real_db_upgrade
-
-Testing with turbo-hipster
-==========================
-
-When turbo-hipster completes a test, it sends the result of the test back to
-Gearman. These results contain a link to a compiled logfile for the test.
-
-If the test fails, or takes too long to complete, turbo-hipster will add a
-review to your patchset that looks like this:
-
-.. image:: ../images/THTestResult.png
-
-Reading test reports
---------------------
-
-An example of a standard log file:
-http://thw01.rcbops.com/results/54/54202/5/check/gate-real-db-upgrade_nova_mysql_devstack_150/ddd6d53/20130910_devstack_applied_to_150.log
-
-An example of the same logfile, using the javascript logviewer:
-http://thw01.rcbops.com/logviewer/?q=/results/54/54202/5/check/gate-real-db-upgrade_nova_mysql_devstack_150/ddd6d53/20130910_devstack_applied_to_150.log
-
-Test failure codes
-------------------
-
-This section gives a list of failure codes, including some steps you can
-take for troubleshooting errors:
-
- FAILURE - Did not find the end of a migration after a start
-
-If you look at the log you should find that a migration began but never
-finished. Hopefully there'll be a traceroute for you to follow through to
-get some hints about why it failed.
-
- WARNING - Migration %s took too long
-
-In this case your migration took a long time to run against one of the test
-datasets. You should reconsider what operations your migration is performing
-and see if there are any optimizations you can make, or if it is really
-necessary. If there is no way to speed up your migration you can email us at
-rcbau@rcbops.com for an exception.
-
- FAILURE - Final schema version does not match expectation
-
-Somewhere along the line the migrations stopped and did not reach the
-expected version. Our datasets start at previous releases and have to
-upgrade all the way through to the most current release. If you see this,
-inspect the log for traceroutes or other hints about the failure.
-
- FAILURE - Could not setup seed database.
- FAILURE - Could not find seed database.
-
-These errors are internal errors. If you see either of these, contact us at
-rcbau@rcbops.com to let us know so we can fix and rerun the tests for you.
-
- FAILURE - Could not import required module.
-
-This error probably shouldn't happen as Jenkins should catch it in the unit
-tests before Turbo-Hipster launches. If you see this, please contact us at
-rcbau@rcbops.com and let us know.
-
-If you receive an error that you think is a false positive, leave a comment
-on the review with the sole contents of "recheck migrations".
-If you have any questions/problems please contact us at rcbau@rcbops.com.
diff --git a/doc/source/running.rst b/doc/source/running.rst
index b0903db..9a60686 100644
--- a/doc/source/running.rst
+++ b/doc/source/running.rst
@@ -4,4 +4,154 @@
Running
=======
-todo...
\ No newline at end of file
+Starting turbo-hipster
+----------------------
+
+Turbo-hipster can be run from the command line::
+
+ $ ./turbo-hipster/worker_server.py
+
+This option allows you to pass parameters to turbo-hipster. Use the --help
+parameter to see a full list.
+
++-------+--------------+--------------------------------------------------------+
+| Short | Long | Description |
++=======+==============+========================================================+
+| -c | --config | Print the path to the configuration file and exit |
++-------+--------------+--------------------------------------------------------+
+| -b | --background | Run as a daemon in the background |
++-------+--------------+--------------------------------------------------------+
+| -p | --pidfile | Specify the PID file to lock while running as a daemon |
++-------+--------------+--------------------------------------------------------+
+
+Alternatively, you can start turbo-hipster as a service.
+
+1. Copy the turbo-hipster init.d script to /etc/init.d/:
+
+ $ sudo cp etc/init.d/turbo-hipster /etc/init.d/
+
+2. Reload the script with the default configuration:
+
+ $ sudo update-rc.d turbo-hipster defaults
+
+3. Start the service:
+
+ $ sudo service turbo-hipster start
+
+Plugins
+=======
+
+Plugins can be used to extend turbo-hipster's capabilities.
+
+.. note::
+ Currently, the only available plugin for turbo-hipster is the
+ Database Migration plugin, ``gate_real_db_upgrade``, which tests code
+ against a variety of real-world databases.
+
+Installing plugins
+------------------
+
+Turbo-hipster plugins are responsible for handling the jobs that are passed
+to it. They must successfully build reports and publish them according to
+their configuration. They must also be able to communicate test results back
+to Zuul using Gearman.
+
+Plugins must take a standard format in order to be able to work correctly
+with turbo-hipster. They must contain a ``task.py`` file with a ``Runner``
+class.
+
+Once you have created a turbo-hipster plugin, you need to configure it in
+the ``config.json`` configuration file.
+
+.. FIXME More config information required here
+
+Plugin: Database migration with ``gate_real_db_upgrade``
+--------------------------------------------------------
+
+The database migration plugin, ``gate_real_db_upgrade``, is used to test
+datasets against real-world, anonymized, databases.
+
+Migrating a database
+--------------------
+
+In order to use turbo-hipster with the ``gate_real_db_upgrade`` plugin, you
+need to set up the databases to test against, and point to the plugin in
+turbo-hipster's configuration file.
+
+1. Create a directory for the datasets:
+
+ $ mkdir -p /var/lib/turbo-hipster/datasets
+
+2. Copy the json dataset to the directory you created:
+
+ $ cp /my/dataset.json /var/lib/turbo-hipster/datasets/
+
+3. Open the ``/etc/turbo-hipster/config.json`` file in your preferred
+editor, locate the plugins section, and add this line::
+
+ **plugins**
+ gate_real_db_upgrade
+
+Testing with turbo-hipster
+==========================
+
+When turbo-hipster completes a test, it sends the result of the test back to
+Gearman. These results contain a link to a compiled logfile for the test.
+
+If the test fails, or takes too long to complete, turbo-hipster will add a
+review to your patchset that looks like this:
+
+.. image:: ../images/THTestResult.png
+
+Reading test reports
+--------------------
+
+An example of a standard log file:
+http://thw01.rcbops.com/results/54/54202/5/check/gate-real-db-upgrade_nova_mysql_devstack_150/ddd6d53/20130910_devstack_applied_to_150.log
+
+An example of the same logfile, using the javascript logviewer:
+http://thw01.rcbops.com/logviewer/?q=/results/54/54202/5/check/gate-real-db-upgrade_nova_mysql_devstack_150/ddd6d53/20130910_devstack_applied_to_150.log
+
+Test failure codes
+------------------
+
+This section gives a list of failure codes, including some steps you can
+take for troubleshooting errors:
+
+ FAILURE - Did not find the end of a migration after a start
+
+If you look at the log you should find that a migration began but never
+finished. Hopefully there'll be a traceroute for you to follow through to
+get some hints about why it failed.
+
+ WARNING - Migration %s took too long
+
+In this case your migration took a long time to run against one of the test
+datasets. You should reconsider what operations your migration is performing
+and see if there are any optimizations you can make, or if it is really
+necessary. If there is no way to speed up your migration you can email us at
+rcbau@rcbops.com for an exception.
+
+ FAILURE - Final schema version does not match expectation
+
+Somewhere along the line the migrations stopped and did not reach the
+expected version. Our datasets start at previous releases and have to
+upgrade all the way through to the most current release. If you see this,
+inspect the log for traceroutes or other hints about the failure.
+
+ FAILURE - Could not setup seed database.
+ FAILURE - Could not find seed database.
+
+These errors are internal errors. If you see either of these, contact us at
+rcbau@rcbops.com to let us know so we can fix and rerun the tests for you.
+
+ FAILURE - Could not import required module.
+
+This error probably shouldn't happen as Jenkins should catch it in the unit
+tests before Turbo-Hipster launches. If you see this, please contact us at
+rcbau@rcbops.com and let us know.
+
+If you receive an error that you think is a false positive, leave a comment
+on the review with the sole contents of "recheck migrations".
+
+If you have any questions/problems please contact us at rcbau@rcbops.com.
diff --git a/doc/source/structure.rst b/doc/source/structure.rst
deleted file mode 100644
index c203466..0000000
--- a/doc/source/structure.rst
+++ /dev/null
@@ -1,21 +0,0 @@
-:title:Turbo Hipster Structure
-
-Structure
-=======================
-
-Plugins
--------
-
-A little bit about plugins to come here soon.
-
- - You need to configure it in config.json
- - Folder name must be the same as 'name'
- - You probably want to specify a gate in the config
- - see something (introduction?) about gates and how turbo-hipster
- uses them
- - Each plugin has a task.py with a Runner class
- - Responsible for registering functions
- - handling jobs
- - checking for errors
- - building reports
- - publishing them according to configuration