Gitiles
Code Review Sign In
gerrit.cesnet.cz / github / openstack-infra / zuul / faf8198f2a360e9a780c3006c660d1178ad86d8a / . / doc / source / admin / monitoring.rst
blob: dc6be0de3be6359ea44885cab3723dc50853cded [file] [log] [blame]
James E. Blaireff5a9d2017-06-20 00:00:37 -0700[diff] [blame]1:title: Monitoring
2
3Monitoring
4==========
Antoine Mussoa8eea7d2013-10-05 16:08:00 +0200[diff] [blame]5
James E. Blairded241e2017-10-10 13:22:40 -0700[diff] [blame]6.. _statsd:
7
Antoine Mussoa8eea7d2013-10-05 16:08:00 +0200[diff] [blame]8Statsd reporting
James E. Blaireff5a9d2017-06-20 00:00:37 -0700[diff] [blame]9----------------
Antoine Mussoa8eea7d2013-10-05 16:08:00 +0200[diff] [blame]10
11Zuul comes with support for the statsd protocol, when enabled and configured
Michael Prokop526926a2013-10-24 16:16:57 +0200[diff] [blame]12(see below), the Zuul scheduler will emit raw metrics to a statsd receiver
James E. Blaireff5a9d2017-06-20 00:00:37 -0700[diff] [blame]13which let you in turn generate nice graphics.
Antoine Mussoa8eea7d2013-10-05 16:08:00 +0200[diff] [blame]14
15Configuration
James E. Blaireff5a9d2017-06-20 00:00:37 -0700[diff] [blame]16~~~~~~~~~~~~~
Antoine Mussoa8eea7d2013-10-05 16:08:00 +0200[diff] [blame]17
James E. Blairded241e2017-10-10 13:22:40 -0700[diff] [blame]18Statsd support uses the ``statsd`` python module. Note that support
19is optional and Zuul will start without the statsd python module
20present.
Antoine Mussoa8eea7d2013-10-05 16:08:00 +0200[diff] [blame]21
James E. Blairded241e2017-10-10 13:22:40 -0700[diff] [blame]22Configuration is in the :attr:`statsd` section of ``zuul.conf``.
Antoine Mussoa8eea7d2013-10-05 16:08:00 +0200[diff] [blame]23
24Metrics
James E. Blaireff5a9d2017-06-20 00:00:37 -0700[diff] [blame]25~~~~~~~
Antoine Mussoa8eea7d2013-10-05 16:08:00 +0200[diff] [blame]26
David Shrewsbury1c61c712017-08-16 16:02:33 -0400[diff] [blame]27These metrics are emitted by the Zuul :ref:`scheduler`:
Antoine Mussoa8eea7d2013-10-05 16:08:00 +0200[diff] [blame]28
James E. Blair80ac1582017-10-09 07:02:40 -0700[diff] [blame]29.. stat:: zuul.event.<driver>.event.<type>
James E. Blair91c9dde2017-08-04 11:10:24 -0700[diff] [blame]30 :type: counter
James E. Blaireff5a9d2017-06-20 00:00:37 -0700[diff] [blame]31
James E. Blair80ac1582017-10-09 07:02:40 -0700[diff] [blame]32 Zuul will report counters for each type of event it receives from
33 each of its configured drivers.
Antoine Mussoa8eea7d2013-10-05 16:08:00 +0200[diff] [blame]34
James E. Blairfaf81982017-10-10 15:42:26 -0700[diff] [blame^]35.. stat:: zuul.tenant.<tenant>.pipeline
Antoine Mussoa8eea7d2013-10-05 16:08:00 +0200[diff] [blame]36
James E. Blair91c9dde2017-08-04 11:10:24 -0700[diff] [blame]37 Holds metrics specific to jobs. This hierarchy includes:
Antoine Mussoa8eea7d2013-10-05 16:08:00 +0200[diff] [blame]38
James E. Blair91c9dde2017-08-04 11:10:24 -0700[diff] [blame]39 .. stat:: <pipeline name>
Antoine Mussoa8eea7d2013-10-05 16:08:00 +0200[diff] [blame]40
James E. Blair91c9dde2017-08-04 11:10:24 -0700[diff] [blame]41 A set of metrics for each pipeline named as defined in the Zuul
42 config.
Antoine Mussoa8eea7d2013-10-05 16:08:00 +0200[diff] [blame]43
James E. Blair91c9dde2017-08-04 11:10:24 -0700[diff] [blame]44 .. stat:: all_jobs
45 :type: counter
Antoine Mussoa8eea7d2013-10-05 16:08:00 +0200[diff] [blame]46
James E. Blair91c9dde2017-08-04 11:10:24 -0700[diff] [blame]47 Number of jobs triggered by the pipeline.
Antoine Mussoa8eea7d2013-10-05 16:08:00 +0200[diff] [blame]48
James E. Blair91c9dde2017-08-04 11:10:24 -0700[diff] [blame]49 .. stat:: current_changes
50 :type: gauge
Antoine Mussoa8eea7d2013-10-05 16:08:00 +0200[diff] [blame]51
James E. Blair91c9dde2017-08-04 11:10:24 -0700[diff] [blame]52 The number of items currently being processed by this
53 pipeline.
Antoine Mussoa8eea7d2013-10-05 16:08:00 +0200[diff] [blame]54
James E. Blair80ac1582017-10-09 07:02:40 -0700[diff] [blame]55 .. stat:: project
James E. Blair91c9dde2017-08-04 11:10:24 -0700[diff] [blame]56
James E. Blair80ac1582017-10-09 07:02:40 -0700[diff] [blame]57 This hierarchy holds more specific metrics for each project
58 participating in the pipeline.
James E. Blair91c9dde2017-08-04 11:10:24 -0700[diff] [blame]59
James E. Blair80ac1582017-10-09 07:02:40 -0700[diff] [blame]60 .. stat:: <canonical_hostname>
James E. Blair91c9dde2017-08-04 11:10:24 -0700[diff] [blame]61
James E. Blair80ac1582017-10-09 07:02:40 -0700[diff] [blame]62 The canonical hostname for the triggering project.
63 Embedded ``.`` characters will be translated to ``_``.
James E. Blair91c9dde2017-08-04 11:10:24 -0700[diff] [blame]64
James E. Blair80ac1582017-10-09 07:02:40 -0700[diff] [blame]65 .. stat:: <project>
James E. Blair91c9dde2017-08-04 11:10:24 -0700[diff] [blame]66
James E. Blair80ac1582017-10-09 07:02:40 -0700[diff] [blame]67 The name of the triggering project. Embedded ``/`` or
68 ``.`` characters will be translated to ``_``.
69
70 .. stat:: <branch>
71
72 The name of the triggering branch. Embedded ``/`` or
73 ``.`` characters will be translated to ``_``.
74
75 .. stat:: job
76
77 Subtree detailing per-project job statistics:
78
79 .. stat:: <jobname>
80
81 The triggered job name.
82
83 .. stat:: <result>
84 :type: counter, timer
85
86 A counter for each type of result (e.g., ``SUCCESS`` or
87 ``FAILURE``, ``ERROR``, etc.) for the job. If the
88 result is ``SUCCESS`` or ``FAILURE``, Zuul will
89 additionally report the duration of the build as a
90 timer.
91
92 .. stat:: current_changes
93 :type: gauge
94
95 The number of items of this project currently being
96 processed by this pipeline.
97
98 .. stat:: resident_time
99 :type: timer
100
101 A timer metric reporting how long each item for this
102 project has been in the pipeline.
103
104 .. stat:: total_changes
105 :type: counter
106
107 The number of changes for this project processed by the
108 pipeline since Zuul started.
James E. Blair91c9dde2017-08-04 11:10:24 -0700[diff] [blame]109
110 .. stat:: resident_time
111 :type: timer
112
113 A timer metric reporting how long each item has been in the
114 pipeline.
115
116 .. stat:: total_changes
117 :type: counter
118
David Shrewsbury1c61c712017-08-16 16:02:33 -0400[diff] [blame]119 The number of changes processed by the pipeline since Zuul
James E. Blair91c9dde2017-08-04 11:10:24 -0700[diff] [blame]120 started.
121
122 .. stat:: wait_time
123 :type: timer
124
125 How long each item spent in the pipeline before its first job
126 started.
127
James E. Blairfaf81982017-10-10 15:42:26 -0700[diff] [blame^]128.. stat:: zuul.executor.<executor>
129
130 Holds metrics emitted by individual executors. The ``<executor>``
131 component of the key will be replaced with the hostname of the
132 executor.
133
134 .. stat:: builds
135 :type: counter
136
137 Incremented each time the executor starts a build.
138
139 .. stat:: running_builds
140 :type: gauge
141
142 The number of builds currently running on this executor.
143
144 .. stat:: load_average
145 :type: gauge
146
147 The one-minute load average of this executor, multiplied by 100.
148
James E. Blair91c9dde2017-08-04 11:10:24 -0700[diff] [blame]149
James E. Blair80ac1582017-10-09 07:02:40 -0700[diff] [blame]150As an example, given a job named `myjob` in `mytenant` triggered by a
151change to `myproject` on the `master` branch in the `gate` pipeline
152which took 40 seconds to build, the Zuul scheduler will emit the
153following statsd events:
James E. Blair91c9dde2017-08-04 11:10:24 -0700[diff] [blame]154
James E. Blair80ac1582017-10-09 07:02:40 -0700[diff] [blame]155 * ``zuul.tenant.mytenant.pipeline.gate.project.example_com.myproject.master.job.myjob.SUCCESS`` +1
156 * ``zuul.tenant.mytenant.pipeline.gate.project.example_com.myproject.master.job.myjob.SUCCESS`` 40 seconds
157 * ``zuul.tenant.mytenant.pipeline.gate.all_jobs`` +1