blob: 3398892613322d8500933836dc74ce685d029bf8 [file] [log] [blame]
James E. Blaircdd00072012-06-08 19:17:28 -07001:title: Project Gating
2
James E. Blaireff5a9d2017-06-20 00:00:37 -07003.. _project_gating:
4
James E. Blaircdd00072012-06-08 19:17:28 -07005Project Gating
6==============
7
8Traditionally, many software development projects merge changes from
9developers into the repository, and then identify regressions
10resulting from those changes (perhaps by running a test suite with a
James E. Blaireff5a9d2017-06-20 00:00:37 -070011continuous integration system), followed by more patches to fix those
12bugs. When the mainline of development is broken, it can be very
13frustrating for developers and can cause lost productivity,
14particularly so when the number of contributors or contributions is
15large.
James E. Blaircdd00072012-06-08 19:17:28 -070016
17The process of gating attempts to prevent changes that introduce
18regressions from being merged. This keeps the mainline of development
19open and working for all developers, and only when a change is
20confirmed to work without disruption is it merged.
21
22Many projects practice an informal method of gating where developers
23with mainline commit access ensure that a test suite runs before
24merging a change. With more developers, more changes, and more
25comprehensive test suites, that process does not scale very well, and
26is not the best use of a developer's time. Zuul can help automate
27this process, with a particular emphasis on ensuring large numbers of
28changes are tested correctly.
29
Antoine Musso55867532014-01-10 18:24:35 +010030Testing in parallel
31-------------------
32
James E. Blaircdd00072012-06-08 19:17:28 -070033A particular focus of Zuul is ensuring correctly ordered testing of
34changes in parallel. A gating system should always test each change
35applied to the tip of the branch exactly as it is going to be merged.
36A simple way to do that would be to test one change at a time, and
37merge it only if it passes tests. That works very well, but if
38changes take a long time to test, developers may have to wait a long
39time for their changes to make it into the repository. With some
40projects, it may take hours to test changes, and it is easy for
41developers to create changes at a rate faster than they can be tested
42and merged.
43
James E. Blaireff5a9d2017-06-20 00:00:37 -070044Zuul's :ref:`dependent pipeline manager<dependent_pipeline_manager>`
45allows for parallel execution of test jobs for gating while ensuring
46changes are tested correctly, exactly as if they had been tested one
47at a time. It does this by performing speculative execution of test
48jobs; it assumes that all jobs will succeed and tests them in parallel
49accordingly. If they do succeed, they can all be merged. However, if
50one fails, then changes that were expecting it to succeed are
51re-tested without the failed change. In the best case, as many
52changes as execution contexts are available may be tested in parallel
53and merged at once. In the worst case, changes are tested one at a
54time (as each subsequent change fails, changes behind it start again).
James E. Blaircdd00072012-06-08 19:17:28 -070055
56For example, if a core developer approves five changes in rapid
57succession::
58
59 A, B, C, D, E
60
61Zuul queues those changes in the order they were approved, and notes
Antoine Musso3a43e142013-10-30 23:51:58 +010062that each subsequent change depends on the one ahead of it merging:
James E. Blaircdd00072012-06-08 19:17:28 -070063
Antoine Musso3a43e142013-10-30 23:51:58 +010064.. blockdiag::
65
66 blockdiag foo {
67 node_width = 40;
68 span_width = 40;
69 A <- B <- C <- D <- E;
70 }
James E. Blaircdd00072012-06-08 19:17:28 -070071
72Zuul then starts immediately testing all of the changes in parallel.
73But in the case of changes that depend on others, it instructs the
74test system to include the changes ahead of it, with the assumption
75they pass. That means jobs testing change *B* include change *A* as
76well::
77
78 Jobs for A: merge change A, then test
79 Jobs for B: merge changes A and B, then test
80 Jobs for C: merge changes A, B and C, then test
81 Jobs for D: merge changes A, B, C and D, then test
82 Jobs for E: merge changes A, B, C, D and E, then test
83
Antoine Musso3a43e142013-10-30 23:51:58 +010084Hence jobs triggered to tests A will only test A and ignore B, C, D:
James E. Blaircdd00072012-06-08 19:17:28 -070085
Antoine Musso3a43e142013-10-30 23:51:58 +010086.. blockdiag::
James E. Blaircdd00072012-06-08 19:17:28 -070087
Antoine Musso3a43e142013-10-30 23:51:58 +010088 blockdiag foo {
89 node_width = 40;
90 span_width = 40;
91 master -> A -> B -> C -> D -> E;
92 group jobs_for_A {
93 label = "Merged changes for A";
94 master -> A;
95 }
96 group ignored_to_test_A {
97 label = "Ignored changes";
98 color = "lightgray";
99 B -> C -> D -> E;
100 }
101 }
James E. Blaircdd00072012-06-08 19:17:28 -0700102
Antoine Musso3a43e142013-10-30 23:51:58 +0100103The jobs for E would include the whole dependency chain: A, B, C, D, and E.
104E will be tested assuming A, B, C, and D passed:
105
106.. blockdiag::
107
108 blockdiag foo {
109 node_width = 40;
110 span_width = 40;
111 group jobs_for_E {
112 label = "Merged changes for E";
113 master -> A -> B -> C -> D -> E;
114 }
115 }
116
117If changes *A* and *B* pass tests (green), and *C*, *D*, and *E* fail (red):
118
119.. blockdiag::
120
121 blockdiag foo {
122 node_width = 40;
123 span_width = 40;
124
125 A [color = lightgreen];
126 B [color = lightgreen];
127 C [color = pink];
128 D [color = pink];
129 E [color = pink];
130
131 master <- A <- B <- C <- D <- E;
132 }
133
134Zuul will merge change *A* followed by change *B*, leaving this queue:
135
136.. blockdiag::
137
138 blockdiag foo {
139 node_width = 40;
140 span_width = 40;
141
142 C [color = pink];
143 D [color = pink];
144 E [color = pink];
145
146 C <- D <- E;
147 }
James E. Blaircdd00072012-06-08 19:17:28 -0700148
149Since *D* was dependent on *C*, it is not clear whether *D*'s failure is the
Antoine Musso3a43e142013-10-30 23:51:58 +0100150result of a defect in *D* or *C*:
James E. Blaircdd00072012-06-08 19:17:28 -0700151
Antoine Musso3a43e142013-10-30 23:51:58 +0100152.. blockdiag::
James E. Blaircdd00072012-06-08 19:17:28 -0700153
Antoine Musso3a43e142013-10-30 23:51:58 +0100154 blockdiag foo {
155 node_width = 40;
156 span_width = 40;
James E. Blaircdd00072012-06-08 19:17:28 -0700157
Antoine Musso3a43e142013-10-30 23:51:58 +0100158 C [color = pink];
159 D [label = "D\n?"];
160 E [label = "E\n?"];
161
162 C <- D <- E;
163 }
164
165Since *C* failed, Zuul will report its failure and drop *C* from the queue,
166keeping D and E:
167
168.. blockdiag::
169
170 blockdiag foo {
171 node_width = 40;
172 span_width = 40;
173
174 D [label = "D\n?"];
175 E [label = "E\n?"];
176
177 D <- E;
178 }
James E. Blaircdd00072012-06-08 19:17:28 -0700179
180This queue is the same as if two new changes had just arrived, so Zuul
181starts the process again testing *D* against the tip of the branch, and
Antoine Musso3a43e142013-10-30 23:51:58 +0100182*E* against *D*:
183
184.. blockdiag::
185
186 blockdiag foo {
187 node_width = 40;
188 span_width = 40;
189 master -> D -> E;
190 group jobs_for_D {
191 label = "Merged changes for D";
192 master -> D;
193 }
194 group ignored_to_test_D {
195 label = "Skip";
196 color = "lightgray";
197 E;
198 }
199 }
200
201.. blockdiag::
202
203 blockdiag foo {
204 node_width = 40;
205 span_width = 40;
206 group jobs_for_E {
207 label = "Merged changes for E";
208 master -> D -> E;
209 }
210 }
211
Antoine Musso55867532014-01-10 18:24:35 +0100212
James E. Blair096a80a2015-09-28 14:19:31 -0700213Cross Project Testing
214---------------------
Antoine Musso55867532014-01-10 18:24:35 +0100215
216When your projects are closely coupled together, you want to make sure
217changes entering the gate are going to be tested with the version of
218other projects currently enqueued in the gate (since they will
219eventually be merged and might introduce breaking features).
220
James E. Blaireff5a9d2017-06-20 00:00:37 -0700221Such relationships can be defined in Zuul configuration by placing
222projects in a shared queue within a dependent pipeline. Whenever
223changes for any project enter a pipeline with such a shared queue,
224they are tested together, such that the commits for the changes ahead
225in the queue are automatically present in the jobs for the changes
226behind them. See :ref:`project` for more details.
Antoine Musso55867532014-01-10 18:24:35 +0100227
James E. Blaireff5a9d2017-06-20 00:00:37 -0700228A given dependent pipeline may have as many shared change queues as
229necessary, so groups of related projects may share a change queue
230without interfering with unrelated projects. Independent pipelines do
231not use shared change queues, however, they may still be used to test
232changes across projects using cross-project dependencies.
Antoine Musso55867532014-01-10 18:24:35 +0100233
James E. Blaireff5a9d2017-06-20 00:00:37 -0700234.. _dependencies:
Antoine Musso55867532014-01-10 18:24:35 +0100235
James E. Blaireff5a9d2017-06-20 00:00:37 -0700236Cross-Project Dependencies
237--------------------------
Antoine Musso55867532014-01-10 18:24:35 +0100238
James E. Blaireff5a9d2017-06-20 00:00:37 -0700239Zuul permits users to specify dependencies across projects. Using a
240special footer in Git commit messages, users may specify that a change
241depends on another change in any repository known to Zuul.
Antoine Musso55867532014-01-10 18:24:35 +0100242
James E. Blaireff5a9d2017-06-20 00:00:37 -0700243Zuul's cross-project dependencies behave like a directed acyclic graph
244(DAG), like git itself, to indicate a one-way dependency relationship
245between changes in different git repositories. Change A may depend on
246B, but B may not depend on A.
Antoine Musso55867532014-01-10 18:24:35 +0100247
James E. Blaireff5a9d2017-06-20 00:00:37 -0700248.. TODO: update for v3 crd syntax
James E. Blair096a80a2015-09-28 14:19:31 -0700249
Antoine Musso281a89e2015-09-30 15:05:49 +0200250To use them, include ``Depends-On: <gerrit-change-id>`` in the footer of
James E. Blair096a80a2015-09-28 14:19:31 -0700251a commit message. Use the full Change-ID ('I' + 40 characters).
252
253
Antoine Musso281a89e2015-09-30 15:05:49 +0200254Dependent Pipeline
255~~~~~~~~~~~~~~~~~~
James E. Blair096a80a2015-09-28 14:19:31 -0700256
James E. Blaireff5a9d2017-06-20 00:00:37 -0700257When Zuul sees changes with cross-project dependencies, it serializes
258them in the usual manner when enqueuing them into a pipeline. This
259means that if change A depends on B, then when they are added to a
260dependent pipeline, B will appear first and A will follow:
James E. Blair096a80a2015-09-28 14:19:31 -0700261
Antoine Musso281a89e2015-09-30 15:05:49 +0200262.. blockdiag::
263 :align: center
James E. Blair096a80a2015-09-28 14:19:31 -0700264
Antoine Musso281a89e2015-09-30 15:05:49 +0200265 blockdiag crd {
266 orientation = portrait
267 span_width = 30
268 class greendot [
269 label = "",
270 shape = circle,
271 color = green,
272 width = 20, height = 20
273 ]
James E. Blair096a80a2015-09-28 14:19:31 -0700274
Antoine Musso281a89e2015-09-30 15:05:49 +0200275 A_status [ class = greendot ]
276 B_status [ class = greendot ]
277 B_status -- A_status
278
279 'Change B\nChange-Id: Iabc' <- 'Change A\nDepends-On: Iabc'
280 }
281
282If tests for B fail, both B and A will be removed from the pipeline, and
283it will not be possible for A to merge until B does.
284
285
286.. note::
287
James E. Blaireff5a9d2017-06-20 00:00:37 -0700288 If changes with cross-project dependencies do not share a change
289 queue then Zuul is unable to enqueue them together, and the first
290 will be required to merge before the second is enqueued.
Antoine Musso281a89e2015-09-30 15:05:49 +0200291
292Independent Pipeline
293~~~~~~~~~~~~~~~~~~~~
294
295When changes are enqueued into an independent pipeline, all of the
James E. Blaireff5a9d2017-06-20 00:00:37 -0700296related dependencies (both normal git-dependencies that come from
297parent commits as well as cross-project dependencies) appear in a
298dependency graph, as in a dependent pipeline. This means that even in
299an independent pipeline, your change will be tested with its
300dependencies. Changes that were previously unable to be fully tested
301until a related change landed in a different repository may now be
302tested together from the start.
James E. Blair096a80a2015-09-28 14:19:31 -0700303
James E. Blaireff5a9d2017-06-20 00:00:37 -0700304All of the changes are still independent (you will note that the whole
305pipeline does not share a graph as in a dependent pipeline), but for
306each change tested, all of its dependencies are visually connected to
307it, and they are used to construct the git repositories that Zuul uses
Antoine Musso281a89e2015-09-30 15:05:49 +0200308when testing.
309
James E. Blair096a80a2015-09-28 14:19:31 -0700310When looking at this graph on the status page, you will note that the
311dependencies show up as grey dots, while the actual change tested shows
Antoine Musso281a89e2015-09-30 15:05:49 +0200312up as red or green (depending on the jobs results):
313
314.. blockdiag::
315 :align: center
316
317 blockdiag crdgrey {
318 orientation = portrait
319 span_width = 30
320 class dot [
321 label = "",
322 shape = circle,
323 width = 20, height = 20
324 ]
325
326 A_status [class = "dot", color = green]
327 B_status [class = "dot", color = grey]
328 B_status -- A_status
329
330 "Change B" <- "Change A\nDepends-On: B"
331 }
332
333This is to indicate that the grey changes are only there to establish
334dependencies. Even if one of the dependencies is also being tested, it
335will show up as a grey dot when used as a dependency, but separately and
336additionally will appear as its own red or green dot for its test.
337
James E. Blair096a80a2015-09-28 14:19:31 -0700338
James E. Blaireff5a9d2017-06-20 00:00:37 -0700339.. TODO: relevant for v3?
340
James E. Blair096a80a2015-09-28 14:19:31 -0700341Multiple Changes
342~~~~~~~~~~~~~~~~
343
344A Gerrit change ID may refer to multiple changes (on multiple branches
345of the same project, or even multiple projects). In these cases, Zuul
346will treat all of the changes with that change ID as dependencies. So
347if you say that change in project A Depends-On a change ID that has
348changes in two branches of project B, then when testing the change to
349project A, both project B changes will be applied, and when deciding
350whether the project A change can merge, both changes must merge ahead
351of it.
352
Antoine Musso281a89e2015-09-30 15:05:49 +0200353.. blockdiag::
354 :align: center
355
356 blockdiag crdmultirepos {
357 orientation = portrait
358 span_width = 30
359 class greendot [
360 label = "",
361 shape = circle,
362 color = green,
363 width = 20, height = 20
364 ]
365
366 B_stable_status [ class = "greendot" ]
367 B_master_status [ class = "greendot" ]
368 A_status [ class = "greendot" ]
369 B_stable_status -- B_master_status -- A_status
370
371 A [ label = "Repo A\nDepends-On: I123" ]
372 group {
373 orientation = portrait
374 label = "Dependencies"
375 color = "lightgray"
376
377 B_stable [ label = "Repo B\nChange-Id: I123\nBranch: stable" ]
378 B_master [ label = "Repo B\nChange-Id: I123\nBranch: master" ]
379 }
380 B_master <- A
381 B_stable <- A
382
383 }
384
James E. Blair096a80a2015-09-28 14:19:31 -0700385A change may depend on more than one Gerrit change ID as well. So it
386is possible for a change in project A to depend on a change in project
Antoine Musso281a89e2015-09-30 15:05:49 +0200387B and a change in project C. Simply add more ``Depends-On:`` lines to
388the commit message footer.
389
390.. blockdiag::
391 :align: center
392
393 blockdiag crdmultichanges {
394 orientation = portrait
395 span_width = 30
396 class greendot [
397 label = "",
398 shape = circle,
399 color = green,
400 width = 20, height = 20
401 ]
402
403 C_status [ class = "greendot" ]
404 B_status [ class = "greendot" ]
405 A_status [ class = "greendot" ]
406 C_status -- B_status -- A_status
407
408 A [ label = "Repo A\nDepends-On: I123\nDepends-On: Iabc" ]
409 group {
410 orientation = portrait
411 label = "Dependencies"
412 color = "lightgray"
413
414 B [ label = "Repo B\nChange-Id: I123" ]
415 C [ label = "Repo C\nChange-Id: Iabc" ]
416 }
417 B, C <- A
418 }
James E. Blair096a80a2015-09-28 14:19:31 -0700419
James E. Blaireff5a9d2017-06-20 00:00:37 -0700420.. TODO: update for v3
421
James E. Blair096a80a2015-09-28 14:19:31 -0700422Cycles
423~~~~~~
424
James E. Blaireff5a9d2017-06-20 00:00:37 -0700425If a cycle is created by use of cross-project dependencies, Zuul will
426abort its work very early. There will be no message in Gerrit and no
427changes that are part of the cycle will be enqueued into any pipeline.
428This is to protect Zuul from infinite loops.