Checking of coverage changes
Enable running the build + test twice, once as usual and export the
resulting coverage, and then one more time with the previous state of
the project's repo. Then use coverage data from both runs for a nice
visualization.
Here's how to enable it:
1) Enable coverage output manually in the main build job of the project.
Commit the change. This change must produce `coverage.xml` in a
well-known location that will be set in the next commit.
2) Apply this change (the one you're reading right now) to create a new
job. It's going to be similar to the main build job, but it should *not*
inherit from the main build job. Here's how this job works:
- It uses `playbooks/checkout-original-state/run.yaml` as a `pre-run` to
ensure that it will build *previous* version of the code and therefore
regenerate the previous state. Internally, this uses git's workdirs, so
it switches to a fake branch in order to be able to go back to prepare
the "new" tree as well later on.
A possible gotcha is that this also checks out the previous version of
any shell scripts (etc) used for building (hence step #1 above).
- The main `run` should contain two playbooks: the first one same as in
the "main build job" which just performs the build of the previous
version of the project. This creates a file `coverage.xml`, this time for
the "previous commit".
The second `run` playbook is `playbooks/coverage-diff/generate.yaml`. It
uses pycobertura to create a report of changed coverage as introduced by
this change, as well as a full report for troubleshooting and debugging.
I'm not using `post_run` for coverage checking because any failure would
"only" result in POST_FAILURE instead of a proper FAILURE.
The original job's coverage.xml is passed to the newly added job as an
artifact. This requires both `provides`/`requires` relation *and*
setting up job dependencies. This is better done in each "leaf"
project's own .zuul.yaml, especially if these projects are already using
artifacts for passing other stuff between jobs. The CObertura XML
artifact is compared with the one for the previous version of code
(prior to this change) that we just generated, and it's used for a
success/failure indication of this job's result.
Change-Id: Idd96dbb3b2641444d4e4f26d2778b1a3a67be8cc
7 files changed