Gitiles
Code Review Sign In
gerrit.cesnet.cz / github / onqtam / doctest / 80064af216f02d79ff75a0988769020cc7ed7cfd / . / doc / html_generated / benchmarks.html
blob: f82a623c44e5fbb24beb03eac2b6b412523b600f [file] [log] [blame]
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]1<!DOCTYPE html>
2<html>
3<title>benchmarks</title>
4<xmp theme="united" style="display:none;">
5
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]6# Compile time benchmarks
7
8The benchmarks are done with [**this**](../../scripts/bench/bench.py) script using CMake. There are 2 benchmarking scenarios:
onqtamb6c69672016-09-21 15:41:52 +0300[diff] [blame]9
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]10- [the cost of including the header](#cost-of-including-the-header)
11- [the cost of an assertion macro](#cost-of-an-assertion-macro)
12
13Compilers used:
onqtamb6c69672016-09-21 15:41:52 +0300[diff] [blame]14
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]15- WINDOWS: Microsoft Visual Studio Community 2015 - Version 14.0.25431.01 Update 3
16- WINDOWS: gcc 6.2.0 (x86_64-posix-seh-rev0, Built by MinGW-W64 project)
17- LINUX: gcc 5.2.1 20151010 (Ubuntu 5.2.1-22ubuntu2)
18- LINUX: clang 3.6.2-1 (tags/RELEASE_362/final) (based on LLVM 3.6.2) Ubuntu - Target: x86_64-pc-linux-gnu
19
onqtamb6c69672016-09-21 15:41:52 +0300[diff] [blame]20Environment used (Intel i7 3770k, 16g RAM):
21
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]22- Windows 7 - on an SSD
23- Ubuntu 15.10 in a VirtualBox VM - on a HDD
24
25**doctest** version: 1.1.0 (released on 2016.09.21)
26
27[**Catch**](https://github.com/philsquared/Catch) version: 1.5.6 (released on 2016.06.09)
28
29## Cost of including the header
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]30
31This is a benchmark that is relevant only to single header and header only frameworks - like **doctest** and [**Catch**](https://github.com/philsquared/Catch).
32
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]33The script generates 201 source files and in 200 of them makes a function in the form of ```int f135() { return 135; }``` and in ```main.cpp``` it forward declares all the 200 such dummy functions and accumulates their result to return from the ```main()``` function. This is done to ensure that all source files are built and that the linker doesn't remove/optimize anything.
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]34
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]35- **baseline** - how much time the source files need for a single threaded build with ```msbuild```/```make```
36- **+ implement** - only in ```main.cpp``` the header is included with a ```#define``` before it so the test runner gets implemented:
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]37
38 ```
39#define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN
40#include "doctest.h"
41 ```
42- **+ header everywhere** - the framework header is also included in all the other source files
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]43- **+ disabled** - **doctest** specific - only this framework can remove everything related to it from the binary
44
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]45| doctest | baseline | + implement | + header everywhere | + disabled |
46|---------------------|----------|-------------|---------------------|------------|
47| MSVC Debug | 5.9 | 7.1 | 8.3 | 7.0 |
48| MSVC Release | 5.4 | 6.9 | 8.7 | 6.5 |
49| MinGW GCC Debug | 9.4 | 11.7 | 14.4 | 11.1 |
50| MinGW GCC Release | 9.6 | 12.3 | 14.9 | 11.4 |
51| Linux GCC Debug | 6.3 | 7.1 | 10.2 | 7.4 |
52| Linux GCC Release | 6.5 | 8.4 | 10.8 | 7.8 |
53| Linux Clang Debug | 6.9 | 7.6 | 10.6 | 8.2 |
54| Linux Clang Release | 7.2 | 8.4 | 11.4 | 8.4 |
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]55
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]56| Catch | baseline | + implement | + header everywhere |
57|---------------------|----------|-------------|---------------------|
58| MSVC Debug | 5.9 | 8.5 | 102 |
59| MSVC Release | 5.4 | 10.3 | 96 |
60| MinGW GCC Debug | 9.4 | 24.5 | 125 |
61| MinGW GCC Release | 9.6 | 18.4 | 113 |
62| Linux GCC Debug | 6.3 | 10.4 | 59 |
63| Linux GCC Release | 6.5 | 14.1 | 64 |
64| Linux Clang Debug | 6.9 | 9.8 | 64 |
65| Linux Clang Release | 7.2 | 12.8 | 67 |
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]66
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]67### Conclusion
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]68
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]69#### doctest
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]70
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]71- instantiating the test runner in one source file costs ~1.5 seconds ```implement - baseline```
72- the inclusion of ```doctest.h``` in one source file costs below 9ms ```(header_everywhere - implement) / 200```
73- including the library everywhere but everything disabled costs less than 2 seconds ```disabled - baseline``` for 200 files
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]74
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]75#### [Catch](https://github.com/philsquared/Catch)
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]76
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]77- instantiating the test runner in one source file costs ~5 second ```implement - baseline```
78- the inclusion of ```catch.hpp``` in one source file costs around 430ms ```(header_everywhere - implement) / 200```
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]79
80----------
81
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]82So if ```doctest.h``` costs 8ms and ```catch.hpp``` costs 430ms on MSVC - then the **doctest** header is >> **54** << times lighter!
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]83
84----------
85
86The results are in seconds and are in **no way** intended to bash [**Catch**](https://github.com/philsquared/Catch) - the **doctest** framework wouldn't exist without it.
87
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]88The reason the **doctest** header is so light on compile times is because it forward declares everything and doesn't drag any headers in the source files (except for the source file where the test runner gets implemented). This was a key design decision.
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]89
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]90## Cost of an assertion macro
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]91
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]92The script generates 11 ```.cpp``` files and in 10 of them makes 50 test cases with 100 asserts in them (of the form ```CHECK(a==b)``` where ```a``` and ```b``` are always the same ```int``` variables) - **50k** asserts! The testing framework gets implemented in ```main.cpp```.
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]93
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]94- **baseline** - how much time a single threaded build takes with the header included everywhere - no test cases or asserts!
95- ```CHECK(a==b)``` - will add ```CHECK()``` asserts which decompose the expression with template machinery
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]96
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]97**doctest** specific:
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]98
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]99- ```CHECK_EQ(a,b)``` - will use ```CHECK_EQ(a,b)``` instead of the expression decomposing ones
100- ```FAST_CHECK_EQ(a,b)``` - will use ```FAST_CHECK_EQ(a,b)``` instead of the expression decomposing ones
101- **+faster** - will add [**```DOCTEST_CONFIG_SUPER_FAST_ASSERTS```**](configuration.html#doctest_config_super_fast_asserts) which speeds up ```FAST_CHECK_EQ(a,b)``` even more
102- **+disabled** - all test case and assert macros will be disabled with [**```DOCTEST_CONFIG_DISABLE```**](configuration.html#doctest_config_disable)
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]103
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]104| doctest | baseline | ```CHECK(a==b)``` | ```CHECK_EQ(a,b)``` | ```FAST_CHECK_EQ(a,b)``` | +faster | +disabled |
105|---------------------|----------|-------------------|---------------------|--------------------------|---------|-----------|
106| MSVC Debug | 2.5 | 21 | 16.2 | 6.7 | 4.4 | 2.2 |
107| MSVC Release | 2.6 | 64 | 55 | 63 | 5.3 | 1.8 |
108| MinGW GCC Debug | 3.2 | 77 | 52 | 29.5 | 12.2 | 1.6 |
109| MinGW GCC Release | 3.9 | 425 | 295 | 81 | 18.6 | 1.9 |
110| Linux GCC Debug | 1.3 | 72 | 48 | 20.3 | 9.5 | 0.9 |
111| Linux GCC Release | 2.3 | 339 | 210 | 42 | 18.3 | 1.3 |
112| Linux Clang Debug | 1.3 | 70 | 46 | 18.8 | 7.0 | 0.9 |
113| Linux Clang Release | 1.8 | 205 | 136 | 30 | 10.8 | 1.1 |
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]114
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]115And here are [**Catch**](https://github.com/philsquared/Catch) and **doctest 1.0** which only have normal ```CHECK(a==b)``` asserts:
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]116
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]117| Catch | baseline | ```CHECK(a==b)``` | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | doctest 1.0 | ```CHECK(a==b)``` |
118|---------------------|----------|-------------------|--------------------------------------------------------------------------------------------------------------------------------------------------|---------------------|-------------------|
119| MSVC Debug | 8.4 | 34 | | MSVC Debug | 58 |
120| MSVC Release | 9.7 | 77 | | MSVC Release | 367 |
121| MinGW GCC Debug | 20.5 | 115 | | MinGW GCC Debug | 202 |
122| MinGW GCC Release | 15.1 | 496 | | MinGW GCC Release | 1257 |
123| Linux GCC Debug | 7.3 | 101 | | Linux GCC Debug | 204 |
124| Linux GCC Release | 10.3 | 435 | | Linux GCC Release | 1090 |
125| Linux Clang Debug | 6.0 | 91 | | Linux Clang Debug | 167 |
126| Linux Clang Release | 8.5 | 159 | | Linux Clang Release | 494 |
127
128### Conclusion
129
130**doctest 1.1**:
131
132- improves the compile time of it's ```CHECK(a==b)``` asserts by roughly 3 times compared to **doctest** 1.0 (released 2016.05.22)
133- is around 25% faster than [**Catch**](https://github.com/philsquared/Catch) when using expression decomposing ```CHECK(a==b)``` asserts
134- adds asserts of the form ```CHECK_EQ(a,b)``` with no expression decomposition - around 20% faster than ```CHECK(a==b)```
135- adds fast asserts like ```FAST_CHECK_EQ(a,b)``` with no ```try/catch``` blocks - around 30-70% faster than ```CHECK_EQ(a,b)```
136- adds the [**```DOCTEST_CONFIG_SUPER_FAST_ASSERTS```**](configuration.html#doctest_config_super_fast_asserts) identifier which makes the fast assertions even faster by another 35-80%
137- using the [**```DOCTEST_CONFIG_DISABLE```**](configuration.html#doctest_config_disable) identifier the assertions just disappear as if they were never written
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]138
onqtambf4f4532016-09-22 21:24:54 +0300[diff] [blame]139----------
140
141If you want a benchmark that is not synthetic - check out [**this blog post**](http://baptiste-wicht.com/posts/2016/09/blazing-fast-unit-test-compilation-with-doctest-11.html) of [**Baptiste Wicht**](https://github.com/wichtounet) who tested the compile times of the asserts in the 1.1 release with his [**Expression Templates Library**](https://github.com/wichtounet/etl)!
142
143While reading the post - keep in mind that if a part of a process takes 50% of the time and is made 10000 times faster - the overall process would still be only roughly 50% faster.
144
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]145---------------
146
147[Home](readme.html#reference)
148
149
150</xmp>
151<script src="strapdown.js/strapdown.js"></script>
152</html>