Gitiles
Code Review Sign In
gerrit.cesnet.cz / github / onqtam / doctest / 4ca98877b41a3f01bccc1c9d057b69c1fe0620f4 / . / doc / html_generated / benchmarks.html
blob: 038c20cd4e32f6049d9d453e3193e80ddb563839 [file] [log] [blame]
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]1<!DOCTYPE html>
2<html>
3<title>benchmarks</title>
4<xmp theme="united" style="display:none;">
5
onqtamb8220c52017-05-16 00:21:15 +0300[diff] [blame]6# Benchmarks
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]7
onqtamb8220c52017-05-16 00:21:15 +0300[diff] [blame]8The benchmarks are done with [**this**](../../scripts/bench/bench.py) script using CMake. There are 3 benchmarking scenarios:
onqtamb6c69672016-09-21 15:41:52 +0300[diff] [blame]9
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]10- [the cost of including the header](#cost-of-including-the-header)
11- [the cost of an assertion macro](#cost-of-an-assertion-macro)
onqtamb8220c52017-05-16 00:21:15 +0300[diff] [blame]12- [runtime speed of lots of asserts](#runtime-benchmarks)
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]13
14Compilers used:
onqtamb6c69672016-09-21 15:41:52 +0300[diff] [blame]15
onqtam6ff3d352017-09-11 14:35:06 +0300[diff] [blame]16- WINDOWS: Microsoft Visual Studio Community 2017 - Version 15.3.3+26730.12
17- WINDOWS: gcc 7.1.0 (x86_64-posix-seh-rev2, Built by MinGW-W64 project)
onqtamb8220c52017-05-16 00:21:15 +0300[diff] [blame]18- LINUX: gcc 6.3.0 20170406 (Ubuntu 6.3.0-12ubuntu2)
19- LINUX: clang 4.0.0-1 (tags/RELEASE_400/rc1) Target: x86_64-pc-linux-gnu
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]20
onqtamb6c69672016-09-21 15:41:52 +0300[diff] [blame]21Environment used (Intel i7 3770k, 16g RAM):
22
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]23- Windows 7 - on an SSD
onqtamb8220c52017-05-16 00:21:15 +0300[diff] [blame]24- Ubuntu 17.04 in a VirtualBox VM - on a HDD
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]25
onqtam6ff3d352017-09-11 14:35:06 +0300[diff] [blame]26**doctest** version: 1.2.2 (released on 2017.09.05)
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]27
onqtam6ff3d352017-09-11 14:35:06 +0300[diff] [blame]28[**Catch**](https://github.com/philsquared/Catch) version: 2.0.0-develop.3 (released on 2017.08.30)
onqtamb8220c52017-05-16 00:21:15 +0300[diff] [blame]29
30# Compile time benchmarks
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]31
32## Cost of including the header
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]33
34This is a benchmark that is relevant only to single header and header only frameworks - like **doctest** and [**Catch**](https://github.com/philsquared/Catch).
35
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]36The script generates 201 source files and in 200 of them makes a function in the form of ```int f135() { return 135; }``` and in ```main.cpp``` it forward declares all the 200 such dummy functions and accumulates their result to return from the ```main()``` function. This is done to ensure that all source files are built and that the linker doesn't remove/optimize anything.
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]37
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]38- **baseline** - how much time the source files need for a single threaded build with ```msbuild```/```make```
39- **+ implement** - only in ```main.cpp``` the header is included with a ```#define``` before it so the test runner gets implemented:
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]40
onqtamb8220c52017-05-16 00:21:15 +0300[diff] [blame]41```
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]42#define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN
43#include "doctest.h"
onqtamb8220c52017-05-16 00:21:15 +0300[diff] [blame]44```
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]45- **+ header everywhere** - the framework header is also included in all the other source files
onqtam6ff3d352017-09-11 14:35:06 +0300[diff] [blame]46- **+ disabled** - remove everything testing-related from the binary
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]47
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]48| doctest | baseline | + implement | + header everywhere | + disabled |
49|---------------------|----------|-------------|---------------------|------------|
onqtam6ff3d352017-09-11 14:35:06 +0300[diff] [blame]50| MSVC Debug | 6.77 | 8.28 | 11.73 | 8.73 |
51| MSVC Release | 6.35 | 8.57 | 12.18 | 8.28 |
52| MinGW GCC Debug | 10.23 | 13.03 | 17.62 | 12.29 |
53| MinGW GCC Release | 10.33 | 13.68 | 17.87 | 13.11 |
54| Linux GCC Debug | 5.01 | 6.24 | 10.48 | 6.49 |
55| Linux GCC Release | 4.58 | 7.30 | 11.70 | 7.41 |
56| Linux Clang Debug | 8.80 | 9.70 | 14.92 | 10.89 |
57| Linux Clang Release | 9.29 | 12.05 | 17.51 | 11.56 |
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]58
onqtam6ff3d352017-09-11 14:35:06 +0300[diff] [blame]59| Catch | baseline | + implement | + header everywhere | + disabled |
60|---------------------|----------|-------------|---------------------|------------|
61| MSVC Debug | 6.78 | 10.00 | 107.85 | 115.05 |
62| MSVC Release | 6.36 | 11.19 | 102.69 | 109.06 |
63| MinGW GCC Debug | 10.36 | 41.83 | 124.41 | 126.70 |
64| MinGW GCC Release | 10.49 | 21.93 | 97.81 | 105.47 |
65| Linux GCC Debug | 4.40 | 12.39 | 94.34 | 93.68 |
66| Linux GCC Release | 4.55 | 15.75 | 94.28 | 93.80 |
67| Linux Clang Debug | 9.30 | 15.00 | 105.84 | 103.05 |
68| Linux Clang Release | 9.68 | 22.75 | 114.36 | 111.32 |
onqtamb8220c52017-05-16 00:21:15 +0300[diff] [blame]69
70<img src="../../scripts/data/benchmarks/header.png" width="430" align="right">
71<img src="../../scripts/data/benchmarks/implement.png" width="430">
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]72
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]73### Conclusion
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]74
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]75#### doctest
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]76
onqtam0c65bef2017-09-11 15:34:43 +0300[diff] [blame]77- instantiating the test runner in one source file costs ~1-3 seconds ```implement - baseline```
78- the inclusion of ```doctest.h``` in one source file costs between 17ms - 27ms ```(header_everywhere - implement) / 200```
79- including the library everywhere but everything disabled costs around 2 seconds ```disabled - baseline``` for 200 files
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]80
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]81#### [Catch](https://github.com/philsquared/Catch)
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]82
onqtam0c65bef2017-09-11 15:34:43 +0300[diff] [blame]83- instantiating the test runner in one source file costs ~4-31 seconds ```implement - baseline```
84- the inclusion of ```catch.hpp``` in one source file costs between 390ms - 490ms ```(header_everywhere - implement) / 200```
85- using the config option to disable the library (**```CATCH_CONFIG_DISABLE```**) has no effect on the header cost
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]86
87----------
88
onqtam0c65bef2017-09-11 15:34:43 +0300[diff] [blame]89So if ```doctest.h``` costs 17ms and ```catch.hpp``` costs 490ms on MSVC - then the **doctest** header is >> **27** << times lighter (for MSVC)!
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]90
91----------
92
93The results are in seconds and are in **no way** intended to bash [**Catch**](https://github.com/philsquared/Catch) - the **doctest** framework wouldn't exist without it.
94
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]95The reason the **doctest** header is so light on compile times is because it forward declares everything and doesn't drag any headers in the source files (except for the source file where the test runner gets implemented). This was a key design decision.
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]96
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]97## Cost of an assertion macro
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]98
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]99The script generates 11 ```.cpp``` files and in 10 of them makes 50 test cases with 100 asserts in them (of the form ```CHECK(a==b)``` where ```a``` and ```b``` are always the same ```int``` variables) - **50k** asserts! The testing framework gets implemented in ```main.cpp```.
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]100
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]101- **baseline** - how much time a single threaded build takes with the header included everywhere - no test cases or asserts!
102- ```CHECK(a==b)``` - will add ```CHECK()``` asserts which decompose the expression with template machinery
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]103
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]104**doctest** specific:
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]105
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]106- ```CHECK_EQ(a,b)``` - will use ```CHECK_EQ(a,b)``` instead of the expression decomposing ones
107- ```FAST_CHECK_EQ(a,b)``` - will use ```FAST_CHECK_EQ(a,b)``` instead of the expression decomposing ones
108- **+faster** - will add [**```DOCTEST_CONFIG_SUPER_FAST_ASSERTS```**](configuration.html#doctest_config_super_fast_asserts) which speeds up ```FAST_CHECK_EQ(a,b)``` even more
109- **+disabled** - all test case and assert macros will be disabled with [**```DOCTEST_CONFIG_DISABLE```**](configuration.html#doctest_config_disable)
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]110
onqtamb8220c52017-05-16 00:21:15 +0300[diff] [blame]111[**Catch**](https://github.com/philsquared/Catch) specific:
112
113- **+faster** - will add [**```CATCH_CONFIG_FAST_COMPILE```**](https://github.com/philsquared/Catch/blob/master/docs/configuration.html#catch_config_fast_compile) which speeds up the compilation of the normal asserts ```CHECK(a==b)```
onqtam6ff3d352017-09-11 14:35:06 +0300[diff] [blame]114- **+disabled** - all test case and assert macros will be disabled with **```CATCH_CONFIG_DISABLE```**
onqtamb8220c52017-05-16 00:21:15 +0300[diff] [blame]115
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]116| doctest | baseline | ```CHECK(a==b)``` | ```CHECK_EQ(a,b)``` | ```FAST_CHECK_EQ(a,b)``` | +faster | +disabled |
117|---------------------|----------|-------------------|---------------------|--------------------------|---------|-----------|
onqtam6ff3d352017-09-11 14:35:06 +0300[diff] [blame]118| MSVC Debug | 3.08 | 23.72 | 18.15 | 8.38 | 5.67 | 2.23 |
119| MSVC Release | 3.61 | 43.75 | 24.28 | 11.36 | 7.22 | 2.15 |
120| MinGW GCC Debug | 3.90 | 85.47 | 58.62 | 24.40 | 12.12 | 1.71 |
121| MinGW GCC Release | 4.51 | 224.49 | 148.84 | 47.25 | 18.73 | 2.40 |
122| Linux GCC Debug | 2.01 | 78.38 | 50.61 | 17.62 | 9.87 | 1.11 |
123| Linux GCC Release | 3.20 | 199.78 | 123.42 | 32.47 | 19.52 | 1.97 |
124| Linux Clang Debug | 1.71 | 77.39 | 49.97 | 17.60 | 7.57 | 1.18 |
125| Linux Clang Release | 3.64 | 136.82 | 80.19 | 20.72 | 12.34 | 1.45 |
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]126
onqtamb8220c52017-05-16 00:21:15 +0300[diff] [blame]127And here is [**Catch**](https://github.com/philsquared/Catch) which only has normal ```CHECK(a==b)``` asserts:
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]128
onqtam6ff3d352017-09-11 14:35:06 +0300[diff] [blame]129| Catch | baseline | ```CHECK(a==b)``` | +faster | +disabled |
130|---------------------|----------|-------------------|---------|-----------|
131| MSVC Debug | 9.58 | 37.69 | 25.21 | 10.40 |
132| MSVC Release | 10.85 | 260.55 | 121.38 | 11.56 |
133| MinGW GCC Debug | 36.24 | 159.15 | 133.98 | 33.57 |
134| MinGW GCC Release | 16.15 | 740.71 | 562.60 | 16.41 |
135| Linux GCC Debug | 12.71 | 142.92 | 108.07 | 12.05 |
136| Linux GCC Release | 15.62 | 825.42 | 612.06 | 15.51 |
137| Linux Clang Debug | 10.48 | 115.19 | 89.59 | 10.78 |
138| Linux Clang Release | 18.25 | 393.31 | 316.98 | 17.19 |
onqtamb8220c52017-05-16 00:21:15 +0300[diff] [blame]139
140<img src="../../scripts/data/benchmarks/asserts.png">
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]141
142### Conclusion
143
onqtamb8220c52017-05-16 00:21:15 +0300[diff] [blame]144**doctest**:
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]145
onqtam0c65bef2017-09-11 15:34:43 +0300[diff] [blame]146- is around 30% to 75% faster than [**Catch**](https://github.com/philsquared/Catch) when using normal expression decomposing ```CHECK(a==b)``` asserts
onqtamb8220c52017-05-16 00:21:15 +0300[diff] [blame]147- asserts of the form ```CHECK_EQ(a,b)``` with no expression decomposition - around 25%-45% faster than ```CHECK(a==b)```
148- fast asserts like ```FAST_CHECK_EQ(a,b)``` with no ```try/catch``` blocks - around 60-80% faster than ```CHECK_EQ(a,b)```
149- the [**```DOCTEST_CONFIG_SUPER_FAST_ASSERTS```**](configuration.html#doctest_config_super_fast_asserts) identifier which makes the fast assertions even faster by another 50-80%
onqtam1435c012016-09-21 15:29:11 +0300[diff] [blame]150- using the [**```DOCTEST_CONFIG_DISABLE```**](configuration.html#doctest_config_disable) identifier the assertions just disappear as if they were never written
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]151
onqtamb8220c52017-05-16 00:21:15 +0300[diff] [blame]152[**Catch**](https://github.com/philsquared/Catch):
153
onqtam0c65bef2017-09-11 15:34:43 +0300[diff] [blame]154- using [**```CATCH_CONFIG_FAST_COMPILE```**](https://github.com/philsquared/Catch/blob/master/docs/configuration.html#catch_config_fast_compile) results in 15%-55% faster build times for asserts.
155- using the **```CATCH_CONFIG_DISABLE```** identifier provides the same great benefits for assertion macros as the doctest version ([**```DOCTEST_CONFIG_DISABLE```**](configuration.html#doctest_config_disable)) - unlike the case for the header cost
onqtamb8220c52017-05-16 00:21:15 +0300[diff] [blame]156
157## Runtime benchmarks
158
159The runtime benchmarks consist of a single test case with a loop of 10 million iterations performing the task - a single normal assert (using expression decomposition) or the assert + the logging of the loop iterator ```i```:
160
161```
162for(int i = 0; i < 10000000; ++i)
163 CHECK(i == i);
164```
165
166or
167
168```
169for(int i = 0; i < 10000000; ++i) {
170 INFO(i);
171 CHECK(i == i);
172}
173```
174
175Note that the assert always passes - the goal should be to optimize for the common case - lots of passing test cases and a few that maybe fail.
176
onqtam4aff18c2017-05-17 04:10:03 +0300[diff] [blame]177| doctest | assert | + info | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | Catch | assert | + info |
178|---------------------|---------|---------|-|---------------------|---------|---------|
onqtam6ff3d352017-09-11 14:35:06 +0300[diff] [blame]179| MSVC Debug | 5.04 | 13.03 | | MSVC Debug | 101.07 | 338.41 |
180| MSVC Release | 0.73 | 1.67 | | MSVC Release | 1.75 | 10.99 |
181| MinGW GCC Debug | 2.11 | 4.50 | | MinGW GCC Debug | 4.76 | 18.22 |
182| MinGW GCC Release | 0.36 | 0.86 | | MinGW GCC Release | 1.24 | 7.29 |
183| Linux GCC Debug | 2.49 | 4.97 | | Linux GCC Debug | 5.41 | 19.01 |
184| Linux GCC Release | 0.29 | 0.66 | | Linux GCC Release | 1.20 | 7.88 |
185| Linux Clang Debug | 2.39 | 4.76 | | Linux Clang Debug | 5.12 | 17.66 |
186| Linux Clang Release | 0.39 | 0.70 | | Linux Clang Release | 0.99 | 7.26 |
onqtamb8220c52017-05-16 00:21:15 +0300[diff] [blame]187
188<img src="../../scripts/data/benchmarks/runtime_info.png" width="430" align="right">
189<img src="../../scripts/data/benchmarks/runtime_assert.png" width="430">
190
191### Conclusion
192
onqtam0c65bef2017-09-11 15:34:43 +0300[diff] [blame]193**doctest** is significantly faster - between 2.5 and 26 times.
onqtamb8220c52017-05-16 00:21:15 +0300[diff] [blame]194
195In these particular cases **doctest** makes 0 allocations when the assert doesn't fail - it uses lazy stringification (meaning it stringifies the expression or the logged loop counter only if it has to) and a small-buffer optimized string class to achieve these results.
196
onqtambf4f4532016-09-22 21:24:54 +0300[diff] [blame]197----------
198
onqtamef9da522017-09-11 17:16:12 +0300[diff] [blame]199The bar charts were generated using [**this google spreadsheet**](https://docs.google.com/spreadsheets/d/1p3MAURUfPzKT7gtJOVuJU2_yVKSqkoD1nbypA1K3618) by pasting the data from the tables.
200
onqtambf4f4532016-09-22 21:24:54 +0300[diff] [blame]201If you want a benchmark that is not synthetic - check out [**this blog post**](http://baptiste-wicht.com/posts/2016/09/blazing-fast-unit-test-compilation-with-doctest-11.html) of [**Baptiste Wicht**](https://github.com/wichtounet) who tested the compile times of the asserts in the 1.1 release with his [**Expression Templates Library**](https://github.com/wichtounet/etl)!
202
203While reading the post - keep in mind that if a part of a process takes 50% of the time and is made 10000 times faster - the overall process would still be only roughly 50% faster.
204
onqtam8126b562016-05-27 17:01:15 +0300[diff] [blame]205---------------
206
207[Home](readme.html#reference)
208
209
210</xmp>
211<script src="strapdown.js/strapdown.js"></script>
212</html>