version 1.2 almost out!
diff --git a/doc/html_generated/benchmarks.html b/doc/html_generated/benchmarks.html
index f82a623..82f28ca 100644
--- a/doc/html_generated/benchmarks.html
+++ b/doc/html_generated/benchmarks.html
@@ -3,28 +3,31 @@
<title>benchmarks</title>
<xmp theme="united" style="display:none;">
-# Compile time benchmarks
+# Benchmarks
-The benchmarks are done with [**this**](../../scripts/bench/bench.py) script using CMake. There are 2 benchmarking scenarios:
+The benchmarks are done with [**this**](../../scripts/bench/bench.py) script using CMake. There are 3 benchmarking scenarios:
- [the cost of including the header](#cost-of-including-the-header)
- [the cost of an assertion macro](#cost-of-an-assertion-macro)
+- [runtime speed of lots of asserts](#runtime-benchmarks)
Compilers used:
-- WINDOWS: Microsoft Visual Studio Community 2015 - Version 14.0.25431.01 Update 3
-- WINDOWS: gcc 6.2.0 (x86_64-posix-seh-rev0, Built by MinGW-W64 project)
-- LINUX: gcc 5.2.1 20151010 (Ubuntu 5.2.1-22ubuntu2)
-- LINUX: clang 3.6.2-1 (tags/RELEASE_362/final) (based on LLVM 3.6.2) Ubuntu - Target: x86_64-pc-linux-gnu
+- WINDOWS: Microsoft Visual Studio Community 2017 - Version 15.1.26403.7
+- WINDOWS: gcc 7.1.0 (x86_64-posix-seh-rev0, Built by MinGW-W64 project)
+- LINUX: gcc 6.3.0 20170406 (Ubuntu 6.3.0-12ubuntu2)
+- LINUX: clang 4.0.0-1 (tags/RELEASE_400/rc1) Target: x86_64-pc-linux-gnu
Environment used (Intel i7 3770k, 16g RAM):
- Windows 7 - on an SSD
-- Ubuntu 15.10 in a VirtualBox VM - on a HDD
+- Ubuntu 17.04 in a VirtualBox VM - on a HDD
-**doctest** version: 1.1.0 (released on 2016.09.21)
+**doctest** version: 1.2.0 (released on 2017.05.16)
-[**Catch**](https://github.com/philsquared/Catch) version: 1.5.6 (released on 2016.06.09)
+[**Catch**](https://github.com/philsquared/Catch) version: 1.9.3 (released on 2017.04.25)
+
+# Compile time benchmarks
## Cost of including the header
@@ -35,51 +38,54 @@
- **baseline** - how much time the source files need for a single threaded build with ```msbuild```/```make```
- **+ implement** - only in ```main.cpp``` the header is included with a ```#define``` before it so the test runner gets implemented:
- ```
+```
#define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN
#include "doctest.h"
- ```
+```
- **+ header everywhere** - the framework header is also included in all the other source files
- **+ disabled** - **doctest** specific - only this framework can remove everything related to it from the binary
| doctest | baseline | + implement | + header everywhere | + disabled |
|---------------------|----------|-------------|---------------------|------------|
-| MSVC Debug | 5.9 | 7.1 | 8.3 | 7.0 |
-| MSVC Release | 5.4 | 6.9 | 8.7 | 6.5 |
-| MinGW GCC Debug | 9.4 | 11.7 | 14.4 | 11.1 |
-| MinGW GCC Release | 9.6 | 12.3 | 14.9 | 11.4 |
-| Linux GCC Debug | 6.3 | 7.1 | 10.2 | 7.4 |
-| Linux GCC Release | 6.5 | 8.4 | 10.8 | 7.8 |
-| Linux Clang Debug | 6.9 | 7.6 | 10.6 | 8.2 |
-| Linux Clang Release | 7.2 | 8.4 | 11.4 | 8.4 |
+| MSVC Debug | 7.17 | 8.50 | 12.50 | 9.42 |
+| MSVC Release | 6.72 | 8.47 | 12.51 | 8.97 |
+| MinGW GCC Debug | 10.65 | 13.56 | 18.59 | 13.34 |
+| MinGW GCC Release | 10.81 | 14.53 | 19.21 | 14.11 |
+| Linux GCC Debug | 4.98 | 6.55 | 11.22 | 6.94 |
+| Linux GCC Release | 4.49 | 7.27 | 11.92 | 7.87 |
+| Linux Clang Debug | 8.52 | 9.42 | 15.25 | 11.14 |
+| Linux Clang Release | 9.01 | 11.85 | 18.18 | 11.75 |
| Catch | baseline | + implement | + header everywhere |
|---------------------|----------|-------------|---------------------|
-| MSVC Debug | 5.9 | 8.5 | 102 |
-| MSVC Release | 5.4 | 10.3 | 96 |
-| MinGW GCC Debug | 9.4 | 24.5 | 125 |
-| MinGW GCC Release | 9.6 | 18.4 | 113 |
-| Linux GCC Debug | 6.3 | 10.4 | 59 |
-| Linux GCC Release | 6.5 | 14.1 | 64 |
-| Linux Clang Debug | 6.9 | 9.8 | 64 |
-| Linux Clang Release | 7.2 | 12.8 | 67 |
+| MSVC Debug | 7.17 | 10.60 | 128.02 |
+| MSVC Release | 6.72 | 10.90 | 119.25 |
+| MinGW GCC Debug | 10.65 | 26.12 | 127.00 |
+| MinGW GCC Release | 10.81 | 19.95 | 114.15 |
+| Linux GCC Debug | 4.98 | 9.37 | 105.66 |
+| Linux GCC Release | 4.49 | 13.04 | 105.57 |
+| Linux Clang Debug | 8.52 | 11.33 | 70.57 |
+| Linux Clang Release | 9.01 | 16.59 | 75.85 |
+
+<img src="../../scripts/data/benchmarks/header.png" width="430" align="right">
+<img src="../../scripts/data/benchmarks/implement.png" width="430">
### Conclusion
#### doctest
-- instantiating the test runner in one source file costs ~1.5 seconds ```implement - baseline```
-- the inclusion of ```doctest.h``` in one source file costs below 9ms ```(header_everywhere - implement) / 200```
-- including the library everywhere but everything disabled costs less than 2 seconds ```disabled - baseline``` for 200 files
+- instantiating the test runner in one source file costs ~1.5-3 seconds ```implement - baseline```
+- the inclusion of ```doctest.h``` in one source file costs between 20ms - 30ms ```(header_everywhere - implement) / 200```
+- including the library everywhere but everything disabled costs less than 3 seconds ```disabled - baseline``` for 200 files
#### [Catch](https://github.com/philsquared/Catch)
-- instantiating the test runner in one source file costs ~5 second ```implement - baseline```
-- the inclusion of ```catch.hpp``` in one source file costs around 430ms ```(header_everywhere - implement) / 200```
+- instantiating the test runner in one source file costs ~4-8 seconds ```implement - baseline```
+- the inclusion of ```catch.hpp``` in one source file costs between 300ms - 575ms ```(header_everywhere - implement) / 200```
----------
-So if ```doctest.h``` costs 8ms and ```catch.hpp``` costs 430ms on MSVC - then the **doctest** header is >> **54** << times lighter!
+So if ```doctest.h``` costs 20ms and ```catch.hpp``` costs 560ms on MSVC - then the **doctest** header is >> **28** << times lighter (for MSVC)!
----------
@@ -101,41 +107,101 @@
- **+faster** - will add [**```DOCTEST_CONFIG_SUPER_FAST_ASSERTS```**](configuration.html#doctest_config_super_fast_asserts) which speeds up ```FAST_CHECK_EQ(a,b)``` even more
- **+disabled** - all test case and assert macros will be disabled with [**```DOCTEST_CONFIG_DISABLE```**](configuration.html#doctest_config_disable)
+[**Catch**](https://github.com/philsquared/Catch) specific:
+
+- **+faster** - will add [**```CATCH_CONFIG_FAST_COMPILE```**](https://github.com/philsquared/Catch/blob/master/docs/configuration.html#catch_config_fast_compile) which speeds up the compilation of the normal asserts ```CHECK(a==b)```
+
| doctest | baseline | ```CHECK(a==b)``` | ```CHECK_EQ(a,b)``` | ```FAST_CHECK_EQ(a,b)``` | +faster | +disabled |
|---------------------|----------|-------------------|---------------------|--------------------------|---------|-----------|
-| MSVC Debug | 2.5 | 21 | 16.2 | 6.7 | 4.4 | 2.2 |
-| MSVC Release | 2.6 | 64 | 55 | 63 | 5.3 | 1.8 |
-| MinGW GCC Debug | 3.2 | 77 | 52 | 29.5 | 12.2 | 1.6 |
-| MinGW GCC Release | 3.9 | 425 | 295 | 81 | 18.6 | 1.9 |
-| Linux GCC Debug | 1.3 | 72 | 48 | 20.3 | 9.5 | 0.9 |
-| Linux GCC Release | 2.3 | 339 | 210 | 42 | 18.3 | 1.3 |
-| Linux Clang Debug | 1.3 | 70 | 46 | 18.8 | 7.0 | 0.9 |
-| Linux Clang Release | 1.8 | 205 | 136 | 30 | 10.8 | 1.1 |
+| MSVC Debug | 3.22 | 24.17 | 18.54 | 8.32 | 5.75 | 2.34 |
+| MSVC Release | 3.63 | 41.10 | 23.20 | 10.89 | 6.87 | 2.31 |
+| MinGW GCC Debug | 4.09 | 91.98 | 60.19 | 25.51 | 12.61 | 1.82 |
+| MinGW GCC Release | 4.74 | 240.58 | 156.79 | 50.16 | 19.72 | 2.53 |
+| Linux GCC Debug | 2.06 | 81.32 | 52.14 | 18.07 | 10.15 | 1.16 |
+| Linux GCC Release | 3.28 | 207.21 | 126.89 | 33.17 | 19.92 | 2.03 |
+| Linux Clang Debug | 1.75 | 79.41 | 51.20 | 17.78 | 7.65 | 1.20 |
+| Linux Clang Release | 3.73 | 140.79 | 82.61 | 21.19 | 12.64 | 1.46 |
-And here are [**Catch**](https://github.com/philsquared/Catch) and **doctest 1.0** which only have normal ```CHECK(a==b)``` asserts:
+And here is [**Catch**](https://github.com/philsquared/Catch) which only has normal ```CHECK(a==b)``` asserts:
-| Catch | baseline | ```CHECK(a==b)``` | | doctest 1.0 | ```CHECK(a==b)``` |
-|---------------------|----------|-------------------|--------------------------------------------------------------------------------------------------------------------------------------------------|---------------------|-------------------|
-| MSVC Debug | 8.4 | 34 | | MSVC Debug | 58 |
-| MSVC Release | 9.7 | 77 | | MSVC Release | 367 |
-| MinGW GCC Debug | 20.5 | 115 | | MinGW GCC Debug | 202 |
-| MinGW GCC Release | 15.1 | 496 | | MinGW GCC Release | 1257 |
-| Linux GCC Debug | 7.3 | 101 | | Linux GCC Debug | 204 |
-| Linux GCC Release | 10.3 | 435 | | Linux GCC Release | 1090 |
-| Linux Clang Debug | 6.0 | 91 | | Linux Clang Debug | 167 |
-| Linux Clang Release | 8.5 | 159 | | Linux Clang Release | 494 |
+| Catch | baseline | ```CHECK(a==b)``` | +faster |
+|---------------------|----------|-------------------|---------|
+| MSVC Debug | 9.94 | 40.14 | 36.66 |
+| MSVC Release | 10.66 | 231.60 | 81.90 |
+| MinGW GCC Debug | 21.20 | 129.26 | 110.95 |
+| MinGW GCC Release | 14.59 | 297.04 | 207.75 |
+| Linux GCC Debug | 10.05 | 115.53 | 98.84 |
+| Linux GCC Release | 13.29 | 294.26 | 218.37 |
+| Linux Clang Debug | 6.38 | 103.06 | 85.02 |
+| Linux Clang Release | 11.15 | 195.62 | 156.04 |
+
+<img src="../../scripts/data/benchmarks/asserts.png">
### Conclusion
-**doctest 1.1**:
+**doctest**:
-- improves the compile time of it's ```CHECK(a==b)``` asserts by roughly 3 times compared to **doctest** 1.0 (released 2016.05.22)
-- is around 25% faster than [**Catch**](https://github.com/philsquared/Catch) when using expression decomposing ```CHECK(a==b)``` asserts
-- adds asserts of the form ```CHECK_EQ(a,b)``` with no expression decomposition - around 20% faster than ```CHECK(a==b)```
-- adds fast asserts like ```FAST_CHECK_EQ(a,b)``` with no ```try/catch``` blocks - around 30-70% faster than ```CHECK_EQ(a,b)```
-- adds the [**```DOCTEST_CONFIG_SUPER_FAST_ASSERTS```**](configuration.html#doctest_config_super_fast_asserts) identifier which makes the fast assertions even faster by another 35-80%
+- is around 30% faster than [**Catch**](https://github.com/philsquared/Catch) when using normal expression decomposing ```CHECK(a==b)``` asserts
+- asserts of the form ```CHECK_EQ(a,b)``` with no expression decomposition - around 25%-45% faster than ```CHECK(a==b)```
+- fast asserts like ```FAST_CHECK_EQ(a,b)``` with no ```try/catch``` blocks - around 60-80% faster than ```CHECK_EQ(a,b)```
+- the [**```DOCTEST_CONFIG_SUPER_FAST_ASSERTS```**](configuration.html#doctest_config_super_fast_asserts) identifier which makes the fast assertions even faster by another 50-80%
- using the [**```DOCTEST_CONFIG_DISABLE```**](configuration.html#doctest_config_disable) identifier the assertions just disappear as if they were never written
+[**Catch**](https://github.com/philsquared/Catch):
+
+- using [**```CATCH_CONFIG_FAST_COMPILE```**](https://github.com/philsquared/Catch/blob/master/docs/configuration.html#catch_config_fast_compile) results in 10%-40% faster build times for asserts.
+
+## Runtime benchmarks
+
+The runtime benchmarks consist of a single test case with a loop of 10 million iterations performing the task - a single normal assert (using expression decomposition) or the assert + the logging of the loop iterator ```i```:
+
+```
+for(int i = 0; i < 10000000; ++i)
+ CHECK(i == i);
+```
+
+or
+
+```
+for(int i = 0; i < 10000000; ++i) {
+ INFO(i);
+ CHECK(i == i);
+}
+```
+
+Note that the assert always passes - the goal should be to optimize for the common case - lots of passing test cases and a few that maybe fail.
+
+| doctest | assert | + info |
+|---------------------|---------|---------|
+| MSVC Debug | 5.29 | 14.32 |
+| MSVC Release | 0.77 | 1.62 |
+| MinGW GCC Debug | 2.25 | 4.71 |
+| MinGW GCC Release | 0.39 | 0.90 |
+| Linux GCC Debug | 2.84 | 5.13 |
+| Linux GCC Release | 0.30 | 0.69 |
+| Linux Clang Debug | 2.47 | 5.02 |
+| Linux Clang Release | 0.41 | 0.75 |
+
+| Catch | assert | + info |
+|---------------------|---------|---------|
+| MSVC Debug | 365.37 | 621.78 |
+| MSVC Release | 7.04 | 18.64 |
+| MinGW GCC Debug | 9.22 | 21.89 |
+| MinGW GCC Release | 7.29 | 13.95 |
+| Linux GCC Debug | 11.17 | 24.79 |
+| Linux GCC Release | 6.45 | 12.68 |
+| Linux Clang Debug | 10.40 | 22.64 |
+| Linux Clang Release | 5.81 | 13.83 |
+
+<img src="../../scripts/data/benchmarks/runtime_info.png" width="430" align="right">
+<img src="../../scripts/data/benchmarks/runtime_assert.png" width="430">
+
+### Conclusion
+
+**doctest** is significantly faster - between 4 and 40 times.
+
+In these particular cases **doctest** makes 0 allocations when the assert doesn't fail - it uses lazy stringification (meaning it stringifies the expression or the logged loop counter only if it has to) and a small-buffer optimized string class to achieve these results.
+
----------
If you want a benchmark that is not synthetic - check out [**this blog post**](http://baptiste-wicht.com/posts/2016/09/blazing-fast-unit-test-compilation-with-doctest-11.html) of [**Baptiste Wicht**](https://github.com/wichtounet) who tested the compile times of the asserts in the 1.1 release with his [**Expression Templates Library**](https://github.com/wichtounet/etl)!