| onqtam | 8126b56 | 2016-05-27 17:01:15 +0300 | [diff] [blame] | 1 | <!DOCTYPE html> |
| 2 | <html> |
| 3 | <title>roadmap</title> |
| 4 | <xmp theme="united" style="display:none;"> |
| 5 | |
| 6 | ## Roadmap |
| 7 | |
| onqtam | e34600e | 2016-11-15 02:16:56 +0200 | [diff] [blame] | 8 | This library is free, and will stay free but needs your support to sustain its development. There are lots of [**new features**](roadmap.html) and maintenance to do. If you work for a company using **doctest** or have the means to do so, please consider financial support. |
| onqtam | 8126b56 | 2016-05-27 17:01:15 +0300 | [diff] [blame] | 9 | |
| onqtam | 1435c01 | 2016-09-21 15:29:11 +0300 | [diff] [blame] | 10 | [](http://www.patreon.com/onqtam) |
| onqtam | 6a5da42 | 2017-09-04 17:56:08 +0300 | [diff] [blame] | 11 | [](https://www.paypal.me/onqtam/10) |
| onqtam | 8126b56 | 2016-05-27 17:01:15 +0300 | [diff] [blame] | 12 | |
| onqtam | b8220c5 | 2017-05-16 00:21:15 +0300 | [diff] [blame] | 13 | Planned features for future releases - order changes constantly... |
| onqtam | 1435c01 | 2016-09-21 15:29:11 +0300 | [diff] [blame] | 14 | |
| onqtam | b8220c5 | 2017-05-16 00:21:15 +0300 | [diff] [blame] | 15 | ### For 1.3: |
| 16 | |
| 17 | - move from printf to using streams in the current console reporter and clean that mess |
| onqtam | 8126b56 | 2016-05-27 17:01:15 +0300 | [diff] [blame] | 18 | - reporters |
| onqtam | b8220c5 | 2017-05-16 00:21:15 +0300 | [diff] [blame] | 19 | - output to file |
| onqtam | 8126b56 | 2016-05-27 17:01:15 +0300 | [diff] [blame] | 20 | - a system for writing custom reporters |
| 21 | - ability to use multiple reporters at once (but only 1 to stdout) |
| 22 | - a compact reporter |
| onqtam | b8220c5 | 2017-05-16 00:21:15 +0300 | [diff] [blame] | 23 | - a progress reporter - or maybe just an option for the console reporter |
| onqtam | 8126b56 | 2016-05-27 17:01:15 +0300 | [diff] [blame] | 24 | - an xml reporter |
| onqtam | b8220c5 | 2017-05-16 00:21:15 +0300 | [diff] [blame] | 25 | - xUnit reporter |
| 26 | - a listener interface - similar to a reporter - look at Catch |
| onqtam | 1435c01 | 2016-09-21 15:29:11 +0300 | [diff] [blame] | 27 | - ability to have no output when everything succeeds |
| onqtam | b8220c5 | 2017-05-16 00:21:15 +0300 | [diff] [blame] | 28 | - option to output summary only |
| 29 | - log levels - like in [boost test](http://www.boost.org/doc/libs/1_63_0/libs/test/doc/html/boost_test/utf_reference/rt_param_reference/log_level.html) |
| 30 | - matchers - should investigate what they are - look at google test/mock and Catch (also predicates and boost test) |
| 31 | - convolution support for the assertion macros (with a predicate) |
| 32 | - Value-Parameterized test cases |
| 33 | - generators? - look at Catch - and investigate what they are |
| onqtam | 6a5da42 | 2017-09-04 17:56:08 +0300 | [diff] [blame] | 34 | - proper conan package |
| onqtam | 8cec917 | 2018-02-06 23:34:14 +0200 | [diff] [blame^] | 35 | - https://github.com/onqtam/doctest/issues/103 |
| 36 | - https://github.com/bincrafters/conan-doctest |
| onqtam | 6a5da42 | 2017-09-04 17:56:08 +0300 | [diff] [blame] | 37 | - https://github.com/uilianries/conan-catch |
| 38 | - https://github.com/philsquared/Catch/pull/938 |
| 39 | - https://stackoverflow.com/questions/40311363/conan-io-header-only-package |
| 40 | - https://github.com/conan-io/docs/issues/67 |
| 41 | - https://github.com/agauniyal/rang |
| 42 | - https://github.com/conan-io/conan/issues/324 |
| onqtam | b8220c5 | 2017-05-16 00:21:15 +0300 | [diff] [blame] | 43 | - look at property based testing |
| 44 | - [rapidcheck](https://github.com/emil-e/rapidcheck) |
| 45 | - [autocheck](https://github.com/thejohnfreeman/autocheck) |
| 46 | - [CppQuickCheck](https://github.com/grogers0/CppQuickCheck) |
| onqtam | 6ff3d35 | 2017-09-11 14:35:06 +0300 | [diff] [blame] | 47 | - support for LibIdentify |
| onqtam | b8220c5 | 2017-05-16 00:21:15 +0300 | [diff] [blame] | 48 | - IDE integration |
| 49 | - https://blogs.msdn.microsoft.com/vcblog/2017/05/10/unit-testing-and-the-future-announcing-the-test-adapter-for-google-test/ |
| 50 | - https://www.reddit.com/r/cpp/comments/65c0f1/run_cpp_unit_tests_from_xcode_and_visual_studio/ |
| 51 | - https://github.com/k-brac/CUTI |
| onqtam | 6a5da42 | 2017-09-04 17:56:08 +0300 | [diff] [blame] | 52 | - https://github.com/csoltenborn/GoogleTestAdapter |
| onqtam | b8220c5 | 2017-05-16 00:21:15 +0300 | [diff] [blame] | 53 | - MSTest |
| 54 | - http://accu.org/index.php/journals/1851 |
| 55 | - https://msdn.microsoft.com/en-us/library/hh270865.aspx |
| 56 | - https://msdn.microsoft.com/en-us/library/hh598953.aspx |
| 57 | - https://blogs.msdn.microsoft.com/vcblog/2017/04/19/cpp-testing-in-visual-studio/ |
| 58 | - https://msdn.microsoft.com/en-us/library/hh419385.aspx |
| 59 | - XCode - https://github.com/philsquared/Catch/pull/454 |
| 60 | - CLion |
| 61 | - https://www.jetbrains.com/clion/features/unit-testing.html |
| 62 | - https://blog.jetbrains.com/clion/2017/03/clion-2017-1-released/#catch |
| 63 | |
| 64 | ### For 1.4: |
| 65 | |
| 66 | - running tests a [few times](https://github.com/google/googletest/blob/master/googletest/docs/AdvancedGuide.html#repeating-the-tests) |
| 67 | - test execution in [separate processes](https://github.com/philsquared/Catch/issues/853) - ```fork()``` for UNIX and [this](https://github.com/nemequ/munit/issues/2) for Windows |
| 68 | - killing a test that exceeds a time limit (will perhaps require threading or processes) |
| 69 | - [symbolizer](https://github.com/facebook/folly/tree/master/folly/experimental/symbolizer) - for a stack trace - when an assertion fails - and it's in a user function with some deep callstack away from the current test case - how to know the exact code path that lead to the failing assert |
| 70 | - ability to make the framework not capture unexpected exceptions - as requested [here](https://github.com/onqtam/doctest/issues/12#issuecomment-235334585) |
| 71 | - add Approx ability to compare with absolute epsilon - [Catch PR](https://github.com/philsquared/Catch/pull/538) |
| onqtam | 8cec917 | 2018-02-06 23:34:14 +0200 | [diff] [blame^] | 72 | - ability to customize the colors in the console output (may also use styles - based on [this](https://github.com/agauniyal/rang) or [this](https://github.com/ikalnytskyi/termcolor)) |
| onqtam | b8220c5 | 2017-05-16 00:21:15 +0300 | [diff] [blame] | 73 | - implement breaking into the debugger under linux - see [here](https://github.com/philsquared/Catch/pull/585) and [here](https://github.com/scottt/debugbreak) |
| 74 | - better testing of the library |
| 75 | - unit test the String class |
| 76 | - should unit test internals - currently even if a bug is caught by different output it's very difficult to track the reason |
| 77 | - should test stuff that should not compile |
| onqtam | 6a5da42 | 2017-09-04 17:56:08 +0300 | [diff] [blame] | 78 | - https://github.com/ldionne/dyno/blob/master/cmake/CompileFailTest.cmake |
| 79 | - see slide 38 here - https://github.com/boostcon/cppnow_presentations_2017/blob/master/05-19-2017_friday/effective_cmake__daniel_pfeifer__cppnow_05-19-2017.pdf |
| onqtam | b8220c5 | 2017-05-16 00:21:15 +0300 | [diff] [blame] | 80 | - should test crash handling |
| 81 | - should test more config options |
| onqtam | c80eed1 | 2017-10-06 11:01:05 +0300 | [diff] [blame] | 82 | - don't cheat for maxing out code coverage (see [coverage_maxout.cpp](../../examples/all_features/coverage_maxout.cpp)) |
| onqtam | b8220c5 | 2017-05-16 00:21:15 +0300 | [diff] [blame] | 83 | - should test C++11 stuff - perhaps inspect the CMAKE_CXX_FLAGS for -std=c++11 on the CI and add more targets/tests |
| 84 | - test tricky stuff like expressions with commas in asserts |
| 85 | |
| 86 | ### For 2.0: |
| 87 | |
| 88 | - remove C++98 support |
| 89 | - remove the config identifiers for C++11 features |
| 90 | - use variadic templates where appropriate |
| 91 | - update type lists to C++11 |
| 92 | - update traits - use declval, etc. |
| 93 | - move initialization of fields from initializer lists to class bodies |
| 94 | - update static code analysis - less warning suppressing |
| 95 | |
| 96 | ### For 3.0: |
| 97 | |
| 98 | - use modules - use ```std::string``` and whatever else comes from the standard - no more hand rolled traits and classes |
| 99 | - minimize the use of the preprocessor |
| 100 | |
| 101 | ### Things that are being considered but not part of the roadmap yet: |
| 102 | |
| 103 | - FakeIt mocking integration - like [catch](https://github.com/eranpeer/FakeIt/tree/master/config/catch) |
| onqtam | 6a5da42 | 2017-09-04 17:56:08 +0300 | [diff] [blame] | 104 | - look into https://github.com/cpp-testing/GUnit - https://www.youtube.com/watch?v=NVrZjT5lW5o |
| onqtam | b8220c5 | 2017-05-16 00:21:15 +0300 | [diff] [blame] | 105 | - consider the following 2 properties for the MSVC static code analyzer: EnableCppCoreCheck, EnableExperimentalCppCoreCheck |
| 106 | - rpm package? like this: https://github.com/vietjtnguyen/argagg/blob/master/packaging/rpm/argagg.spec |
| 107 | - get the current test case/section path - https://github.com/philsquared/Catch/issues/522 |
| 108 | - when no assertion is encountered in a test case it should fail |
| 109 | - failure reporting should print out previous SECTIONs for data-driven testing - as requested [here](https://github.com/philsquared/Catch/issues/734) |
| 110 | - ```Bitwise()``` class that has overloaded operators for comparison - to be used to check objects bitwise against each other |
| 111 | - detect floating point exceptions |
| 112 | - checkpoint/passpoint - like in [boost test](http://www.boost.org/doc/libs/1_63_0/libs/test/doc/html/boost_test/test_output/test_tools_support_for_logging/checkpoints.html) (also make all assert/subcase/logging macros to act as passpoints and print the last one on crashes or exceptions) |
| 113 | - queries for the current test case - name (and probably decorators) |
| 114 | - thread safety - asserts/subcases/captures should be safe to be used by multiple threads simultaneously |
| onqtam | 8cec917 | 2018-02-06 23:34:14 +0200 | [diff] [blame^] | 115 | - https://github.com/blastrock/doctest/tree/threadsafe |
| onqtam | 1435c01 | 2016-09-21 15:29:11 +0300 | [diff] [blame] | 116 | - support for running tests in parallel in multiple threads |
| onqtam | b8220c5 | 2017-05-16 00:21:15 +0300 | [diff] [blame] | 117 | - death tests - as in [google test](https://github.com/google/googletest/blob/master/googletest/docs/AdvancedGuide.html#death-tests) |
| 118 | - config options |
| 119 | - test case name uniqueness - reject the ones with identical names |
| 120 | - command line options |
| 121 | - ability to specify ASC/DESC for the order option |
| 122 | - global timeout option (per test or per entire session?) |
| 123 | - command line error handling/reporting |
| onqtam | 6a5da42 | 2017-09-04 17:56:08 +0300 | [diff] [blame] | 124 | - option to not print context info when the --success option is used |
| onqtam | b8220c5 | 2017-05-16 00:21:15 +0300 | [diff] [blame] | 125 | - ability for the user to extend the command line - as requested [here](https://github.com/philsquared/Catch/issues/622) |
| 126 | - option to list files in which there are test cases who match the current filters |
| 127 | - option for filters to switch from "match any" to "match all" mode |
| 128 | - option to list test suites and test cases in a tree view |
| 129 | - add a "wait key" option - as requested [here](https://github.com/philsquared/Catch/issues/477#issuecomment-256417686) |
| 130 | - decorators for test cases and test suites- like in boost test |
| 131 | - depends_on |
| 132 | - precondition |
| 133 | - fixture |
| 134 | - label (tag) - with the ability to have multiple labels (tags) for a test case and also the ability to list them |
| 135 | - run X times (should also multiply with (or just override) the global test run times) |
| 136 | - throw an exception when incompatible decorators are given in the same list of decorators - like may_fail and should_fail |
| 137 | - setup / teardown support |
| 138 | - global setup / teardown - can be currently achieved by providing a custom main function |
| 139 | - per test suite (block? only? and not all blocks of the same test suite?) |
| 140 | - as decorators |
| 141 | - see how it's done in boost test - with the fixture decorator |
| 142 | - perhaps for fixtures in addition to the constructor / destructor - since throwing in the destructor might terminate the program |
| 143 | - or just ignore all of this this - it would require globals or classes and inheritance - and we already have subcases |
| 144 | - doctest in a GUI environment? with no console? APIs for attaching a console? querying if there is one? [investigate...](https://github.com/philsquared/Catch/blob/master/docs/configuration.html#stdout) |
| 145 | - runtime performance |
| 146 | - startup - the set holding all registered tests should use a specialized allocator to minimize program startup time |
| 147 | - failing - optimize createStream/freeStream to reuse a pool of ostringstream objects |
| 148 | - ability to provide a temp folder that is cleared between each test case |
| 149 | - make the _MESSAGE assert macros work with variadic arguments - and maybe write the ones for binary/unary/fast asserts as well |
| 150 | - move from operator "<<" to "<=" for capturing the left operand when decomposing binary expressions with templates |
| 151 | - think about silencing warnings about unused variables when DOCTEST_CONFIG_DISABLE is used - see commit 6b61e8aa3818c5ea100cedc1bb48a60ea10df6e8 or issue #61 |
| 152 | - think about optionally using ```<typeinfo>``` and libcxxabi for demangling so users don't have to use ```TYPE_TO_STRING()``` |
| 153 | - handle more complex expressions - ```CHECK(foo() == 1 || bar() == 2);``` |
| 154 | - think about using a string view of some sorts |
| 155 | - benchmark against google test and boost test |
| onqtam | 1435c01 | 2016-09-21 15:29:11 +0300 | [diff] [blame] | 156 | |
| onqtam | b8220c5 | 2017-05-16 00:21:15 +0300 | [diff] [blame] | 157 | ### Things that are very unlikely to enter the roadmap: |
| onqtam | 1435c01 | 2016-09-21 15:29:11 +0300 | [diff] [blame] | 158 | |
| onqtam | b8220c5 | 2017-05-16 00:21:15 +0300 | [diff] [blame] | 159 | - think about removing the binary asserts (leaving only the fast binary asserts) because normal asserts + no try/catch in asserts are almost the same |
| 160 | - move the "react()" part (the one that throws for REQUIRE asserts - or for when "abort-after=<int>" is reached) to a function call in the while() part of the asserts |
| 161 | - stop using underscores for the begining of identifiers - the anonymous variables - against the standard... |
| 162 | - templated fixture test cases |
| 163 | - test with missed warning flags for GCC |
| 164 | - https://github.com/Barro/compiler-warnings |
| 165 | - http://stackoverflow.com/a/34971392/3162383 |
| 166 | - utf8 / unicode ??? |
| 167 | - https://github.com/philsquared/Catch/pull/903 |
| onqtam | 1435c01 | 2016-09-21 15:29:11 +0300 | [diff] [blame] | 168 | - handle ```wchar``` strings??? |
| onqtam | b8220c5 | 2017-05-16 00:21:15 +0300 | [diff] [blame] | 169 | - hierarchical test suites - using a stack for the pushed ones |
| 170 | - ability to specify the width of the terminal in terms of characters (for example 60 - less than 80 - the default) |
| onqtam | 8126b56 | 2016-05-27 17:01:15 +0300 | [diff] [blame] | 171 | - ability to re-run only newly compiled tests based on time stamps using ```__DATE__``` and ```__TIME__``` - stored in some file |
| onqtam | 1435c01 | 2016-09-21 15:29:11 +0300 | [diff] [blame] | 172 | - add underscores to all preprocessor identifiers not intended for use by the user |
| 173 | - put everything from the ```detail``` namespace also in a nested anonymous namespace to make them with internal linkage |
| 174 | - ability to put everything from doctest into an anonymous namespace - to allow the use of multiple different versions of **doctest** within the same binary (executable/dll) - like the [**stb**](https://github.com/nothings/stb) libraries can |
| onqtam | 8126b56 | 2016-05-27 17:01:15 +0300 | [diff] [blame] | 175 | |
| 176 | --------------- |
| 177 | |
| 178 | [Home](readme.html#reference) |
| 179 | |
| 180 | |
| 181 | </xmp> |
| 182 | <script src="strapdown.js/strapdown.js"></script> |
| 183 | </html> |