Catch 1.0
Since Catch first went public, two and a half years ago, at time of this writing, I've made a point of describing it as a "developer preview". Think of it as you might a Google beta and you won't go far wrong. I did this because I knew that there was a lot that needed doing - and in particular that some of the public interfaces would be subject to change. While I have tried to mitigate exposure to this as much as possible (as we'll see) I had wanted to reach a point that I could say things have stabilised and I'm happy to call it a true 1.0 release.
That time has come.
As of today the version of Catch available on the Master branch on GitHub is 1.0 and I would encourage you to update to it if you're already using an older version. But before you do so please read the rest of this post as there are a few breaking changes that may impact you
What's new?
Output
One of the biggest changes is in the console reporter's output. This has been rethought from the ground up (perhaps more accurately: it has now been thought through at all). For example a failure now looks like this:
ClassTests.cpp:28: FAILED:
REQUIRE( s == "world" )
with expansion:
"hello" == "world"
That indentation is applied after a wrap too, so long lines of output are much more clearly separated from the surrounding context. This use of indentation has been used throughout.
But there's a lot more to the new look. You'll just have to try it for yourself
Naming and tags
One of the features of Catch since day one has been the ability to name test cases (and sections) using free-form strings. Except that I then went and imposed a convention on those names so they should be hierarchical. You didn't have to follow the convention but if you did you got the ability to group related tests together in a similar manner to related files in a folder in a file system. When combined with wild-cards this gave a lot of power.
The trouble was test names needed to be short and simple otherwise they got very long. So I felt the need to have a second argument where you could supply a longer form description. Of course this was rarely used (even by me!) and so you'd see a lot of this:
TEST_CASE( "project/widgets/foo/bar/size", "" ) { /*..*/ }
The name doesn't really tell you what the test does and the description (which should have) is unused but must be supplied anyway so is just an ugly empty string.
This was not what I signed up for!
Now there is a better way.
It has all the advantages of the old system, but none of the disadvantages - and all without breaking backwards compatibility - so you won't have to go back and rewrite all your existing test cases. Phew!
Test cases can now be tagged. Tags are placed in the second argument (that description argument that nobody was using) and are each enclosed in square brackets. Anything outside of square brackets are still considered the description - although that use is now deprecated. Tags fulfil the same role (and more) as the old hierarchical names, so the name field is now freed up to be more descriptive. The previous example might now look like:
TEST_CASE( "the size changes when the bar grows", "[widgets][foo][bar]" )
{ /*..*/ }
But now you can run all tests with the [widgets] tag. Or with the [foo] tag. Or with the [bar] tag. Or all tests tagged [foo] and [bar] but not [widgets]. Tags are much more powerful.
Variadic macros
But if you don't need tags the second argument is now optional (assuming your compiler supports variadic macros - or, more specifically, Catch knows that it supports them). So TEST_CASEs can be written with one argument - or even none at all (an anonymous test case is given a generated name internally - useful if you're just exploring an idea).
Most, if not all, macros where it makes sense now take advantage of variadic macro support.
If you know that your compiler supports variadic macros, yet Catch is not letting you, please let me know and we'll see if we can add the support in.
On your best behaviour
In my first post on Catch, under "A SECTION on specifications", I talked a little about how, while Catch is not a BDD framework, it supports writing in a BDD style. Of note I said,There is more planned in this area. For example I'm considering offering a GIVEN() macro for defining instances of test data, which can then be logged.Well I've taken this further and you can now write tests using the following form:
SCENARIO( "name for scenario", "[optional tags]" ) {
GIVEN( "some initial state" ) {
// set up initial state
WHEN( "an operation is performed" ) {
// perform operation
THEN( "we arrive at some expected state" ) {
// assert expected state
}
}
}
}
You can have as many peer WHEN and THEN and even GIVEN sections as you like. You can even nest them with AND_WHEN and AND_THEN. In fact all of these macros are (currently) just aliases for SECTION. SCENARIO is an alias for TEST_CASE.
Although I mentioned BDD you do not need to assert on behaviour here. I typically use the THEN block to assert purely on state. Nonetheless I often find the GIVEN-WHEN-THEN structure useful in organising my tests. They also read well in the output. Here's an example straight from the self test suite:
-------------------------------------------------------------------------------
Scenario: Vector resizing affects size and capacity
Given: an empty vector
When: we reserve more space
Then: the capacity is increased but the size remains the same
-------------------------------------------------------------------------------
Meet Clara
Catch has always had rich command line support. The first implementation was very ad-hoc but as it evolved it become more like an embedded library in itself. For this release I have taken this to its logical conclusion and spun the - completely rewritten - command line parser out into its own library. At time of writing this is still part of the Catch code-base, and depends on a couple of other parts of Catch. The intention is to break those dependencies and extract the code into its own repository on GitHub. But what of the zero-dependency ethos of Catch? Don't worry - the new library will follow the same principle of being header-only and embeddable. So a copy will continue to be included in the Catch code-base and Catch will continue to be distributed as a single header file.
A new library needs a new name. Since it's a Command Line ARgument Assigner I felt Clara was a good name.
As a result of this change some of the specific options have changed (details in the "breaking changes" section). This is to accommodate a closer adherence to POSIX guidelines for command line options. All short-form option names are now single characters and those that take no arguments can be combined after a single -. e.g. to combine -s, -a and -b you can now use -sab.
Options with arguments always have arguments (and can only have one). This leads to a couple of interesting consequences: first the separator character between option and argument can be a space or either : or =. Secondly the non-option arguments (test specs) can appear before or after options.
So the following are all equivalent:./CatchSelfTest "test name" -b -x 1
./CatchSelfTest "test name" -b -x:1
./CatchSelfTest -b -x 1 "test name"
./CatchSelfTest -x=1 "test name" -b
What's up, Doc?
The documentation for Catch, such as it was, had been provided in the wiki for the GitHub repos. There were a couple of drawbacks to this - most significantly it meant I couldn't have different documentation for different branches, or earlier versions. I also find it much easier to edit documents offline.
So I've now moved (and updated) all the existing documentation into markdown files in the repository itself. These are in the /docs folder, but the README.md file in the root links into them, acting as a convenient launch point.
Breaking changes
This section is only really of interest if you are an active user of an earlier version of Catch.Under new command
As well as the improvements described there have had to be some changes to the command line options to accommodate them. The list of available options continues to be available by running with the -?, -h or --help switches. They are more fully described in the documentation, now in the repository (rather than the wiki). The in-depth descriptions have been removed from the code.
But here's a quick summary of the changes to be aware of
- Test case specs (names or wild carded patterns) and tags are now only specified as application arguments (previously they were introduced using the -t or -g options). In fact -t now means something different!
- Listing tests, tags or reporters now all have their own options. Previously you used -l for all of them, with an optional argument to disambiguate. -l no longer takes an argument and just means "list tests". Tags are listed with -t (which formerly meant "run with this/ these test case(s)". Listing reporters is less commonly used so has no short-form. They can be listed with --list-reporters
- -nt ("no throw") has become -e (because short form options are single character only)
- -a ? has been split into -a and -x ? (because options may have zero or on arguments - but not both)
Writing your own main()
Catch can provide its own main() function but if you write your own there were a few points you could hook into Catch, with different degrees of control over how it is configured.
This continues to be the case but the interface has completely changed. The new interface is more flexible, safer and better encapsulates things like the clean-up of statically allocated state (important if you do leak-detection).
The new interface is documented in the own-main.md file in the docs folder. It is based around a Session class - which must have exactly one instantiation in your code. However, within the instantiation you can invoke Catch test runs as many times as you like (the Session class encapsulates the config and is responsible for the clean-up of statics - in the future those statics may migrate to the session class itself).
Reporters
Catch has a modular reporting system and comes with three reporters bundled (console, xml and JUnit). You can also supply your own reporter by (previously) implementing the IReporter interface. This was one area that was often being slightly tweaked - and would frequently break implementations of the interface. More often than not any changes need not be used by client code - but they would have to update their interfaces anyway!
To make the reporter interface more robust to change I've created a whole new interface, (IStreamingReporter). Most of the methods of this new interface take structs instead of a list of arguments. Those structs can now change with little to no impact on client code (obviously depending on the changes). They are also richer and provide more information than before so I think we're set for a while now
To ease the transition for anyone who has already implemented IReporter I've provided the INTERNAL_CATCH_REGISTER_LEGACY_REPORTER macro (which wraps your reporter in the LegacyReporterAdapter adapter class).
At time of writing documentation for the new reporter interface is coming
It's not just me
Although I have used the personal pronoun, I, a lot in this post (and I continue to be the benevolent dictator on this project) Catch has greatly benefited from the on-going contributions of others - whether that be through pull-requests, bug reports, feature requests and other suggestions, actively maintained forks or just plain evangelising. All of this has been much appreciated and I hope to grow that even more now we have a stable base. Thanks!
Where to go from here
Catch is hosted on GitHub. The preferred url to follow is catch-lib.net, which redirects there - but may become a landing page in the future (an embryonic version of which is already at builds.catch-lib.net).
There's also a forum on Google Groups.