Macros? Where we're going, we don't need macros

It’s been a bit longer than I intended since my last post on the development of Catch23. But that doesn’t mean it has stalled. In this post I want to take a little diversion, since I know that the committee are currently considering use cases for Reflection - and one that’s on their radar is test frameworks!

There are a number of ways that reflection may impact the implementation - and API - of test frameworks. Most of them involve reducing or, ideally, eliminating the use of macros.

I actually wrote about this 12 years ago. Of course we didn’t have reflection then - or an expectation of getting it any time soon. But I did walk through the various obstacles to having an ergonomic framework without macros. My conclusion then was that there were two major obstacles: capturing file/ line info and catching unexpected exceptions.

Of course, since C++20 we now have std::source_location - and that completely addresses the first gap. The second one is still there, though, and I don’t think anything currently proposed for reflection addresses it.

Macro free - with one exception

Originally, Catch23 was designed to support a macro-free syntax. This is an extract from the readme a year ago showing what was then supported:

int const two = 2;
check() << 1 == two;

using namespace CatchKit::Matchers;
check() << "hello", starts_with("hell") && ends_with("lo");
require.handle_unexpected_exceptions( [&]{ require("this_throws()") << this_throws(); } );

check() introduced an object that, among other things, captured the file and line number. It overloaded << to kick off expression decomposition in much the same way that Catch/Catch2 always did within its macros (the decomposition, itself, looks a bit different now - being able to take full advantage of concepts and if constexpr). That decomposition had a special branch on the comma operator, so we could do matchers the same way. That was probably an unnecessary step - having check_that( expr, matcher ) would have been simpler and cleaner.

This would have all been fine but for that last line. In order to trap unexpected exceptions at the point of the assertion we needed to wrap the whole thing in a lambda and pass that to a function. Maybe require.handle_unexpected_exceptions was a little punitive in its length. It could certainly be made shorter - but the problem was that to get the expression decomposition you still had to kick off another check() or require() inside the lambda. Furthermore, you needed to pass the expression as a string, too, so it could be printed.

Perhaps this is where reflection might be able to help? If we can do expression decomposition using reflection, instead of needing an object with an operator to introduce the expression templates for decomposition, maybe this gets simple enough that it can at least be entertained? Perhaps something like:

require( [&]{ this_throws(); } );

I can see that some people would tolerate that, if it meant they could go macro-free. Personally I think:

REQUIRE( this_throws() );

still wins. But at least we can have a conversation. At that point the macro becomes a very light wrapper for the direct call.

But I’m not sure if even what is being proposed for C++29 would give us the necessary introspection to be able to look into the argument, interpret it as a lambda, look inside the lambda, and be able to decompose the expression in a way that the values could be interpreted independently - while keeping it in the lambda, so the calling code (i.e. the framework) could wrap the whole thing in a try-catch.

If it could then we might be getting somewhere. If not, then I’m not sure we get much further along than we already are with C++23 - or at least C++26.

Going back to Catch23, I eventually ditched the macro-free design. I was trading off too much for too little gain - and it was adding a lot of extra complexity and constraints on the internals. My thinking was that I would reconsider in the context of reflection - and I may still do that. But I think the answers to the preceding questions are central to that.

Where reflection really does help

The Catch family has had expression decomposition from the start, of course. As we’ve seen, above, we can even do an acceptable job of retaining that in a macro-free implementation. But it could certainly be cleaner with reflection. So that would be one advantage.

Converting enums and type names to strings is another area that, historically, has been waiting for reflection to solve cleanly. That is still the case, but Catch23 stopped waiting. Internally, I have implemented a best-effort approach that extracts the names from source_location strings. It’s a bit brittle and has some limits (and probably a noticeable compile-time cost - not measured at time of writing) - so I still intend to conditionally use reflection, if available. But it’s no longer the big quality-of-life improvement that it was always going to be.

So, where do we go from here?

I still look forward to a time that I can write tests in C++ without using macros - and without compromise. We’re tantalisingly close, but I fear the last 10% of the way may take 90% of the time.

In the meantime I continue to work on Catch23 and will have more content soon.