At the QA Hackathon (http://qa-hackathon.org/) I helped Michael G Schwern (http://use.perl.org/~schwern/journal/) get over his metaphorical hump with the design of the new Test::Builder2. I was one of the programmers without an agenda at the hackathon there just to try things. A disinterested+ party, I pitched in (at least a bit) when a bit of feedback and programming was called for.
To explain what we tried to do and why we were doing it I’ll explain what the Test::Builder is first. The Test::Builder is the library used under the hood by various testing libraries including Test::Simple and Test::More to produce the TAP output and manage the test numbers so that they don’t end up out of line. It also does a few other convenient things but those were the core bits we were focusing on. To do that it’s a singleton and has a bunch of helper methods for those libraries that use it. While this Builder library may change, the existing test libraries aren’t likely to change, except perhaps to add new facilities.
Schwern had some goals with the new version,
- Make the output configurable and replaceable (then you wouldn’t be tied to TAP)
- To make it so that you can tell whether you are currently passing the tests (so that you can tag on extra debug output at the end if you’re not)
- To allow extra information like diagnostics to be tied to tests more easily
The second item, the telling whether you were currently passing your tests was something he’d already done with a history object.
The configurable output was a relatively simple thing to do. For now we’ve created a class that implements a few basic output messages for start, end and test result and set the test builder to be created with a default implementation that outputs TAP. Schwern is now looking at other testing frameworks to see if other messages are likely to be needed. His look at the POSIX testing framework for example suggested there may need to be some additional start/end messages at different levels.
Allowing extra information to be associated with the tests is a little trickier. Schwern had already figured out that the way to do it was to use a result object so that you had something to tack the extra information to. This way rather than simply returning a pass or fail the test methods will return a ‘Result’ object. You can then tack on extra diagnostics or mark the fact that it’s a todo.
In the simplest case to mark an object with some diagnostics,
ok($func('a'), 'some test')->diag('some cheap diagnostics');
Or if the diagnostics are expensive to run and you only want to run them when the test fails,
{
my $result = ok $func('a'), 'A test';
if(!$result) {
$result->diag("expensive diagnostics we'd rather not run");
}
}
There are a couple of things to note in this instance. The first is that you can do a true/false comparison on the object to see if it’s passed or failed. One delegate described this as the ‘truthiness’ of the object. This way we can check easily whether it’s a pass or fail without checking the ‘is_fail’ property (as it was named at the time I was writing this).
The other thing to note are the scope blocks. They are probably good practice around tests in general but in this case they are essential. The way that the builder knows when to display the output is by only doing so when you are finished with the object, i.e. by waiting for the object to be destroyed. Without those scope blocks the destructor wouldn’t be called in the appropriate place. This design decision seems like the best way to give you a simple interface while allowing you to deal with the more complex scenario’s without too much difficulty. You can of course make a mess of it, but that’s programming for you.
Another bit of ugliness is that the result isn’t actually the real result but a wrapper that implements the destructor functionality and pretends to be the result. That’s because the result is a pure data object and because it’s also stored in the history and so wouldn’t naturally be destructed at the correct point just because your code doesn’t reference it at the test any more. The wrapper just acts as a facade on the result and is given the output object by the builder so that it can output the test result when the wrapper is destroyed.
The syntax also allows you to set a single test as a TODO very easily.
ok($func('a'), 'some test')->todo('still need to implement this');
You can chain multiple pieces of data together and each setter returns the result back so that you can keep doing it. It’s also possible to set a todo without a message. The net effect of that is that it’s not possible to use the todo call to tell whether or not the result is a todo or what the reason is. Sounds obvious when I say it like that but it was something we had to think about! Instead there is a ‘is_todo’ property and a ‘reason’ property for those pieces of information.
Note that setting the information like that doesn’t mean that you cannot use the existing libraries features like Test::More’s TODO blocks. They can still be used for multiple tests where they are appropriate; it’s just that now you can also do quick todo’s with minimal fuss too.
The weekend was spent figuring out how to meet these goals with an implementation that was reasonably clean. Schwern had already figured out the basic mechanics of it, we just thrashed out what worked practically and what didn’t. After a few rewrites of parts of the structure it’s starting to look like something useful. What I’ve mentioned above was roughly where we were by the end of the weekend. Whether it still looks like that now is another matter. Of course how much you’ll actually see of these changes I don’t know, this library is used within most of the other test libraries so most changes will be invisible to the users.
+ disinterested in the sense that I’ve not got a vested interest in the development of the QA or testing code. As far as I’m concerned it already works great and I’m really grateful for it.
