Exploration Through Example

Earlier in the series:
Controller channels

“Omit needless words” is number 13 of Strunk&White’s elementary rules for composing English. It’s a pretty good rule for composing tests, too, but not followed often enough. Let’s rewrite an example of the sort of test I often see.

Here’s the sample test. I’m switching back from Ruby to Fit because Fit allows more concision. More on that later.

Which of the columns in the first table are actually required? I’ll cross out the ones I don’t like:

All the test needs is some flight on some date for which only increases are allowed. We don’t care which flight it is, much less what all the flights covered by a particular contract are. Since the rule is about the number of seats allocated, we don’t even care about which seats the travel agency has. The test would be clearer if it just said “6 seats” instead of listing them. That way, a reader wouldn’t have to count seats to check that the test is right.

Why does this setup table have so much extra information? I usually see one of two reasons:

1. The programmers write the fixtures (code that converts the test input into a form the application understands). They’ve written a fixture for creating contracts. They don’t want to create a new fixture when an old one can be made to do.

2. The testers know contracts are stored in the CNTRCT table in the database. Since they know what that table looks like, they’ve modeled their setup table after it.

Neither of those are good reasons.

1. The programmers wouldn’t put up with one-and-only-one way to create an object in the “real” code. (At least, they shouldn’t.) Why not? Suppose they’re writing some code that needs one of those objects. They don’t want to have to understand and create secondary data that has nothing to do with their present purposes. They don’t want another programmer reading the code later to have to wonder whether that secondary data is relevant. And they know there’s a greater chance that changes to the way the object is created will ripple out to affect lots of code.

   The same is true of tests. Who wants to have to get the date format right when dates don’t matter? Who wants to wonder, later, whether dates are somehow important to this test? Who wants to worry about all the tests when contracts are given a new field?

   The right approach in this situation, I think, is not to be so dependent on the programmers. In the Ruby-style tests I’ve been advocating, a tester who knows scripting will be able to write the fixtures himself, just as a programmer would write a new “constructor” to suit her needs.

2. Business-facing tests are supposed to be about the business rules, not about unfortunate details like database tables. There are probably plenty of tricky cases in the business rules. Tests with lots of irrelevant detail make it harder to search for edge cases systematically or notice later that they were overlooked.

So my inclination would be to omit words other than “increase the allocation”, “current allocation”, and “6″.

That’s not to say there’s not danger in terseness. Writing out the detailed test encouraged me to ask questions:

• Is an increase for all the dates or for one particular date? What happens if an airline wants to give an extra five seats for all the dates covered by a contract? Does it increase each date separately?

• Does the airline say “take five more seats” or “take these particular five more seats”? If the latter, what happens if the travel agency has seats 1A and 1B, then the airline says “take seats 21A, 21B, and 21C”? That’s an increase in total number of seats, but it’s taking away seats that the travel agency might be in the process of selling.

There’s a conundrum here. More detail makes it easier to be imaginative; less detail makes it easier to be systematic. You need both: but not necessarily at the same time, not necessarily in the same test. I’m not saying you should rigidly defer all detail. It’s good to ask questions as they come up, since the answers may change your understanding and make you rewrite tests. But it’s also OK to hold questions until later—and, as we’ll see, the same questions will likely come up during test implementation. (That’s another advantage of having the same person who’s thinking about test design implementing the tests: because the person is wearing two hats at once, there’s a greater chance that an implementation issue will flip her into thinking about what the app should really be doing.)

One reason I don’t mind deferring questions of detail is that I don’t care so much if a test is wrong, so long as it’s useful. Any given test is one snippet of dialog in a long conversation during which the business teaches the team to grasp the domain in at least an instrumental way. As with most kinds of teaching, it helps to simplify—even oversimplify, especially at first. Once some understanding has been gained, you cycle back to revisit old truths, to correct or refine them. The whole point of Agile is to reduce the cost of being wrong, and you might as well take advantage of that.

Now let’s return to the second part of the test, here:

This, too, is a kind of verboseness I see often. In this case, we have two subtests: one that checks that decreases are not allowed, and one that checks that increases are. In both cases, the same thing is done, just to different input data. Column formats are perfect for that:

The first table is perhaps a more natural way of thinking about the test, since it mimics what you do when running the test by hand: do something, check something, do something, check something. It’s a little more effort to convert those thoughts into a column format, but I think it’s worth it because it makes it easy to see what a test is testing and systematically compare it to similar, nearby tests.

I would take the next step and replace specific seat names with counts:

(Notice that I used different starting values in the two rows. I do that because, first, I don’t want to ever look at debugging output, see a “6″, and wonder which row it came from. Second, I like to introduce variety whenever it’s inexpensive. There’s a slight chance it will stumble over a bug by sheer dumb luck, or make me think of a case I’d overlooked.)

That’s the final test, including setup. I’ll let you be the judge: would you rather read it or the original?

A needless word in a test is one that can be omitted without making the test harder to understand. Omit needless words. I think the previous test has omitted all needless words. (I suppose you could argue that “the” and “our” should also be omitted, but there must be a reason you and I outgrew telegraphic speech around age three.)

We have to be a bit careful, though, because the cost of dropping a word from an English sentence is low, but doing the same for a test word can be high. As we’ll see, clarity of expression requires more test support code. For example, I tentatively prefer writing tabular tests in Ruby instead of HTML. Here’s the Ruby version of the above:

In addition to the noise symbols like quotes and commas and def and header, I had to repeat a statement that the airline was allowed to increase. That redundancy is bad. (It’s not unusual for me to change the body of the test in a way that makes the test name incorrect, but forget to change the test name.)

Nevertheless, I think the advantages of Ruby outweigh the way it makes tests harder to read. I’m buying implementation power with test clutter.

Other than that exception, I prefer to write tests in the most expressive way I can. If the implementation turns out to be too hard, I might backtrack. I have a fairly high threshold for “too hard” for reasons I’ll give later in the series.