Exploration Through Example

Example-driven development, Agile testing, context-driven testing, Agile programming, Ruby, and other things of interest to Brian Marick
191.8 167.2 186.2 183.6 184.0 183.2 184.6

Fri, 28 Feb 2003

Does test-driven development lead to inadequate tests?

In the test-driven development group, John Roth is unnecessarily modest:

The suite of tests that come out of TDD looks pretty insufficient to a classical tester.

Here's my response, which I decided to preserve.

I have to disagree, and I do it with my classical tester hat on. For two reasons.

First, let's suppose that TDD code is less well tested than the classical ideal. Doesn't matter, since the classical tester almost never sees code that comes close to that ideal. Testers spend much of their day finding, laboriously, bugs that programmers could have found easily. Typically, those are bugs where the programmer intended something, but didn't check that she had in fact achieved her intention (or didn't check that her intention stayed achieved as the program changed). TDD certainly does that.

Next, I argue that TDD code isn't all that far from the classical ideal. There are really three kinds of classical ideal.

1) The test suite meets some structural coverage criterion. Generally, the minimal criterion considered acceptable is that all logical tests evaluate both true and false. For if (a && b), that requires three tests.

I believe a competent set of TDD tests will come awfully close to that criterion. It might not reach more stringent ones, such as those based on dataflow coverage, but I argue that those criterion don't have much to recommend them (The Craft of Software Testing).

2) The code meets a bug coverage criterion: some variant of mutation testing. (Jester measures a kind of mutation coverage.) Generally, mutation testing works by making one-token changes to the program and checking whether any test notices that change. The idea is that a test suite that can't tell whether the code should be a < b or a > b must be missing lots of other bugs. That seems plausible.

What doesn't seem plausible to me is the assumption that a test suite that can detect one-token bugs is necessarily good at detecting more complicated bugs. As far as I know, there's no hard evidence for that, and I suspect it's false (because of faults of omission). There are other problems with mutation testing - it can require a lot of tests, and my (limited) personal experience is that it has a low "Aha!" factor. That is, when I designed what I considered to be good tests, then checked them with a mutation testing tool I'd written, the work rarely resulted in my saying, "Oh, that's a good test."

So I wouldn't worry about that criterion.

3) Note that I said "pretty good tests" above. Where did those pretty good tests come from? Experience. I have in my head (and, sometimes, on paper) a catalog of plausible bugs and the kinds of test cases that find them. I've built this by, first, standing on the shoulders of others and, second, by paying attention when I see bugs.

TDD asks programmers to pay attention. In notes they've written to this list, John Arrizza and Bill Wake have shown that they are. Ron Jeffries has spoken of learning how to test better by paying attention to what bugs slip past. As paying attention becomes part of the received wisdom, all TDD programmers will write tests that look pretty darn sufficient.

## Posted at 14:05 in category /testing [permalink] [top]

About Brian Marick
I consult mainly on Agile software development, with a special focus on how testing fits in.

Contact me here: marick@exampler.com.




Agile Testing Directions
Tests and examples
Technology-facing programmer support
Business-facing team support
Business-facing product critiques
Technology-facing product critiques
Testers on agile projects

Permalink to this list


Working your way out of the automated GUI testing tarpit
  1. Three ways of writing the same test
  2. A test should deduce its setup path
  3. Convert the suite one failure at a time
  4. You should be able to get to any page in one step
  5. Extract fast tests about single pages
  6. Link checking without clicking on links
  7. Workflow tests remain GUI tests
Permalink to this list


Design-Driven Test-Driven Design
Creating a test
Making it (barely) run
Views and presenters appear
Hooking up the real GUI


Popular Articles
A roadmap for testing on an agile project: When consulting on testing in Agile projects, I like to call this plan "what I'm biased toward."

Tacit knowledge: Experts often have no theory of their work. They simply perform skillfully.

Process and personality: Every article on methodology implicitly begins "Let's talk about me."


Related Weblogs

Wayne Allen
James Bach
Laurent Bossavit
William Caputo
Mike Clark
Rachel Davies
Esther Derby
Michael Feathers
Developer Testing
Chad Fowler
Martin Fowler
Alan Francis
Elisabeth Hendrickson
Grig Gheorghiu
Andy Hunt
Ben Hyde
Ron Jeffries
Jonathan Kohl
Dave Liebreich
Jeff Patton
Bret Pettichord
Hiring Johanna Rothman
Managing Johanna Rothman
Kevin Rutherford
Christian Sepulveda
James Shore
Jeff Sutherland
Pragmatic Dave Thomas
Glenn Vanderburg
Greg Vaughn
Eugene Wallingford
Jim Weirich


Where to Find Me

Software Practice Advancement


All of 2006
All of 2005
All of 2004
All of 2003



Agile Alliance Logo