Exploration Through Example

Example-driven development, Agile testing, context-driven testing, Agile programming, Ruby, and other things of interest to Brian Marick
191.8 167.2 186.2 183.6 184.0 183.2 184.6

Fri, 18 Apr 2003

Two testing principles, illustrated

I used to have the pleasure of working with Mark Miller, capability security guy. One of his charming quirks was that, when someone found a bug in his code, he'd send email to everyone describing how he'd gone wrong and what he'd learned by fixing the bug. He also bought lunch for the bug finder.

Here's a bug that escaped my tests. I found it through use yesterday. What did it teach me about my testing?

It's in a time-tracking program. Here's the sequence of command-line events.

This starts the background job, which happens to be named 'misc'. That job tracks time that's not tracked by a more specific job.
start 'plop'
Now I'm working on PLoP. If I paused this job, 'misc' would resume accumulating time.
start 'stqe'
Now I'm going to start editing an article for STQE. Starting this new job pauses the 'plop' job. But wait - I remembered something else PLoPish to do. So...
This puts me back to working on PLoP. Whatever time had been accumulated for 'stqe' would be given to 'plop'.

At this point, I should have 'misc' paused, 'plop' running (accumulating time), and 'stqe' should have never been started. As I later discovered, both 'misc' and 'plop' were running. That's one of the worst bugs that could happen in a timeclock program. No double-billing at Testing Foundations!

The cause of the bug was that I incorrectly believed that I could use the 'stop' command to undo most of what 'start' does. That's true, but it also does something extra. As it proceeds, the 'stop' command calls out to all paused jobs and says, "Hey! I'm stopping the running job. Anyone want to volunteer to start up in its place?" The background job, 'misc', volunteered.

Why didn't my tests prevent me from making this mistake?

One of my testing principles is that tests should set and check the background. (Here, 'background' isn't supposed to mean 'background job' - it's supposed to evoke the background of a painting or a photograph - that part that's not the focus of your attention. Specifically, tests should check that what was supposed to be left alone was in fact left alone, not only that what was supposed to be changed was changed.)

Examining my undo test suite, I found that I had some tests of that sort. I had a test that made sure that an unstarted job was left alone. I had a test that checked that a paused non-background job was left paused. As far as I can tell, I only missed the one crucial idea for a test. I bet I wasn't following the background principle consciously enough. When coding in a test-first way, I sometimes find it hard to sit back, breathe deeply, and do a global case analysis of the problem.

Of course, testing should be robust in the face of human frailty. That's the root of another of my principles: use visible reminders. There are certain test ideas that have high value. If they're easy to forget (which you know by forgetting them), they should be written down. A reminder catalog isn't useful unless it's seen, so it should be highly visible (as an information radiator). When I look up from my screen, I should see test ideas like:

The background job
(it likes to restart)

in front of me, instead of a big poster for Gene Wolfe's The Book of the New Sun, a "decsystem10" bumper sticker from 1981, a Claude Monet print, an extra cover from a Patti Smith album, and an oil painting of galaxies.

## Posted at 09:25 in category /testing [permalink] [top]

About Brian Marick
I consult mainly on Agile software development, with a special focus on how testing fits in.

Contact me here: marick@exampler.com.




Agile Testing Directions
Tests and examples
Technology-facing programmer support
Business-facing team support
Business-facing product critiques
Technology-facing product critiques
Testers on agile projects

Permalink to this list


Working your way out of the automated GUI testing tarpit
  1. Three ways of writing the same test
  2. A test should deduce its setup path
  3. Convert the suite one failure at a time
  4. You should be able to get to any page in one step
  5. Extract fast tests about single pages
  6. Link checking without clicking on links
  7. Workflow tests remain GUI tests
Permalink to this list


Design-Driven Test-Driven Design
Creating a test
Making it (barely) run
Views and presenters appear
Hooking up the real GUI


Popular Articles
A roadmap for testing on an agile project: When consulting on testing in Agile projects, I like to call this plan "what I'm biased toward."

Tacit knowledge: Experts often have no theory of their work. They simply perform skillfully.

Process and personality: Every article on methodology implicitly begins "Let's talk about me."


Related Weblogs

Wayne Allen
James Bach
Laurent Bossavit
William Caputo
Mike Clark
Rachel Davies
Esther Derby
Michael Feathers
Developer Testing
Chad Fowler
Martin Fowler
Alan Francis
Elisabeth Hendrickson
Grig Gheorghiu
Andy Hunt
Ben Hyde
Ron Jeffries
Jonathan Kohl
Dave Liebreich
Jeff Patton
Bret Pettichord
Hiring Johanna Rothman
Managing Johanna Rothman
Kevin Rutherford
Christian Sepulveda
James Shore
Jeff Sutherland
Pragmatic Dave Thomas
Glenn Vanderburg
Greg Vaughn
Eugene Wallingford
Jim Weirich


Where to Find Me

Software Practice Advancement


All of 2006
All of 2005
All of 2004
All of 2003



Agile Alliance Logo