Exploration Through Example

Example-driven development, Agile testing, context-driven testing, Agile programming, Ruby, and other things of interest to Brian Marick
191.8 167.2 186.2 183.6 184.0 183.2 184.6

Tue, 24 Feb 2004

Our slant on exploratory testing

For our tutorial on exploratory testing at Agile Development Conference, Elisabeth Hendrickson and I have to answer this question: "What's exploratory testing for, anyway? In an agile project, I mean."

The answer we'll use in the tutorial will be something like this:

We advocate Exploratory Testing as an end-of-iteration ritual in which the Customer, programmers, and other interested stakeholders have the opportunity to take the new Business Value for a test drive to discover New Information.

What we'll want to do is simulate that experience in the tutorial. Our first thought was to do it on (surprise!) software. But there are two problems. First, getting everyone's machine working, getting the software running on it, dealing with configuration problems, etc. - that all takes a good half an hour. That is, maybe 15% of our available time. Ouch.

Second, we can't show the results of the exploratory testing. We can't show a Customer saying, "Oh! What a good idea! Let's put that in right now" - and then having it put in, and then having the Customer see it in action. We can't go through a full iteration, much less more than one. Yet, if exploratory testing's role is largely - as I believe - about shortening the project's feedback loop, we'd be doing a bad thing if we didn't close the loop.

So what we're planning on doing is to have teams of people design a game. The game will be one that demonstrates some property of Agile development. When we tried this out ourselves, we wanted to devise a game that puts across how and why test-driven design feels slower, starts out slower, but catches up in the end.

After the game is designed, groups will divide. Two people will stay, the others will go to join other groups. The changed groups will then playtest the game. They'll use a couple of exploratory techniques we'll describe. They'll come up with new information, things like:

  • Wow, so that's what that feels like.

  • Hmm... I've changed my mind.

  • Oh, this is missing something.

  • Gee, that was unexpected.

  • That didn't work.

  • Maybe what we deferred was more important than we thought.

Then that group will stay together and add something new to the game. Then there'll be another round of playtesting, with a couple of new exploratory techniques.

We're aiming for three things. First, a "Wow, so that's what that feels like" reaction applied to exploratory testing itself. Second, a desire to try it out at a home company. Third, to provide some concrete techniques that apply to software.

And, if we're lucky, we'll get some nice games that really make points well.

Our plans may change.

## Posted at 21:14 in category /2004conferences [permalink] [top]

Definitions get ever more slippery

I've long been fascinated by the notion of incommensurability. It's a term in science studies made popular (sic) by Kuhn's Structure of Scientific Revolutions and Feyerabend's Against Method. Two theories are incommensurable if neither can be fully stated in the vocabulary of the other. Feyerabend argued that incommensurability means that we have no rational (context-independent) way of judging between rival theories.

Terminology can also be incommensurable, since theories are built from terminology (and vice versa). A good example is "velocity". When Galileo was arguing with the Aristotelians about his new world view, both sides used the word "velocity". But Galileo meant something like what we today call "instantaneous velocity", and the Aristotelians meant something like what we call "average velocity". So if they both watched the same experiment they would likely get different answers to the question "What's the ball's velocity?" We can, with hindsight, say they ought to define their terms better. But that's part of the problem. They could define velocity in terms of motion, but "motion" also meant something different to an Aristotelian. A theory of motion must necessarily say something about growth, since the growth of a tree is the same phenomenon as the falling of a ball. And what exactly is the point of a theory of falling balls that can't even begin to explain why fire rises? - the Aristotelian theory could.

You can see a conversation going nowhere.

Kuhn writes (at least tentatively) as if such conversations must go nowhere. People with incommensurable theories live in different worlds. It's as if they have different perceptions. Incommensurability is a gulf that can't be bridged by talk or definitions, only by experience (what Kuhn likens to a gestalt shift).

Because of incommensurability, I accept frustration when communicating between the agile and non-agile worlds. Words like "test" and "design" come freighted with different world views. That's one of the reasons I've tried to talk about tests as "checked examples". Maybe if we use different words, talk will be easier.

But wait - it gets worse.

Last night, I read the first two chapters of Esther-Mirjam Sent's The Evolving Rationality of Rational Expectations: An Assessment of Thomas Sargent's Achievements for a seminar I'm sitting in on. Given that I have nowhere near the economics background the book assumes, I can only give a thumbnail sketch. There was this economist, Sargent. He worked on something called "rational expectations" for many years. Rational expectations holds that people's predictions don't err systematically. They err randomly. That assumption has all sorts of consequences, none of which I understand.

What struck me on page 19 was this sentence: "Sargent's different interpretations of rational expectations were temporally specific." Although that's vaguely worded enough that I'm not sure what Sent was thinking, it made me think this: It's likely that Sargent would say all along that he was working on a single thing named "rational expectations," but what he meant by that term changed over time.

So imagine: not only do Galileo and the Aristotelians face an incommensurability barrier, the Aristotelians have to track the changing connotations and denotations of "velocity". We, today, can say Galileo was always talking about instantaneous velocity, just getting ever better at figuring out what that meant. But that's probably not at all what the story looks like from the inside as it happens, even if it looks like that to Galileo after the fact (since Galileo is doubtless as good at telling stories to himself as we are at telling stories about him).

It's a wonder we can communicate at all about important things. That we do, I humbly submit, has a lot to do with talking about examples, not about definitions. And, perhaps more important, with doing things together. And with imagining what it would be like to do things like someone else does them.

## Posted at 20:39 in category /mangle [permalink] [top]

About Brian Marick
I consult mainly on Agile software development, with a special focus on how testing fits in.

Contact me here: marick@exampler.com.




Agile Testing Directions
Tests and examples
Technology-facing programmer support
Business-facing team support
Business-facing product critiques
Technology-facing product critiques
Testers on agile projects

Permalink to this list


Working your way out of the automated GUI testing tarpit
  1. Three ways of writing the same test
  2. A test should deduce its setup path
  3. Convert the suite one failure at a time
  4. You should be able to get to any page in one step
  5. Extract fast tests about single pages
  6. Link checking without clicking on links
  7. Workflow tests remain GUI tests
Permalink to this list


Design-Driven Test-Driven Design
Creating a test
Making it (barely) run
Views and presenters appear
Hooking up the real GUI


Popular Articles
A roadmap for testing on an agile project: When consulting on testing in Agile projects, I like to call this plan "what I'm biased toward."

Tacit knowledge: Experts often have no theory of their work. They simply perform skillfully.

Process and personality: Every article on methodology implicitly begins "Let's talk about me."


Related Weblogs

Wayne Allen
James Bach
Laurent Bossavit
William Caputo
Mike Clark
Rachel Davies
Esther Derby
Michael Feathers
Developer Testing
Chad Fowler
Martin Fowler
Alan Francis
Elisabeth Hendrickson
Grig Gheorghiu
Andy Hunt
Ben Hyde
Ron Jeffries
Jonathan Kohl
Dave Liebreich
Jeff Patton
Bret Pettichord
Hiring Johanna Rothman
Managing Johanna Rothman
Kevin Rutherford
Christian Sepulveda
James Shore
Jeff Sutherland
Pragmatic Dave Thomas
Glenn Vanderburg
Greg Vaughn
Eugene Wallingford
Jim Weirich


Where to Find Me

Software Practice Advancement


All of 2006
All of 2005
All of 2004
All of 2003



Agile Alliance Logo