Exploration Through Example

Example-driven development, Agile testing, context-driven testing, Agile programming, Ruby, and other things of interest to Brian Marick

191.8

⇒

167.2

⇒

186.2

183.6

184.0

183.2

184.6

Sun, 20 Jun 2004

Exploratory game design

In our tutorial on exploratory testing at Agile Development Conference, Elisabeth Hendrickson and I will be doing something odd. We'll talk about exploratory testing of software, but we'll demonstrate it by having teams design and test a game. Our view of where exploratory testing fits into Agile is that it's a dandy end-of-iteration activity, during which people give the software a test drive and get ideas that feed into later iterations.

But suppose we wanted to demonstrate that with software. We'd spend half an hour getting people's laptops ready, then they'd do the exploratory testing, then... what? You can't have another iteration - the software is what it is. So that would miss the feel of the process, and the feel is important. So we'll concentrate on the feel - and on four key techniques - and defer the direct experience of software exploration until after the session (perhaps later in the conference).

Coming to the tutorial? Here are the game design notes. Couldn't hurt to read them in advance (but it's not required).

## Posted at 13:13 in category /agile [permalink] [top]

The danger of numbers

From a Washington Post article summarizing the state of Iraq:

Bremer acknowledged he was not able to make all the changes to Iraq's political system and economy that he had envisioned, including the privatization of state-run industries. He lamented missing his goal for electricity production and the effects of the violence. In perhaps the most candid self-criticism of his tenure, he said the CPA erred in the training of Iraqi security forces by "placing too much emphasis on numbers" instead of the quality of recruits. (Emphasis mine.)

In a Wall Street Journal article about the Abu Ghraib scandal, we have this:

"The whole ball game over there is numbers," a senior interrogator, Sergeant First Class Roger Brokaw, told the paper. "How many raids did you do last week? How many prisoners were arrested? How many interrogations were conducted? How many [intelligence] reports were written? It was incredibly frustrating."

From a Christian Science Monitor article on the same topic:

Yet Specialist Monath and others say they were frustrated by intense pressure from Colonel Pappas and his superiors - Lt. Gen Ricardo Sanchez and his intelligence officer, Maj. Gen. Barbara Fast - to churn out a high quantity of intelligence reports, regardless of the quality. "It was all about numbers. We needed to send out more intelligence documents whether they were finished or not just to get the numbers up," he said. Pappas was seen as demanding - waking up officers in the middle of the night to get information - but unfocused, ordering analysts to send out rough, uncorroborated interrogation notes. "We were scandalized," Monath said. "We all fought very hard to counter that pressure" including holding up reports in editing until the information could be vetted.

I am reminded of my paper, How to Misuse Code Coverage (PDF). (I'm a little appalled that I'm comparing bad testing to Abu Ghraib. Thank God I lead so sheltered a life that I can make such comparisons. But onward.)

I have a wary relationship with numbers. On the one hand, you do sometimes have to make decisions, and when two parties disagree, numbers can shorten arguments. On the other hand, numbers do not merely measure some chosen aspect of reality, they also serve to create reality, often with horrifying unintended consequences.

What to do?

• Cem Kaner has recommended balanced scorecards, the basic idea - I believe - being that it's harder to "game" multiple numbers than one.

• I often ask people proposing new techniques, "What could go wrong?" That has two sub-questions: "Let's assume that your idea is wonderful in general. But there must be situations for which it's a bad idea. What are they?" And "Even if your idea is wonderful in this situation, it will be implemented by frail, fallible, and probably inexperienced humans. What mistakes are they likely to make?" Those questions can be used when someone proposes a particular measurement. The followup question is "how will we know when things are starting to go wrong?"

• People know when numbers are being misused, if only through a vague feeling of disquiet. They need time and permission to reflect. Enter the retrospective.

• Keep pointing out the dangers until the riskiness of numbers becomes common knowledge. Catchy slogans help.

• Teach people the difference between numbers and reality. Cem Kaner has an article (PDF) on that topic.

But those seem mostly negative, reactive. We also need examples of problems solved through incremental use and adjustment of partial information. It also seems to me we need changed attitudes toward management, subjectivity and objectivity, information flow, problems, and solutions. But those are topics for another day.

## Posted at 11:46 in category /misc [permalink] [top]