Exploration Through Example

One thing I meant to say and forgot: Just as the evolution of amphibians didn’t mean that all the fish disappeared, the creation of a new kind of testing to fit a new niche doesn’t mean existing kinds are now obsolete.

Context-driven testing:

Testing Computer Software, Kaner, Falk, and Nguyen
Lessons Learned in Software Testing, Kaner, Bach, and Pettichord
http://www.context-driven-testing.com
“When Should a Test Be Automated?”, Marick

Exploratory testing:

James Bach
Michael Bolton
Elisabeth Hendrickson
Jonathan Kohl

Left out:

The undescribed fourth age

Bruce Daley posts on how most humans are biased to think they’re less error-prone than they are. As far as I know, that’s a claim solidly based in empirical research. (See also Bruce Schneier’s The Psychology of Security.) From this, he concludes:

Given the nature of their work, software developers and software programmers suffer more from the illusion of knowledge and the illusion of control than most other professions, making them particularly subject to over-looking mistakes in their own code. Which is why software needs to be tested independently.

However. Consider the graph below.

Here, the programmer and independent tester start testing at the same time. (Bad programmer! Bad!) The programmer starts out with more knowledge of the app than the tester (the line marked P/+), but she also has a large amount of cognitive bias (P/-) and lacks testing skill. That makes her miss bugs her knowledge would otherwise allow her to find (the area under the red line). Moveover, her biases seem to be pretty impervious to evidence.

The tester starts out with less knowledge, but has no (relevant) cognitive biases at all. Also, his testing skill lets him ramp up his bug finding pretty fast—but it still takes him a while to overcome her advantage.

Which do you want doing the testing? If you’re shipping at time A, it looks like the programmer has the edge. (Compare the shaded areas under the curve.)

We could expect that advantage to erode over time. If the ship date is farther out, the independent tester would have an advantage, as this graph shows:

Even when all that matters is bug count, the decision is not straightforward, especially since it’s based on information you can’t know until after you’ve decided. (How long will it take the tester to get up to speed? How many and what kind of bugs will the programmer miss?)

On most projects, there are lots of other factors to consider.

So I encourage people not to make the assertion the post’s author does.

In response to a potential client, I wrote something very like the following. The interesting thing is that I’m placing more emphasis on manual exploratory testing. It’s not so much that I suddenly realize its importance as that automated business-facing tests continue to be hard to implement and adopt. More on that anon.

A short sketch of a reasonable growth path would go like this:

1. Get the programmers sold on test-driven design. How difficult that is depends mainly on how much legacy code you have (where legacy code is, as Michael Feathers says, code without unit tests). Legacy code is hard to test, so programmers don’t see the benefits of testing as quickly, so it requires that much more discipline to get over what’s always a higher hump than with greenfield code. (Michael Feathers’ Working Effectively with Legacy Code is the gold standard book, though there’s an important strategy—”strangler applications“—that’s not covered in depth. Also, I’m the track chair for a new Legacy Code track at Agile2008, I just asked Feathers to give the keynote, and he says he has “a number of surprising proposals about how to make things better”.)

   I’ve come to feel that the most important thing to get across to programmers is what it’s like to work with code built on a solid base of tests. If they understand that early on, they’ll have a clear idea of what to shoot for, which helps with the pain of legacy code. I wrote a workbook to that end.

2. At the same time, move testers away from scripted manual tests (if that’s what they’re doing) and toward a more exploratory style of manual testing. The people who are strongest on exploratory testing in Agile are Jonathan Kohl, Elisabeth Hendrickson, and Michael Bolton.

3. As programmers do more unit testing, they will become accustomed to changing their design and adding code in support of their own testing. It becomes more natural for them to do the same for the testers, allowing them to do “automation-assisted exploratory testing”. (Kohl writes about this.) I like to see some of the testers learn a scripting language to help with that. Ruby is my favorite, for a variety of reasons. I wrote a book to help testers learn it.

4. Over this period, the testers and programmers should shed most animosity or wariness they have toward each other. They’re working together and doing things to help each other. It helps a lot if they sit together.

5. Once the programmers are sold on test-driven design, they will start wishing that the product owners would supplement what they say about what they want with clear, concrete, executable examples of what they want. That is: tests, written in the language of the business. That isn’t as easy to do as we thought it would be five years ago, but it can be done more or less well. Often, the testers will find a new role as helpers to the product owners. For example, they should get involved early enough to ask questions that lead to tests that prevent bugs (which is better than discovering the bugs after you’ve paid some programmers to implement them).

6. Throughout this, some kinds of testing (like performance testing) don’t change all that much. For performance testing, I trust Scott Barber.

As a side note: I’m quite fond of the new The Art of Agile Development by Shore & Warden: enough to publicly declare that I’ll bring a copy to every team I work with. Lots of good from-the-trenches experience summarized there.

I’m test-driving some Rails helpers. A helper is a method that runs in a context full of methods magically provided by Rails. Some of those methods are of the type that’s a classic motivation for mocks or stubs: if you don’t want them to blow up, you have to do some annoying behind-the-scenes setup. (And because Rails does so much magic for you, it can be hard for the novice to have a clue what that setup is for helpers.)

Let’s say I want a helper method named reference_to. Here’s a partial “specification”: it’s to generate a link to one of a Certification's associated users. The text of the link will be the full name of the user and the href will be the path to that user’s page. I found myself writing mocks along these lines:

mock.should_receive(:user_path).once.
     with(:id=>@originator.login).
     and_return("**the right path**")
mock.should_receive(:link_to).once.
     with(@originator.full_name, "**the right path**").
     and_return("**correct-text**")

But then it occurred to me: The structure I’m building is isomorphic to the call trace, so why not replace the real methods with recorders? Like this:

  def user_path(keys)
    "user_path to #{keys.canonicalize}"
  end

  def link_to(*args)
    "link to #{args.canonicalize}"
  end

  def test_a_reference_is_normally_a_link
    assert_equal(link_to(@originator.full_name, user_path(:id => @originator.login)),
                 reference_to(@cert, :originator))
  end

This test determines that:

• the methods called are the right ones to implement the specified behavior. There’s a clear correspondence between the text of the spec (”generate a link to”) and calls I know I made (link_to).

• the methods were called in the right order (or in an order-irrelevant way).

• they were called the right number of times.

• the right arguments were given.

So, even though my fake methods are really stubs, they tell you the same things mocks would in this case. And I think the test is much easier to grok than code with mocks (especially if I aliased assert_equal to assert_behaves_like).

What I’m wondering is how often building a structure to capture the behavior of the thing-under-test will be roughly as confidence-building and design-guiding as mocks. The idea seems pretty obvious (even though it took me forever to think of it), so it’s probably either a bad idea or already widely known. Which?

Alternately, I’m still missing the point of mocks.

P.S. For tests to work, you have to deal with the age-old problems of transient values (like dates or object ids) and indeterminate values (like the order of elements in a printed hash). I’m fortunate in that I’m building HTML snippets out of simple objects, so this seems to suffice:

class Object
  def canonicalize; to_s; end
end

class Array
  def canonicalize
    collect { | e | e.canonicalize }
  end
end

class Hash
  def canonicalize
    to_a.sort_by { | a | a.first.object_id }.canonicalize
  end
end

There’s one of those tagging memes going around. This one is: “grab the nearest book, open to page 123, go down to the 5th sentence, and type up the 3 following sentences.”

My first two books had pictures on p. 123.

The next three (Impro: Improvisation and the Theatre, AppleScript: the Definitive Guide, and Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life) didn’t have anything that was amusing, enlightening, or even comprehensible out of context. So I kept going, which is cheating I suppose. The last, How Designers Think, had this:

The designer’s job is never really done and it is probably always possible to do better. In this sense, designing is quite unlike puzzling. The solver of puzzles such as crosswords or mathematical problems can often recognize a correct answer and knows when the task is complete, but not so the designer.

That’s a hit. It made me realize a flaw in my thinking. You see, it reminded me of one of my early, semi-controversial papers, “Working Effectively With Developers” (referred to by one testing consultant as “the ‘how to suck up to programmers’ paper”). In its second section, “Explaining Your Job”, I explicitly liken programmers to problem solvers:

A legendary programmer would be one who was presented a large and messy problem, where simply understanding the problem required the mastery of a great deal of detail, boiled the problem down to its essential core, eliminated ambiguity, devised some simple operations that would allow the complexity to be isolated and tamed, demonstrated that all the detail could be handled by appropriate combinations of those operations, and produced the working system in a week.

Then I point out that this provides a way for testers to demonstrate value. I show a sample problem, then write:

Now, I’d expect any programmer to quickly solve this puzzle - they’re problem solvers, after all. But the key point is that someone had to create the puzzle before someone else could solve it. And problem creation is a different skill than problem solving.

Therefore, the tester’s role can be likened to the maker of a crossword or a mathematical problem: someone who presents a good, fully fleshed-out problem for the programmer to master and solve:

So what a tester does is help the programmer […] by presenting specific details (in the form of test cases) that otherwise would not come to her attention. Unfortunately, you often present this detail too late (after the code is written), so it reveals problems in the abstractions or their use. But that’s an unfortunate side-effect of putting testers on projects too late, and of the unfortunate notion that testing is all about running tests, rather than about designing them. If the programmer had had the detail earlier, the problems wouldn’t have happened.

Despite this weak 1998 gesture in the rough direction of TDD, I still have a rather waterfall conception of things: tester presents a problem, programmer solves it, we all go home.

But what that’s missing is my 2007 intellectual conception of a project as aiming to be less wrong than yesterday, to get progressively closer to a satisfactory answer that is discovered or refined along the way. In short—going back to the original quote—a conception of the project as a matter of design that’s at every level of detail and involves everyone. That whole-project design is something much trickier than mere puzzle-solving.

I used the word “intellectual” in the previous paragraph because I realize that I’m still rather emotionally attached to the idea of presenting a problem, solving it, and moving on. For example, I think of a test case as a matter of pushing us in a particular direction, only indirectly as a way of uncovering more questions. When I think about how testing+programming works, or about how product director + team conversations work, the learning is something of a side effect. I’m strong on doing the thing, weak on the mechanics of learning (a separate thing from the desire to learn).

That’s not entirely bad—I’m glad of my strong aversion to spending much time talking and re-talking about what we’ll build if we ever get around to building anything, of my preference for doing something and then taking stock once we have more concrete experience—but to the extent that it’s a habit rather than a conscious preference, it’s limiting. I’ll have to watch out for it.

On the agile-testing list, I answered this question:

So for the first time in many, many years I’m not in a test management position, and I’m writing tests, automating them, etc. We’re using a tool called soapUI to automate our web services testing–it’s a handy tool, supports Groovy scripting which allows me to go directly to the DB to validate the results of a given method, etc. One feature of soapUI is centralized test scripts; I can create helper scripts to do a bunch of stuff–basically, I write the Groovy code I need to validate something and then I often find I’m moving it into a helper function, refactoring, etc.. My question is, how do you know the right balance between just mashing up the automation (ie, writing a script as an embeded test script) vs. creating the helper function and calling it?

Bret Pettichord suggested I blog my answer, so here it is:

I use these ideas and recommend them as a not-bad place to start:

1. At the end of every half-day or full-day of on-task work (depending on my mood), I’ll spend a half an hour cleaning up something ugly I’ve encountered.

2. I’ll periodically ask myself “Is it getting easier to write tests?” If it’s not, I know I should slow down and spend more effort writing helper code. I know from experience what it feels like when I’ve got good test support infrastructure—how easy writing tests can be—so I know what to shoot for.

3. I have this ideal that there should be no words in a test that are not directly about the purpose of the test. So: unless what the test tests is the process of logging in, there should be no code like:
   
create_user "sam", "password"
login "sam", "password"
...


   Instead, the test should say something shorter:
   
using_logged_in_user "sam"
...


   That tends to push you toward more and smarter helper functions.

   Unsurprisingly, I often fall short of that ideal. But if I revisit a test and have any trouble at all figuring out what it’s actually testing, I use that as a prod to make the extra effort.

What was something of a breakthrough for me was realizing that I don’t have to get it right at this precise moment. Especially at the beginning, when you don’t have much test infrastructure, stopping every twelve seconds to write that helper function you ought to have completely throws you out of the flow of what you’re trying to accomplish. I’ve gotten used to writing the wrong code, then fixing it up later: maybe at the end of the task, maybe not until I stumble across the ugliness again.

In product code, not repeating yourself is almost always a good idea. In tests, it’s not so clear-cut. Repeating yourself has the same maintenance dangers as it does for code, but not repeating yourself has two additional downsides:

• A common knock against well-factored object-oriented code is that no object does anything; they all just forward work on to other objects. That structure turns out to be useful once you’re immersed in the system, but it does make systems harder to learn.

  One purpose of tests is to explain the code to the novice. Remove duplication too aggressively, and the tests do a poor job of that.

• Another purpose of tests is to make yourself think. One way to do that is to force yourself to enumerate possibilities and ask “What should happen in this case?” That’s one of the reasons that I, when acting as a tester, will turn a state diagram into a state table. A state diagram doesn’t make it easy to see whether you’ve considered the effect of each possible event in each possible state; a state table does. (It’s not as simple as that, though: it’s hard to stay focused as you work through a lot of identical cases looking for the one that’s really different. It’s like the old joke that ends “1 is a prime, 3 is a prime, 5 is a prime, 7 is a prime, 9 is a prime…”)

  If you factor three distinct assertions into a single master assertion, it’s easy to overlook that the second shouldn’t apply in some particular case. When you factor three distinct setup steps into one method, you can more easily fail to ask what should happen when the second setup step is left out.

So as I balance the different forces, I find myself writing test code like this:

  # Guard against manufactured URLs.
  def test_cannot_update_a_user_unless_logged_in
    new_profile = NewProfile.for(’quentin‘).is_entirely_different_than(users(:quentin))

    put :update,
        {:id => users(:quentin).login, :user => new_profile.contents}
        # Nothing in session
    assert_redirected_to(home_path)
    assert_hackery_notice_delivered
    new_profile.assert_not_installed
  end

  def test_cannot_update_a_user_other_than_self
    new_profile = NewProfile.for(’quentin‘).is_entirely_different_than(users(:quentin))

    put :update,
        {:id => users(:quentin).login, :user => new_profile.contents},
        {:user => users(:aaron).id}
    assert_redirected_to(home_path)
    assert_hackery_notice_delivered
    new_profile.assert_not_installed
  end

There’s duplication there. In an earlier version, I’d in fact reflexively factored it out, but then decided to put it back. I think the tests are better for that, and I’m willing to take the maintenance hit.

Nevertheless, there’s a problem. It’s not obvious enough what’s different about the two tests. What to do about that?

Consider explaining the evolution of a program over time in a book. Authors don’t usually show a minimal difference between before and after versions. Instead, they show both versions with a fair amount of context, somehow highlighting the differences. (When I write, I tend to bold changed words.) I wish I could highlight what’s special about each test in my IDE, so that it would look like this:

  # Guard against manufactured URLs.
  def test_cannot_update_a_user_unless_logged_in
    new_profile = NewProfile.for(’quentin‘).is_entirely_different_than(users(:quentin))

    put :update,
        {:id => users(:quentin).login, :user => new_profile.contents}
        # Nothing in session
    assert_redirected_to(home_path)
    assert_hackery_notice_delivered
    new_profile.assert_not_installed
  end

  def test_cannot_update_a_user_other_than_self
    new_profile = NewProfile.for(’quentin‘).is_entirely_different_than(users(:quentin))

    put :update,
        {:id => users(:quentin).login, :user => new_profile.contents},
        {:user => users(:aaron).id}
    assert_redirected_to(home_path)
    assert_hackery_notice_delivered
    new_profile.assert_not_installed
  end

Something for IDE writers to implement.

Jason Gorman

“And that, folks, is how enterprise-scale reuse works. It is, I tell you. It’s true!”

Ben Simo

“We can’t stop the conversation at ‘I just did that and I’m a user.’”

The “no user would do that” retort is the bane of testers. Ben talks well about moving the conversation past that. But a step further: any project I’d want to work on is a learning project, one that wants to be less wrong than yesterday, one that likes finding out about mistakes. Get past this particular conversation: fine. Maybe testers could even train programmers to swallow that particular reflexive retort. But the defensiveness about having partial understanding will still leak out other places.

Now, I once sat down with Elisabeth Hendrickson while she tested an app of mine. I’d built it with TDD to the max: business-facing tests all the way down to unit tests. It took her about ten minutes to find a high-priority bug. I immediately slipped right into the defensive programmer stance. It took me a few minutes to snap out of it. But if we worked together for longer, I’d like to think I’d get past that.

I aspire to be like Mark “capabilities” Miller, a programmer I once worked with. When someone found a bug in his code, he’d write a long email about it, praising the person, attributing all sorts of cleverness to her, and explaining how he’d come to make that mistake.

Bret Pettichord

“People often recommend that you treat a bug as a story. […] I think this approach is incorrect. We’ve found a better way to handle [bugs].”

I want to disagree with Bret, but I haven’t come up with a counterexample that convinces even me.

Milton Mayer

“What happened here was the gradual habituation of the people, little by little, to being governed by surprise; to receiving decisions deliberated in secret; to believing that the situation was so complicated that the government had to act on information which the people could not understand, or so dangerous that, even if the people could not understand it, it could not be released because of national security….

“And one day, too late, your principles, if you were ever sensible of them, all rush in upon you….The world you live in — your nation, your people — is not the world you were born in at all. The forms are all there, all untouched, all reassuring, the houses, the shops, the jobs, the mealtimes, the visits, the concerts, the cinema, the holidays. But the spirit, which you never noticed because you made the lifelong mistake of identifying it with the forms, is changed.”

Here are some links I will point to when people ask me about test design. Add more in the comments and I’ll promote them to the main posting.

Mnemonics

• Michael Bolton on:

  • SF DePOT (Structure, Function, Data, Platform, Operations, and Time) here and here

  • CRUSSPIC STMPL (capability, reliability, usability, security, scalability, performance, installability, compatibility, supportability, testability, maintainability, portability, and localizability)

  • HICCUPPS (History, Image, Comparable Products, Claims, Users’ expectations, the Product itself, Purpose, Statutes)

  See also various of Bolton’s articles.

• Adam Goucher on SLIME (Security, Languages, requIrements, Measurement, Existing)

• Jonathan Kohl on MUTII (Market, Users, Tasks, Information, Implementation).

• Michael Kelly on test reporting with FCC CUTS VIDS (Feature tour, Complexity tour, Claims tour, Configuration tour, User tour, Testability tour, Scenario tour, Variability tour, Interoperability tour, Data tour, Structure tour)

• Ben Simo on testing failure handling with FAILURE (Functional, Appropriate, Impact, Log, UI, Recovery, Emotions)

• Scott Barber on designing model workloads for performance testing with FIBLOTS (Frequent, Intensive, Business Critical, Legal, Obvious, Technically Risky, Stakeholder Mandated). (He has others that are outside the scope of this posting.)

Other reminders

• Elisabeth Hendrickson’s Test Heuristics Cheat Sheet.

• Michael Hunter’s huge list of things to check.

Mind maps

• Jonathan Kohl on Modeling test heuristics

• Chris McMahon on Mind maps for testing and other things

• Bob Warfield on Using mindmaps to explore user interaction and create QA test plans

• Eric somebody on Using mind maps to explore user interaction

Online course materials

• Black box software testing materials from James Bach, Cem Kaner, Michael Bolton, and others.

Books

• A Practitioner’s Guide to Test Design, by Lee Copeland

• Lessons Learned in Software Testing, by Cem Kaner, James Bach, and Bret Pettichord.

Things that ought to be in books

• Jonathan Kohl has a series on exploratory testing (Investigation, Improvisation, Transitions, Strategies).

• Numerous articles from James Bach, most setting context and providing perspective.

Videos

• Cem Kaner has various Black Box Software Testing videos.

Thanks to Michael Bolton, Adam Goucher, Matthew Heusser, Jonathan Kohl, and Chris McMahon for links.

At the functional testing tools workshop, Ward Cunningham devised a scheme. John Dunham, Adam Geras, Naresh Jain, and I elaborated on it. Before I explain the scheme, I have to reveal the context, which is different kinds of tests with different audiences and affordances:

As when I first encountered wikis, I missed the point of Ward’s newest thing. I didn’t think swimlane tests had anything to do with what I’m interested in: the use of tests and examples as tools for thinking about a piece of what the product owner wants with results that help the programmers. With swimlane tests, the product owner is going to be mostly reactive, saying “these test results are not what I had in mind”.

What I missed is that Ward’s focus wasn’t on driving programming but on making the system more visible to its users. They were written for the Eclipse Foundation. It wants to be maximally transparent, and the swimlane tests are documentation to help the users understand how the Foundation’s process works along with assurance that it really does work as documented.

What Ward noticed is that his output might fit together nicely with my input. Like this:

Here, the product owner begins with an executable graphical example of what she wants. The programmers produce code from it. As a user runs that code, she can at any moment get a bundle of pictures that she can point at during a conversation with the producer and other users. That might lead to some new driving examples .

The point of all of this? It’s a socio-technical trick that might entice users to collaborate in application development the way wikis entice readers to collaborate in the creation of, say, an encyclopedia. It’s a way to make it easier for markets to really be conversations. John Dunham dubbed this “crowdware”.

It won’t be quite as smooth as editing a wiki. First, as Naresh pointed out, the users will be pointing at pictures of a tool in use, and they will likely talk about the product at that level of detail. To produce useful driving examples (and software!), talk will have to be shifted to a discussion of activities to be performed and even goals to be accomplished. The history of software design has shown that we humans aren’t fantastically good at that. Adding more unskilled people to the mix just seems like a recipe for disasterware.

Wikis seemed crazy too, though, so what the heck.

Laurent Bossavit and I are working on a site. Once it reaches the minimal features release, I’ll add some crowdware features.