Exploration Through Example

Part 2

I continue to use little experiments to help me think through TDD in Clojure. (I plan to begin a realistic experiment soon.) Right now, I’m mainly focused on three questions:

• What would mocking or stubbing mean in a strict(ish) functional language?

• What’d be a good mocking notation for Clojure?

• How do you balance the outside-in style associated with mocks and the bottom-up style that the REPL (interpreter) encourages?

Here’s an example from Conway’s Game of Life. It begins with an implementation suggestion from Paul Blair and Michael Nicholaides at the Philly Code Retreat. Instead of thinking of the board as a 2×2 array of cells, with some of them dead and some alive, think instead only of living cells, each of which knows its coordinates. Here’s an example that shows how “blinkers” blink from generation to generation.

A couple of things have happened here:

• This is my notation for a straightforward non-stubbing test. The value on the left is executed and it’s compared (for equality) to the value on the right.

• I’ve started coding outside-in, and I’ve named the first function I need: next-world.

The Blair/Nicholaides approach advances the “world” to the next generation by (conceptually) adding dead cells around the edge of all the living cells, running the normal life rules that govern how cells change because of their neighbors, and then throwing away all the cells that end up dead. In other words:

• The pending bit is just there because (sadly) Clojure makes you declare functions before mentioning them. pending just creates functions that print that they’ve not yet been implemented.

• The rest of the code flows the world argument through a pipeline of three functions. If you’re not familiar with the -> macro, the result is the same as this:

  I don’t feel the need to test this code now because it’s really declarative—it says what it means to produce a next world under this approach. (It will be tested in the very end by the “integration test” that shows a blinker working.)

I can now implement any of the three new functions. I’ll pick tick because it seems to be the heart of the matter. Here’s a first implementation:

There are two odd things going on here.

• I think of the Clojure programs I’ve written this way: there’s a big stateful universe out there. At some moment, a snapshot of relevant bits of that universe is presented to my pure functional program. It churns away, presents the results back to the big stateful universe, and then disappears. Given that bias, I find it convenient to think of the universe-bits not as data to be passed around, but as the unchanging background behind my code’s calculations. Any of that code should be able to reach out and grab bits of the relevant background by calling dynamically bound functions of no arguments.

  Because of against-background, any code operating inside successor (or inside functions it calls) can call (world) and magically get the original bordered-world. As you’ll see, this lets me put off committing to a representation for cells. (The representation I used for the blinkers isn’t enough after the border’s been added, since there’s now a need to distinguish between living and dead cells.)

• This essay is supposed to be a sketch of TDD in Clojure, but here’s the second function I’ve written without tests. What’s up with that? I’ve tentatively concluded that testing functions that map one function over a sequence is the moral equivalent of testing getters and setters: not worth the trouble. But don’t fear, successor is worth testing, and we’ll do that next. Almost next.

In object-oriented languages, I think of mock-driven-design as a way of teasing out collaborators for the object I’m building. I push responsibilities for work onto objects that I’ll implement later. Mocking lets me defer the implementation of those objects until I’m ready, and creating some examples of the API teaches me the (implicit) specification for the new object.

I’ve found that with pure functional programs that don’t modify state, it makes more sense to think of a function like (f 2) => 4 as a fact. What I’m doing as I test-drive a function is describing how facts about its inputs and outputs depend on other facts, in an almost Prolog-like way. For example, consider this code:

That says that, for any cell you care to provide, f of that cell will be 10, provided g of that cell is true and h is 2. If either of those latter two facts don’t apply to the cell, I’m not saying what f’s value is.

I use the funny ...cell... notation in the way that mathematicians use n to talk about any integer. (They call that universal quantification.) I don’t want to create a particular cell because I might need to specify properties that have nothing to do with the function I’m working on. This notation says that nothing about the cell is relevant except for what comes after the provided.

Here’s one way to write a Life rule in this notation:

The falsey bit in the first line is because Clojure has two distinct values that can mean “false”. falsey is a function that takes the result of the left-hand side and fails the test if that result is anything other than one of the two false values. I’m using it because I don’t want to overspecify living?. There’s no reason to care which of the two “false” values it returns.

There’s a problem with this test, though. Remember what I said above: the left-hand side gets evaluated and handed to falsey. That means living? has to have a definition—which means I’d have to settle on how the code knows whether a cell is alive or dead. I like doing one thing at a time and putting off decisions as long as I can, and right now I’d rather be focused on successor instead of cell representations.

Here’s a way to defer that decision:

Here I’m saying something subtly different than before. I’m saying that the result of successor is specifically that cell produced by calling killed on the original cell. The =means=> notation tells the framework to create a mock instead of evaluating the right-hand side for its value. In a more familiar mocking syntax (for Ruby), the whole test is equivalent to:

OK. The next figure gives the whole set of Life rules, expressed as executable tests. (Well, executable as soon as I implement the testing framework.) Notice that I called the outer wrapper know (a fact) instead of example. know seems more appropriate for rules. The two forms mean the same thing.

Notice also that I implemented a notation for saying “run this test for each value in a sequence”. The use of commas, as in [4,,,8], indicates that—conceptually—the fact is true for all values four through eight. Only the ones listed are actually tried. (Commas count as >white space in Clojure.)

This isn’t the tersest possible format—a table would be better—but it’ll do. I think it’s reasonably readable. Do you?

Here, for reference, is code that passes the test:

We now have an expanded choice of functions to write:

I could go breadth-first—with border and unborder—or go depth-first with one of the functions on the second line. In this particular case, I’d rather go depth first. I’ve avoided deciding on a representation, so I don’t know yet what border should do.

If this installment meets your approval, I’ll add another one that begins work on—oh—probably living-neighbor-count is the most complicated, so it’s a good one to chip away at.

While on my grand European trip, I stopped in for a day at a nice company with a fun group of people doing good work on a legacy code base. They challenged me to improve an existing test using mocks. The test was typical of those I’ve seen in legacy code situations: there was a whole lot of setup code because you couldn’t instantiate any single object without instantiating a zillion of them, and the complexity of the test made figuring out its precise purpose difficult.

After some talk, we figured out that what the test really wanted to check was that when a Quote is recalculated because it’s out-of-date, you get a brand-new Quote.

Rather than morph the test, I tried writing it afresh in my mockish style. A lot of the complexity of the test was in setting things up so that an existing quote should be retrieved. Since my style these days is to push off any hard work to a mocked-out new object, I decided we should have a QuoteFinder object that would do all that lookup for us. The test (in Ruby) would look something like this:

quote_finder = flexmock(”quote finder“)
quote = flexmock(”quote“)

during {
  … some function …
}.behold! {
 quote_finder.should_receive(:find_quote).once.
              with(…whatever…).
              and_return(quote)
 …
}

Next, the new quote had to be generated. The lazy way to do that would be to add that behavior to Quote itself:

quote_finder = flexmock(”quote finder“)
quote = flexmock(”quote“)

during {
  … some function …
}.behold! {
  quote_finder.should_receive(:find_quote).once.
               with(…whatever…).
               and_return(quote)
  quote.should_receive(:create_revised_quote).once.
        with(…whatever…).
        and_return(”a new quote“)
}

Finally, the result of the function-under-test should be the new quote:

quote_finder = flexmock(”quote finder“)
quote = flexmock(”quote“)

during {
  … some function …
}.behold! {
  quote_finder.should_receive(:find_quote).once.
               with(…whatever…).
               and_return(quote)
  quote.should_receive(:create_revised_quote).once.
        with(…whatever…).
        and_return(”a new quote“)
}

assert { @result == “a new quote“ }

I felt a bit of a fraud, since I’d shoved a lot of important behavior into tests that would need to be written by someone else (including the original purpose of the test, making sure the next Quote was a different object than the last one.) The team, though, gave me more credit than I did. They’d had two Aha! moments. First, the idea of “finding a quote” was spread throughout the code, and it would be better localized in a QuoteFinder object. Second, they decided it really did make sense to have Quotes make new versions of themselves (rather than leave that responsibility somewhere else). So this test gave the team two paths they could take to improve their code.

In the beginning, the QuoteFinder and Quote#create_revised_quote would likely just delegate their work to the existing legacy code, but there were now two new organizational centers that could attract behavior. So this looks a lot like Strangling an App, but it avoids that trick’s potential “then a miracle occurs” problem of needing a good architecture to strangle with: instead, by following the make-an-object-when-you-hesitate strategy that mocking encourages, you can grow one.

I’ve not seen any writeup on using mocks to deal with legacy code. Have you?

P.S. It’s possible I’ve gotten details of the story wrong, but I think the essentials are correct.

I’ve been fiddling around with top-down (mock-style) TDD of functional programs off-and-on for a few months. I’ve gotten obsessed with deferring the choice of data structures as long as possible. That seems appropriate in a functional language, where we should be talking about functions more than data. (And especially appropriate in Clojure, my language of choice, since Clojure lets you treat maps/dictionaries as if they were functions from keys to values.)

That is, I like to write these kinds of tests:

(example-of "saturating a terrain"
   (saturated? (... terrain ...)) => true
   (because
      (span-between-markers (... terrain ...)) => (... sub-span ...)
      (saturated? (... sub-span ...)) => true
)

… instead of committing to what a terrain or sub-span look like. That’s been working reasonably well for me.

I’ve also been saying that “maps are getters”. By that, I mean that—given that you’ve test-driven raise-position—it really makes no more sense to test-drive this:

(defn raise [terrain]
   (map raise-position terrain))

… than it does to test a getter: it’s too obvious. That leads to a nice flow of testing: I’m always testing the transformation of things to other things. I don’t have to worry, until the very end of test-driving, that the “things” are actually complex data.

The problem I’ve been running into recently, though, is handling cases where complex data structures are converted into single values. For example, I’ve been trying to show a top-down TDD of Conway’s Life. In that case, I have to reduce a set of facts about the neighborhood of a cell into a single yes-or-no decision: should that cell be alive or dead in the next iteration? But expressing that fact is rather awkward when you don’t want to say precisely what a “cell” is or how you know it’s “alive” or “dead” (other than that there’s some function from a cell and its environment to a boolean).

To be concrete, here’s something I want to claim: a cell is alive in the next iteration if (1) it is alive now and (2) exactly two of the cells in its neighborhood are alive. How do you say that while being not-specific? I’ve not found a way that makes me happy.

Part of the problem, I think, is that when you start talking about individual elements of collections, you’re moving from the Land of TDD, which is a land of functions-of-constants to a Land of Quantified Variables (like “there exists an element of the collection such that…”). That way lies madness.

This here post is about making tests terse by specifying what has happened instead of (as in interaction tests) who did it or (as in state-based test) the different kinds of things-it-has-happened-to.

—

I have a test that says that the Availability object should use the TupleCache object to get particular values for: all animals, animals that are still working, and animals that have been removed from service. If one wants to show animals that can be removed from service, it’s this:

all animals - animals still working - animals already removed from service

Here’s a mock-style test that describes how the Availability uses the TupleCache:

 should “use tuple cache to produce a list of animals“ do
      @availability.override(mocks(:tuple_cache))
      during {
        @availability.animals_that_can_be_removed_from_service
      }.behold! {
        @tuple_cache.should_receive(:all_animals).once.
                     and_return([{:animal_name => ‘out-of-service jake‘},
                                 {:animal_name => ‘working betsy‘},
                                 {:animal_name => ‘some…‘},
                                 {:animal_name => ‘…other…‘},
                                 {:animal_name => ‘…animals‘}])
        @tuple_cache.should_receive(:animals_still_working_hard_on).once.
                     with(@timeslice.first_date).
                     and_return([{:animal_name => ‘working betsy‘}])
        @tuple_cache.should_receive(:animals_out_of_service).once.
                     and_return([{:animal_name => ‘out-of-service jake‘}])
       }
      assert_equal([”…animals“, “…other…“, “some…“], @result)
    end

I’m not wild about the amount of detail in the test, but let’s leave that to the side. Notice that the results of the test imply that the Availability is turning the tuples (think of them as hashes or dictionaries) into a simple list of strings. Notice also that the list of strings is sorted. Noticing that brings a couple of questions to mind:

• That sorting - does it use ASCII sorting, which sorts all uppercase characters in front of lowercase? or is it the kind of sorting the users expect (where case is irrelevant)?

• Are duplicates stripped out of the result?

As it happens, I want the responsibility of converting tuples into lists to belong to another object. I’d prefer Availability to have only the responsibility of asking the right questions of the persistent data, not also of massaging the results. I’d like to put that responsibility into a Reshaper object. Here’s an expanded test that does that:

    should “use tuple cache to produce a list of animals“ do
      @availability.override(mocks(:tuple_cache, :reshaper))
      during {
        @availability.animals_that_can_be_removed_from_service
      }.behold! {
        @tuple_cache.should_receive(:all_animals).once.
                     and_return([”…tuples-all…“])
        @tuple_cache.should_receive(:animals_still_working_hard_on).once.
                     with(@timeslice.first_date).
                     and_return([”…tuples-work…“])
        @tuple_cache.should_receive(:animals_out_of_service).once.
                     and_return([”…tuples-os…“])
        # New lines
        @reshaper.should_receive(:extract_to_values).once.
                  with(:animal_name, [’…tuples-work…‘], [”…tuples-os…“], [”…tuples-all…“]).
                  and_return([[”working betsy“], [’out-of-service jake‘],
                              [’working betsy‘, ‘out-of-service jake‘,
                              ‘some…‘, ‘…other…‘, ‘…animals‘]])
        @reshaper.should_receive(:alphasort).once.
                  with([’some…‘, ‘…other…‘, ‘…animals‘]).
                  and_return([”…animals“, “…other…“, “some…“])
      }
      assert_equal([”…animals“, “…other…“, “some…“], @result)
    end

It shows that the Availability method calls Reshaper methods which we could see (if we looked) guarantee the properties that we want. But I don’t like this test. The relationship between Availability and Reshaper doesn’t seem to me nearly as fundamental as that between Availability and TupleCache. And I hate the notion that the general notion of “convert a pile of tuples into a sensible list” is made so specific: it will make maintenance harder. And I’m not thrilled (throughout this test) of the way that the human reader must infer claims about the code from the examples.

So how about this?:

   should “use tuple cache to produce a list of animals“ do
      @availability.override(mocks(:tuple_cache))
      during {
        @availability.animals_that_can_be_removed_from_service
      }.behold! {
        @tuple_cache.should_receive(:all_animals).once.
                     and_return([{:animal_name => ‘out-of-service jake‘},
                                 {:animal_name => ‘working betsy‘},
                                 {:animal_name => ‘some…‘},
                                 {:animal_name => ‘…other…‘},
                                 {:animal_name => ‘…animals‘}])
        @tuple_cache.should_receive(:animals_still_working_hard_on).once.
                     with(@timeslice.first_date).
                     and_return([{:animal_name => ‘working betsy‘}])
        @tuple_cache.should_receive(:animals_out_of_service).once.
                     and_return([{:animal_name => ‘out-of-service jake‘}])
      }
      assert_equal([”…animals“, “…other…“, “some…“], @result)
      assert { @result.history.alphasorted }

The last line of the test claims that—at some point in the past—the result list has been “alphasorted”. A list that’s been alphasorted has the properties we want, which we can check by looking at the tests for the Reshaper#alphasort method.

In essence, we check whether at some point in the past the object we’re looking at has been “stamped” with an appropriate description of its properties. Therefore, we don’t have to construct test input that checks the various ways that description can become true - we simply trust earlier tests of what the stamp means.

Here’s code that adds the stamp:

    def result.history()
      @history = OpenStruct.new unless @history
      @history
    end
    result.history.alphasorted = true
    result.freeze

(Notice that I “freeze” the object. In Ruby, that makes the object immutable. That’s in keeping with my growing conviction that maybe programs should consist of functional code sandwiched between carefully-delimited bits of state-setting code.)

Having said all that, I suspect that the original awkwardness in the tests is a sign that I need a different factoring of responsibilities, rather than making up this elaborate solution. But I haven’t figured out what that factoring should be, so I offer the alternative for consideration.

Somewhere around 1985, I introduced Ralph Johnson to a bigwig in the Motorola software research division. Object-oriented programming was around the beginning of its first hype phase, Smalltalk was the canonical example, and Ralph was heavily into Smalltalk, so I expected a good meeting.

The bigwig started by explaining how a team of his had done object-oriented programming 20 years before in assembly language. I slid under the table in shame. Now, it’s certainly technically possible that they’d implemented polymorphic function calls based on a class tag–after all, that’s what compilers do. Still, the setup required to do that was surely far greater than the burden Smalltalk and its environment put on the programmer. I immediately thought that the difference in the flexibility and ease that Smalltalk and its environment brought to OO programming made the two programming experiences completely incommensurable. (The later discussion confirmed that snap impression.)

I suspect the same is true of mocking frameworks. When you have to write test doubles by hand, doing so is an impediment to the steady cadence of TDD. When you write a statement in a mocking framework’s pseudo-language, doing so is part of the cadence. I bet the difference in experience turns into a difference in design, just as Smalltalk designs were different from even the most object-oriented assembler designs (though I expect not to the same extent).

Simplify, simplify, simplify!
– Henry David Thoreau

(A billboard I saw once.)

Part 1: Mocking as a way of removing words

One of the benefits of mocks is that tests don’t have to build up complicated object structures that have nothing essential to do with the purpose of a test. For example, I have an entry point to a webapp that looks like this:

get ‘/json/animals_that_can_be_taken_out_of_service‘, :date => ‘2009-01-01‘

It is to return a JSON version of something like this:

{ ‘unused animals‘ => [’jake‘] }

Jake can be taken out of service on Jan 1, 2009 because he is not reserved for that day or any following day.

In typical object-oriented fashion, the controller doesn’t do much except ask something else to do something. The code will look something like this:

  get ‘/json/animals_that_can_be_taken_out_of_service‘ do
    # Tell the “timeslice” we are concerned with the date given.

    # Ask the timeslice: What animals can be reserved on/after that date?
    # (That excludes the animals already taken out of service.) 

    # Those animals fall into two categories:
    # - some have reservations after the timeslice date. 
    # - some do not.
    # Ask the timeslice to create the two categories.

    # Return the list of animals without reservations. 
    # Those are the ones that can be taken out of service as of the given date. 
  end

If I were testing this without mocks, I’d be obliged to arrange things so that there would be examples of each of the categories. Here’s the creation of a minimal such structure:

  jake = Animal.random(:name => ‘jake‘)
  brooke = Animal.random(:name => ‘brooke‘)
  Reservation.random(:date => Date.new(2009, 1, 1)) do
    use brooke
    use Procedure.random
  end

The random methods save a good deal of setup by defaulting unmentioned parameters and by hiding the fact that Reservations have_many Groups, Groups have_many Uses, and each Use has an Animal and a Procedure. But they still distract the eye with irrelevant information. For example, the controller method we’re writing really cares nothing for the existence of Reservations or Procedures–but the test has to mention them. That sort of thing makes tests harder to read and more fragile.

In constrast to this style of TDD, mocking lets the test ignore everything that the code can. Here’s a mock test for this controller method:

    should “return a list of animals with no pending reservations“ do
      brooke = Animal.random(:name => ‘brooke‘)
      jake = Animal.random(:name => ‘jake‘)

      during {
        get ‘/json/animals_that_can_be_taken_out_of_service‘, :date => ‘2009-01-01‘
      }.behold! {
        @timeslice.should_receive(:move_to).once.with(Date.new(2009,1,1))
        @timeslice.should_receive(:animals_that_can_be_reserved).once.
                   and_return([brooke, jake])
        @timeslice.should_receive(:hashes_from_animals_to_pending_dates).once.
                   with([brooke, jake]).
                   and_return([{brooke => [Date.new(2009,1,1), Date.new(2010,1,1)]},
                               {jake => []}])
      }
      assert_json_response
      assert_jsonification_of(’unused animals‘ => [’jake‘])
    end

There are no Reservations and no Procedures and no code-discussions of irrelevant connections amongst objects. The test is more terse and–I think–more understandable (once you understand my weird conventions and allow for my inability to choose good method names). That’s an advantage of mocks.

Part 2: Dynamic languages let you remove even more irrelevant detail

But I’m starting to think we can actually go a little further in languages like Ruby and Objective-J. I’ll use different code to show that.

When the client side of this app receives the list of animals that can be removed from service, it uses that to populate the GUI. The user chooses some animals and clicks a button. Various code ensues. Eventually, a PersistentStore object spawns off a Future that asynchronously sends a POST request and deals with the response. It does that by coordinating with two objects: one that knows about converting from the lingo of the program (model objects and so forth) into HTTP/JSON, and a FutureMaker that makes an appropriate future. The real code and its test are written in Objective-J, but here’s a version in Ruby:

should “coordinate taking animals out of service“ do
  during {
    @sut.remove_from_service(”some animals“, “an effective date“)
  }.behold! {
    @http_maker.should_receive(:take_animals_out_of_service_route).at_least.once.
                and_return: “some route“
    @http_maker.should_receive(:POST_content_from).once.
                with(:date => ‘an effective date‘,
                     :animals => “some animals“).
                and_return(’post content‘)
    @future_maker.should_receive(:spawn_POST).once.
                  with(’some route‘, ‘post content‘)
  }
end

I’ve done something sneaky here. In real life, remove_from_service will take actual Animal objects. In Objective-J, they’d be created like this:

  betsy = [[Animal alloc] initWithName: ‘betsy‘ kind: ‘cow‘];

But facts about Animals–that, say, they have names and kinds–are irrelevant to the purpose of this method. All it does is hand an incoming list of them to a converter method. So–in such a case–why not use strings that describe the arguments instead of the arguments themselves?

    @sut.remove_from_service(”some animals“, “an effective date“)

In Java, type safety rarely lets you do that, but why let the legacy of Java affect us in languages like Ruby?

Now, I’m not sure how often these descriptive arguments are a good idea. One could argue that integration errors are a danger with mocks anyway, and that not using real examples of what flows between objects only increases that danger. Or that the increase in clarity for some is outweighed by a decrease for others: if you don’t understand what’s meant by the strings, there’s nothing (like looking at how test data was constructed) to help you. I haven’t found either of those to be a problem yet, but it is my own code after all.

(I will note that I do add some type hints. For example, I’m increasingly likely to write this:

    @sut.remove_from_service([”some animals“], “an effective date“)

I’ve put “some animals” in brackets to emphasize that the argument is an array.)

If you’ve done something similar to this, let’s talk about it at a conference sometime. In the next few months, I’ll be at Speakerconf, the Scandinavian Developer Conference, Philly Emerging Tech, an Agile Day in Costa Rica, and possibly Scottish Ruby Conference.

I’ve been working through Growing Object-Oriented Software (henceforth #goos), translating it into Ruby. An annoyingly high percentage of my time has been spent messing with the end-to-end tests. Part of that is due to a cavalcade of incompatibilities that made me fake out an XMPP server within the same process as the app-under-test (named the Auction Sniper), the Swing GUI thread, and the GUI scraper. Threading hell.

But part of it is not. Part of it is because end-to-end tests just are awkward and fragile (which #goos is careful to point out). If such tests are worth it, it’s because some combination of these sources of value outweighs their cost:

• They help clarify everyone’s understanding of the problem to be solved.

• Trying to make the tests run fast, be less fragile, be easier to debug in the case of failure, etc. makes the app’s overall design better.

• They detect incorrect changes (that is, changes in behavior that were not intended, as distinct from ones you did intend that will require the test to be changed to make it an example of the newly-correct behavior).

• They provide a cadence to the programming, helping to break it up into nicely-sized chunks.

In working through #goos so far (chapter 16), the end-to-end tests have not found any bugs, so zero value there. I realized last night, though, that what most bugged me about them is that they made my programming “ragged”–that is, I kept microtesting away, changing classes, being happy, but when I popped up to run the end-to-end test I was working on, it or another one would break in a way that did not feel helpful. (However, I should note that it’s a different thing to try to mimic someone else’s solution than to conjure up your own, so some of the jerkiness is just inherent to learning from a book.)

I think part of the problem is the style of the tests. Here’s one of them, written with Cucumber:

   Scenario: Sniper makes a higher bid, but loses
       Given the sniper has joined an ongoing auction
       When the auction reports another bidder has bid 1000 (and that the next increment is 98)
       Then the sniper shows that it's bidding 1098 to top the previous price
           And the auction receives a bid of 1098 from the sniper

       When the auction closes
       Then the sniper shows that it's lost the auction

This test describes all the outwardly-observable behavior of the Sniper over time. Most importantly, at each point, it talks about two interfaces: the XMPP interface and the GUI. During coding, I found that context switching unsettling (possibly because I have an uncommonly bad short- and medium-term memory for a programmer). Worse, I don’t believe this style of test really helps to clarify the problem to be solved. There are two issues: what the Sniper does (bid in an auction) and what it shows (information about the known state of the auction). They can be talked about separately.

What the Sniper does is most clearly described by a state diagram (as on p. 85) or state table. A state diagram may not be the right thing to show a non-technical product owner, but the idea of the “state of the auction” is not conceptually very foreign (indeed, the imaginary product owner has asked for it to be shown in the user interface). So we could write something like this on a blackboard:

Just as in #goos, this is enough to get us started. We have an example of a single state transition, so let’s implement it! The blackboard text can be written down in whatever test format suits your fancy: Fit table, Cucumber text, programming language text, etc.

Where do we stand?

At this point, the single Cucumber test I showed above is breaking into at least three tests: the one on the blackboard, a similar one for the BIDDING to LOSING transition, and something as yet undescribed for the GUI. Two advantages to that: first, a correct change to the code should only break one of the tests. That breakage can’t be harder to figure out than breaking the single, more complicated test. Second, and maybe it’s just me, but I feel better getting triumphantly to the end of a medium-sized test than I do getting partway through a bigger end-to-end one.

The test on the blackboard is still a business-facing test; it’s written in the language of the business, not the language of the implementation, and it’s talking about the application, not pieces of it.

Here’s one implementation of the blackboard test. I’ve written it in my normal Ruby microtesting style because that shows more of the mechanism.

context “pending state“ do

  setup do
    start_app_at(AuctionSnapshot.new(:state => PENDING))
  end

  should “respond to a new price by counter-bidding the minimum amount“ do
    during {
      @app.receive_auction_event(AuctionEvent.price(:price => 1000,
                                                    :increment => 98,
                                                    :bidder => “someone else“))
    }.behold! {
      @transport_translator.should_receive(:bid).once.with(1098)
      @anyone_who_cares.should_receive_notification(STATE_CHANGE).at_least.once.
                        with(AuctionSnapshot.new(:state => BIDDING,
                                                 :last_price => 1000,
                                                 :last_bid => 1098))
    }
  end
end

Here’s a picture of that test in action. It is not end-to-end because it doesn’t test the translation to-and-from XMPP.

In order to check that the Sniper has the right internal representation of what’s going on in the auction, I have it fling out (via the Observer or Publish/Subscribe pattern) information about that. That would seem to be an encapsulation violation, but this is only the information that we’ve determined (at the blackboard, above) to be relevant in/to the world outside the app. So it’s not like exposing whether internal data is stored in a dictionary, list, array, or tree.

At this point, I’d build the code that passed this test and others like it in the normal #goos outside-in style. Then I’d microtest the translation layer into existence. And then I’d do an end-to-end test, but I’d do it manually. (Gasp!) That would involve building much the same fake auction server as in #goos, but with some sort of rudimentary user interface that’d let me send appropriately formatted XMPP to the Sniper. (Over the course of the project, this would grow into a more capable tool for manual exploratory testing.)

So the test would mean starting the XMPP server, starting the fake auction and having it log into the server, starting the Sniper, checking that the fake auction got a JOIN request, and sending back a PRICE event. This is just to see the individual pieces fitting together. Specifically:

• Can the translation layer receive real XMPP messages?
• Does it hand the Sniper what it expects?
• Does the outgoing translation layer/object really translate into XMPP?

The final question–is the XMPP message’s payload in the right format for the auction server?–can’t really be tested until we have a real auction server to hook up to. As discussed in #goos, those servers aren’t readily available, which is why the book uses fake ones. So, in a real sense, my strategy is the same as #goos’s: test as end-to-end as you reasonably can and plug in fakes for the ends (or middle pieces) that are too hard to reach. We just have a different interpretation of “reasonably can” and “too hard to reach”.

Having done that for the first test, would I do it again for the BIDDING to LOSING transition test? Well, yeah, probably, just to see a two-step transition. But by the time I finished all the transitions, I suspect code to pass the next transition test would be so unlikely to affect integration of interfaces that I wouldn’t bother.

Moreover, having finished the Nth transition test, I would only exercise what I’d changed. I would not (not, not, not!) run all the previous tests as if I were a slow and error-prone automated test suite. (Most likely, though, I’d try to vary my manual test, making it different from both the transition test that prompted the code changes and from previous manual tests. Adding easy variety to tests can both help you stumble across bugs and–more importantly–make you realize new things about the problem you’re trying to solve and the product you’re trying to build.)

What about real automated end-to-end tests?

I’d let reality (like the reality of missed bugs or tests hard to do manually) force me to add end-to-end tests of the #goos sort, but I would probably never have anywhere near the number of end-to-end scenario/workflow tests that #goos recommends (as of chapter 16). While I think workflows are a nice way of fleshing out a story or feature, a good way to start generating tests, and a dandy conversation tool, none of those things require automation.

I could do any number of my state-transition tests, making the Sniper ever more competent at dealing with auctions, but I’d probably get to the GUI at about the same time as #goos.

What do we know of the GUI? We know it has to faithfully display the externally-relevant known state of the auction. That is, it has to subscribe to what the Sniper already publishes. I imagine I’d have the same microtests and implementation as #goos (except for having the Swing TableModel subscribe instead of being called directly).

Having developed the TableModel to match my tests, I’d still have to check whether it matches the real Swing implementation. I’d do that manually until I was dragged kicking and screaming into using some GUI scraping tool to automate it.

How do I feel?

Nervous. #goos has not changed my opinion about end-to-end tests. But its authors are smarter and more experienced than I am. So why do they love–or at least accept–end-to-end tests while I fear and avoid them?

Here’s a test for the Cappuccino app I’m working on. It’s about what happens when, for example, you click on “blood collection for transfusion” in the right table here:

- (void)testPutBackAProcedure
{
  [scenario
   previousAction: function() {
      [self procedure: “Betical“
            hasBeenSelectedFrom: [”alpha“, “Betical“, “order“]];
    }
  during: function() {
      [self putBackProcedure: “Betical“];
    }
  behold: function() {
      [self listenersWillReceiveNotification: ProcedureUpdateNews
            containingObject: []];
      [self tablesWillReloadData];
    }
  andSo: function() {
      [self unchosenProcedureTableWillContain: [”alpha“, “Betical“, “order“]];
      [self chosenProcedureTableWillContain: []];
    }
   ];
}

Here’s the code to pass the test (after inlining one method):

- (void)unchooseProcedure: (id) sender
{
  [self moveProcedureAtIndex: [chosenProcedureTable clickedRow]
                        from: chosenProcedures
                          to: unchosenProcedures];

  [NotificationCenter postNotificationName: ProcedureUpdateNews
                                    object: chosenProcedures];
  [chosenProcedureTable reloadData];
  [unchosenProcedureTable reloadData];
}

Did the test clarify my design thinking? No, not really. Will it be useful for regression? I doubt it. Is it good documentation for the app’s UI behavior? No.

Something seems wrong here.

Something I say about the ideal of Agile design is that, at any moment when you might ship the system, the code should look as if someone clever had designed a solution tailored to do exactly what the system does, and then implemented that design. The history of how the system actually got that way should be lost.

An equivalent ideal for TDD might be that the set of tests for an interoperating set of classes would be an ideal description-by-example of what they do, of what their behavior is. For tests to be documentation, the tests would have to be organized to suit the needs of a learner (most likely from simple to complex, with error cases deferred, and - for code of any size - probably organized thematically somehow).

That is, the tests would have to be more than what you’d expect from a history of writing them, creating the code, rewriting tests and adding new ones as new goals came into view, and so forth. They shouldn’t be a palimpsest with some sort random dump of tests at the top and the history of old tests showing through. (”Why are these three tests like this?” “Because when behavior X came along, they were tests that needed to be changed and it was easiest to just tweak them into shape.”)

I’ve seen enough to be convinced that, surprisingly, Agile design works as described in the first paragraph, and that it doesn’t require superhuman skill. The tests I see - and write - remind me more of the third paragraph than the second. What am I missing that makes true tests-as-documentation as likely as emergent design is?

(It’s possible that I demand too much from my documentation.)

The value of programmer TDD is well established. It’s natural to extrapolate that practice to business-facing tests, hoping to obtain similar value. We’ve been banging away at that for years, and the results disappoint me. Perhaps it would be better to invest heavily in unprecedented amounts of built-in support for manual exploratory testing.

In 1998, I wrote a paper, “When should a test be automated?“, that sketched some economics behind automation. Crucially, I took the value of a test to be the bugs it found, rather than (as was common at the time) how many times it could be run in the time needed to step through it manually.

My conclusions looked roughly like the following:

Scripted tests, be they automated or manual, are expensive to create (first column). Manual scripts are cheaper, but they still require someone to write steps down carefully, and they likely require polishing before they can truly be followed by someone else. (Note: height of bars not based on actual data.)

In the second column, I assume that a particular set of steps has roughly the same chance of finding a bug whether executed manually or by a computer, and whether the steps were planned or chosen on the fly. (I say “roughly” because computers don’t get bored and miss bugs, but they also don’t notice bugs they weren’t instructed to find.)

Therefore, if the immediate value of a test is all that matters, exploratory manual testing is the right choice. What about long-term value?

Assume that exploratory tests are never intentionally repeated. Both their long-term cost and value are zero. Both kinds of scripted tests have quite substantial maintenance costs (especially in that era, when testing was typically done through an unmodified GUI). So, to pull ahead of exploratory tests in the long term, scripted tests must have substantial bug-finding power. Many people at that time observed that, in fact, most tests either found a bug the first time they were run or never found a bug at all. You were more likely to fix a test because of an intentional GUI change than to fix the code because the test found a bug.

So the answer to “when should a test be automated?” was “not very often”.

Programmer TDD changes the balance in two ways:

1. New sources of value are added. Extremely rapid feedback reduces the cost of debugging. (Most bugs strike while what you did to create them is fresh in your mind.) Many people find the steady pace of TDD allows them to go faster, and that the incremental growth of the code-under-test makes for easier design. And, most importantly as it turns out, the need to make tests run fast and reduce maintenance cost leads to designs with good properties like low coupling and high cohesion. (That is, properties that previously were considered good in the long term—but were routinely violated for short-term gain—now had powerful short-term benefits.)

2. Good design and better programmer tools dramatically lowered the long-term cost of tests.

So, much to my surprise, the balance tipped in favor of automation—for programmer tests. It’s not surprising that many people, including me, hoped the balance could also tip for business-facing tests. Here are some of the hoped-for benefits:

• Tests might clarify communication and avoid some cases where the business asks for something, the team thinks they’ve delivered it, and the business says “that’s not what I wanted.”

• They might sharpen design thinking. The discipline of putting generalizations into concrete examples often does.

• Programmers have learned that TDD supports iterative design of interfaces and behavior. Since whole products are also made of interfaces and behavior, they might also benefit from designers who react to partially-finished products rather than having to get it right up front.

• Because businesses have learned to mistrust teams who show no visible progress for eight months (at which point, they ask for a slip), they might like to see evidence of continuous progress in the form of passing tests.

• People often need documentation. Documentation is often improved by examples. Executable tests are examples. Tests as executable documentation might get two benefits for less than their separate costs.

• And, oh yeah, tests could find regression bugs.

So a number of people launched off to explore this approach, most notably with Fit. But Fit hasn’t lived up to our hopes, I think. The things that particularly bother me about it are:

• It works well for business logic that’s naturally tabular. But tables have proven awkward for other kinds of tests.

• In part, the awkwardness is because there are no decent HTML table editors. That inhibits experimentation: if you don’t get a table format right the first time, you’re tempted to just leave it.

  Note: I haven’t tried ZiBreve. By now, I should have. I do include Word, Excel, and their OpenOffice equivalents among the ranks of the not-decent, at least if you want executable documentation. (I’ve never tried treating .doc files as the real tests that are “compiled” into HTML before they’re executed.)

• Fit is not integrated into programmer editors the way xUnit is. For example, you can’t jump from a column name to the Java method that defines it. Partly for this reason, programmers tend to get impatient with people who invent new table formats—can’t they just get along with the old one?

With my graphical tests, I took aim at those sources of friction. If I have a workflow test, I can express it as boxes and arrows:

I translate the graphical documents into ordinary xUnit tests so that I can use my familiar tools while coding. The graphical editor is pretty decent, so I can readily change tests when I get better ideas. (There are occasional quirks where test content has changed more than it looks like it has. That aspect of using Fit hasn’t gone away entirely.)

I’ve been using these tests, most recently on wevouchfor.org—and they don’t wow me. While I almost always use programmer TDD when coding (and often regret skipping it when I don’t), TDD with these kinds of tests is a chore. It doesn’t feel like enough of the potential value gets realized for the tests to be worth the cost.

• ↓ Writing the executable test doesn’t help clarify or communicate design. Let me be careful here. I’m a big fan of sketching things out on whiteboards or paper:

  

  That does clarify thinking and improve communication. But the subsequent typing of the examples into the computer is work that rarely leads to any more design benefits.

• ↔ Passing tests do continuously show progress to the business, but… Suppose you demonstrate each completed story anyway, at an end-of-iteration demo or (my preference) as soon as it’s finished. Given that, does seeing more tests pass every day really help?

• ↑ Tests do serve as documentation (at least when someone takes the time to surround them with explanatory text, and if the form and content of the test aren’t distorted to cram a new idea into existing test formats).

• ↑ The word I’m hearing is that these tests are finding bugs more often than I expected. I want to dig into that more: if they’re the sort of “I changed this thing over here and broke that supposedly unrelated thing over there” bugs that whole-product regression tests are traditionally supposed to find, that alone may justify the expense of test automation—unless I can find a way to blame it on inadequate unit tests or a need to rejigger the app.

• ↓ (This is the one that made me say “Eureka!”) Tests alone fail at iterative product design in an interesting way. Whenever I’ve made significant progress implementing the next chunk of workflow or other GUI-visible change, I just naturally check what I’ve done through the GUI. Why? This checking makes new bugs (ones the automated tests don’t check for) leap out at me. They also sometimes make me slap my forehead and say, “What I intended here was stupid!”

But if I’m going to be looking at the page for both bugs and to change my intentions, I’m really edging into exploratory testing. Hmm… What if an app did whatever it could to aid exploratory testing? I don’t mean traditional testability features like, say, a scripting interface; I mean a concerted effort to let exploratory testers peek and poke at anything they want within the app. (That may not be different than my old motto “No bug should be hard to find the second time,” but it feels different.)

So, although features of Rails like not having to restart the server after most code changes are nice, I want more. Here’s an example.

The following page contains a bug:

Although you can’t see it, the bottom two links are wrong. They are links to /certifications/4 instead of /promised_certifications/4.

1. Unit tests couldn’t catch that bug. (The two methods that create those types of links are tested and correct; I just used the wrong one.)

2. One test of the action that created the page could have caught the bug, but did not. (To avoid maintenance problems, that test checked the minimum needed to convince me that the correct “certifications” had been displayed. I assumed that if they were displayed at all, the unit tests meant they were displayed correctly. That was actually almost right—every character outside the link’s href value was correct.)

3. I missed the bug when I checked the page. (I suspect that I did click one of the links, but didn’t notice it went to the wrong place. If so, I bet I missed the wrongness because I didn’t have enough variety in the test data I set up—ironic, because I’ve been harping on the importance of “irrelevant” variety since 1994.)

4. A user had no trouble finding the bug when he tried to edit one of his promised certifications and found himself with a form for someone else’s already-accepted certification. (Had he submitted the form, it would have been rejected, but still.)

That’s my bug: a small error in a big pile of HTML the app fired and forgot.
Suppose, though, that the app created and retained an object representing the page. Suppose further that an exploration support app let you switch to another view of that object/page, one that highlights link structure and downplays text:

To the eyes of someone who just added promised certifications to that page, the wrong link targets ought to jump out.

There’s more that I’d like, though. The program knows more about those links than it included in the HTTP Response body. Specifically, it knows they link to a certain kind of object: a PromisedCertification. I should be able to get a view of that object (without committing to following the link). I should be able to get it in both HTML form and in some raw format. (And if the link-to-be-displayed were an object in its own right, I would have had a place to put my method, and I wouldn’t have used the wrong one. Testability changes often feed into error prevention.)

And so on… It’s easy enough for me to come up with a list of ways I’d like the app to speak of its internal workings. So what I’m thinking of doing is grabbing some web framework, doing what’s required to make it explorable, using it to build an app, and also building an exploration assistant in RubyCocoa (allowing me to kill another bird with this stone).

To be explicit, here’s my hypothesis:

An application built with programmer TDD, whiteboard-style and example-heavy business-facing design, exploratory testing of its visible workings, and some small set of automated whole-system sanity tests will be cheaper to develop and no worse in quality than one that differs in having minimal exploratory testing, done through the GUI, plus a full set of business-facing TDD tests derived from the example-heavy design.

We shall see, I hope.