The tenth time you say it, decide that it’s wrong

That title seems like a good motto in general. Here’s a specific instance.

I was on a short consulting trip recently. We talked about Fit. I said the two things that I’ve said to clients at least ten times:

“I recommend using Fit when the product director wants to read the tests or when the data is naturally tabular. If the product director doesn’t care or if the test is a do-this-do-that-check-this type test, I’d rather you used xUnit.”

“There are no decent HTML editors. The most tolerable I’ve found is OpenOffice, but they’re all a pain when you want to modify tables.”

There’s a contradiction here. If the product director doesn’t care, why would you write a tabular test in HTML? Because using xUnit means you have to keep columns aligned, and that’s a pain. So use HTML and have the browser line up the columns. But I just said that editing HTML is also a pain. And I never gave much thought to which is the greater pain.

Well, now I have. Here’s a set of tests for business rules similar to the ones the client uses (obfuscated so that the company can’t be identified):

require "set-standalone-test-paths.rb" unless $started_from_rakefile
require ‘test/util/product-specific/seat-allocation-detailer’
class SeatAllocationTests < SeatAllocationDetailer
  def test_airlines_can_increase_allocation_if_allowed
    header   ’current allocation’,   ’airline suggestion’, ’our new allocation?’

    check       11,                      20,                    20
    check       22,                      5,                     :unchanged
  def test_airlines_cannot_change_allocation_unless_allowed
      header   ’current allocation’,   ’airline suggestion’, ’our new allocation?’ 

      check       11,                      20,                    :unchanged
      check       22,                      5,                     :unchanged
  def test_airline_can_reduce_and_increase_our_allocation
      header   ’current allocation’,  ’seats sold’, ’airline suggestion’, ’our new allocation?’ 
      check       11,                      5,            5,                 5 
      check       22,                      6,            5,                 :unchanged
      check       33,                      7,           44,                 44 

I’ve written a pile of code to make those tests pass, doing some fiddling with them along the way. Subjectively, making a small allowance for bias, I’m thinking it’s been about the same pain as editing HTML. If I wasn’t a complete novice at TextMate, it would probably be less. (Surely TextMate must have an equivalent of a Back button so that you can flip between files easily even if you haven’t dragged the tabs to be next to each other.)

I even think this test isn’t horribly unreadable for a non-technical reader once you tell her to ignore the starting and trailing gobbledeegook, underscores, and quotes. (And it wouldn’t be too hard to write a script that did that for her, or even one that generated HTML.)

The output of a failing test is much less appealing to a product director, but it’s more appealing to someone actually working with that output (a programmer or tester): it’s got a stack trace, probably one that’s clickable.

This approach would encourage people to write test fixtures in Ruby, which is considerably faster than doing it in Java or C#. And it makes fixture-writing accessible to the well-read tester. In this particular case, the app communicates with the outside world via XML over the network, FTP files, and a database, so there’s no need to link the fixtures into the code. If it worked through a GUI, some remote procedure call interface would have to be added, but I persist in thinking that’s not too much to ask.

If anyone wants to pursue this approach on real code, let me know. You’ll get a 50% break on my consulting rate. I make the same offer about my prototype graphical tests.

7 Responses to “The tenth time you say it, decide that it’s wrong”

  1. JeffreyFredrick Says:


    I would also think that in a decent editor (like emacs ;)) you could have a mode that would make it far easier to keep the indentation right automatically.

  2. Brian Marick Says:

    I think M-x picture-mode is supposed to do that, but I always mess things up just as bad.

  3. john hume Says:

    Re TextMate’s back, the Go to File window lists recently edited files in reverse chronological order, so Command-t, Return is what you want.

  4. KevinRutherford Says:

    Spreadsheets are a lot easier to edit than HTML, and I believe Fit can read Excel - would that help?

  5. Brian Marick Says:

    I only tried Fit-with-excel once years ago, so I have no reasonable opinion. My unreasonable opinion is that it’s excellent for test input, maybe excellent for test editing, OK for test display, but as weak as HTML for the back end (making the program pass, and keep passing, the test).

  6. Exploration Through Example » Blog Archive » MoB-FAT 1: Controller channels Says:

    […] The sample application is only a realistic sketch. It pretends to be the server part of a system used by travel agents. For my own convenience, I wrote the application in Ruby, but I’m treating it just as I would were it in Java. The tests are written in Ruby for the reasons I gave earlier. […]

  7. Curt Sampson Says:

    As a programmer, i’ve always hated the lack of flexability in FIT: it seemed hard to use because it was hard to make sophisticated tests that still had a simple and readable “top level.”

    I’ve been using Ruby for web testing for ages, and I feel that it’s no harder to read, even for non-techies, once you get over the “it’s not in Excel or Word” psychological barrier.

    The more user-oriented tests I’ve done in this mold are much like yours, but with “higher-level” support functions, and oriented toward being more “englishy.” E.g., when a customer required he had google analytics working on certain pages:

    def test_google_analytics_tracking_code_exists
    ].each {
    |uri| assert_analytics_code_present(uri)

    Do the footers come out in the right language, and have the right data?

    def test_footer_en
    assert_get ‘/en/page/top’
    assert_body_match ‘日本語
    assert_body_match ‘top
    assert_body_match ‘XXX Co. Copyright 2006′

    def test_footer_ja
    assert_get_response ‘/ja/page/top’
    assert_body_match ‘English
    assert_body_match ‘top
    assert_body_match ‘XXX Co. Copyright 2006′

    Sometimes I name all the pages, so you can do things such as add parameters, post, and make sure that you went to the “main” page or “failed-login” page. For web sites with databases, there are routines to assert that a user is or isn’t in the db, and so on, so that you can test registration.

    All this is dead easy in ruby, and if you hide all the support stuff in files in another directory, you can aim a business guy a the folder full of “high level” tests, associate the extensions with a simple text editor, and he can easily examine the current tests at any time.

    He still needs to work witha programmer to change the tests, but for other reasons (particularly deciding what to test and how to test it in the first place), he’s going to have to do that anyway.

Leave a Reply

You must be logged in to post a comment.