Exploration Through Example

Example-driven development, Agile testing, context-driven testing, Agile programming, Ruby, and other things of interest to Brian Marick
191.8 167.2 186.2 183.6 184.0 183.2 184.6

Fri, 05 Dec 2003

Testing language

Martin Fowler writes (quoted in its entirety):

I'm currently sitting in a session at XP day where Owen Rogers and Rob Styles are talking about the differences between XP's unit and acceptance tests. This triggered a thought in my mind - what should a language for writing acceptance tests be?

Commercial tools for UI testing tend to have their own proprietary language. Brett Pettichord, amongst others, question this; preferring a common scripting language such as Ruby or Python.

But I wonder if a language designed for programming is really the right language for writing tests. The point about tests is that they operate by example. They don't try to cover how to handle any value, instead they describe specific scenarios and responses. I wonder if this implies a different kind of programming language is required. Perhaps this is the truly startling innovation in FIT.

Acceptance tests are things to point at while having a conversation. The speakers will be testers, programmers, and business experts. The neatest surprise with FIT, to my mind, is the way you can turn a web page of tests into a narrative. I've mislaid my good example of this, but here's a snippet of page about an example Ward often uses: a bond trading rule that all months should be treated as having 30 days. (Note: I don't know if the following is correct - I was just playing around with FIT, not writing a real app.)

The end of the month is another special case. The last day always counts as day 30, even though that means a jump between the next-to-last day and it.

from to actual() thirty360()
Feb 27, 2003 Mar 1, 2003 2 4
Feb 28, 2003 Mar 1, 2003 1 1
Feb 28, 1996 March 1, 1996 2 3
Feb 29, 1996 March 1, 1996 1 1

Days past day 30 in a month just count as 30.

from to actual() thirty360()
Jan 29, 1950 Feb 1, 1950 3 2
Jan 30, 1950 Feb 1, 1950 2 1
Jan 31, 1950 Feb 1, 1950 1 1

The page is organized as set of special cases, each with a brief description followed by some checked examples. I really like this style: it's an easy-to-write mini-tutorial for the programmer and reader.

However, when it comes to tests that are sequences (action fixtures), I find that the tabular format grates. The table, instead of being a frame around what I read, is something that intrudes on my reading. It may be that I'm too used to code.

Another difficulty is variables: such useful beasts. Think of constructing a circular data structure. Yes, you can write a parseable language to do that (Common Lisp had one, as I recall), but to my mind it's simpler just to create substructures, give them names, and use the names to lash things together.

Or consider stashing results in a variable and then doing a set of checks. Or comparing two results to each other, where the issue isn't exact values but the relationships between values.

You could invent variables in the FIT language, but you're starting to get into notational problems that scripting languages have already solved. That way lies madness (after inventing loops and subroutines and...)

And yet, there is the niceness of HTML format. I've been toying with the idea of a free-form action-ish fixture that looked like this:

start 'stqe'
check records.size, 1
forget 1
check records.size, 0

Running the test might yield this:

start 'stqe'
check records.size, 1
forget 1
check records.size, 0

0 expected
stack trace

Here, I'm following RoleModel's lead in using Ruby as a somewhat customer-readable language. (<rant>And I also fixed the backwards order of xUnit assert arguments. Few English speakers say, "check that nice is the weather" instead of "check that the weather is nice". When I'm talking to people as I write an assert, it's really awkward that the required typing order isn't the natural order of speaking or thinking - first you think about what you're going to check, then you think about what the value should be. It's too late to fix this in xUnit, but we can do better for our customers.</rant>)

This format also allows for a mixture of scripting and tabular styles. I have some tests that look like this:

# A simple start->stop
start mail # record 1
                        assert_states [mail], []
                        assert_new_record 1, mail, mail_start
stop mail  # record 1 stops
                        assert_states [], []
                        assert_stopped_with(1, 1.minute)

You'll notice that I indent the asserts. That's because the tests (especially the longer ones) do a lot of starting, stopping, and pausing of various interacting jobs. It's too hard to see what's going on if the the asserts aren't visually distinguished from the actions.

Nevertheless, it's still pretty ugly. I showed it to a customer-type (my wife, who knows no programming languages). She understood it fine after I explained it, but it didn't thrill her. This might be better:

step running job paused jobs extra checks
start mail
mail none
check records.last.number, 1
check records.last.job, mail
check records.last.start_time, mail_start
stop mail
none none
check records.last.number 1
check records.last.elapsed_time, 1.minute

The fixture synthesizes all the cells into one Ruby method. Is this better? (I'm not a particularly gifted visual designer, as anyone who's looked at my main website can tell.)

A final thought. I've read a couple of papers on intentional programming. (Can't find any good links.) It didn't click for me. I think it was the examples. One example had a giant embedded in some C code. Stacking and n and C are simultaneously variables in the surrounding function and mathematical variables in the equation. The idea of intentional programming is that there's one underlying representation (our friend, the syntax tree). Different bits of it get converted into the most suitable representation. So the mathematical bit gets turned into an equation1.

The thing is, I found the mixture of C and mathematical notation jarring. Is the equation really clearer - to a programmer immersed in the app - than
Stacking = sum(0, n, C) / det(C)? The thing is, here we have one person reading two styles. I'm not sure that the required switching between styles isn't more cognitive trouble than it's worth.

But when we're talking about tests, we have two people who have very different cognitive styles, goals, and backgrounds reading one text. I can envision a common underlying test representation that can be switched between two modes. One is the "discuss the test with the customer" mode. The other is the "see the test as executable code" mode.

Perhaps Intentional Software will take up Martin's challenge. (If so, my consulting fees are quite modest...)

1 The equation shown isn't the one used in the paper, not unless there's been an amazing coincidence.

## Posted at 16:41 in category /testing [permalink] [top]

About Brian Marick
I consult mainly on Agile software development, with a special focus on how testing fits in.

Contact me here: marick@exampler.com.




Agile Testing Directions
Tests and examples
Technology-facing programmer support
Business-facing team support
Business-facing product critiques
Technology-facing product critiques
Testers on agile projects

Permalink to this list


Working your way out of the automated GUI testing tarpit
  1. Three ways of writing the same test
  2. A test should deduce its setup path
  3. Convert the suite one failure at a time
  4. You should be able to get to any page in one step
  5. Extract fast tests about single pages
  6. Link checking without clicking on links
  7. Workflow tests remain GUI tests
Permalink to this list


Design-Driven Test-Driven Design
Creating a test
Making it (barely) run
Views and presenters appear
Hooking up the real GUI


Popular Articles
A roadmap for testing on an agile project: When consulting on testing in Agile projects, I like to call this plan "what I'm biased toward."

Tacit knowledge: Experts often have no theory of their work. They simply perform skillfully.

Process and personality: Every article on methodology implicitly begins "Let's talk about me."


Related Weblogs

Wayne Allen
James Bach
Laurent Bossavit
William Caputo
Mike Clark
Rachel Davies
Esther Derby
Michael Feathers
Developer Testing
Chad Fowler
Martin Fowler
Alan Francis
Elisabeth Hendrickson
Grig Gheorghiu
Andy Hunt
Ben Hyde
Ron Jeffries
Jonathan Kohl
Dave Liebreich
Jeff Patton
Bret Pettichord
Hiring Johanna Rothman
Managing Johanna Rothman
Kevin Rutherford
Christian Sepulveda
James Shore
Jeff Sutherland
Pragmatic Dave Thomas
Glenn Vanderburg
Greg Vaughn
Eugene Wallingford
Jim Weirich


Where to Find Me

Software Practice Advancement


All of 2006
All of 2005
All of 2004
All of 2003



Agile Alliance Logo