Exploration Through Example

Example-driven development, Agile testing, context-driven testing, Agile programming, Ruby, and other things of interest to Brian Marick
191.8 167.2 186.2 183.6 184.0 183.2 184.6

Thu, 08 Dec 2005

Working your way out of the automated GUI testing tarpit (part 1)

In this series, I'll present two ideas that have been percolating in my head for a while. Last week, I began thinking they might be appropriate for a client. We ended up taking a different approach, but not until after I'd spent an evening building a prototype. Yesterday, I was so sick of replying to mail, chipping away at a task backlog that's metastasized during recent travel, and slogging through other things I really ought to be doing that I rebelled and decided to rewrite the prototype. It was fun.

The general idea here is (1) to gradually work your way toward declarative tests that generate their own page navigation and (2) to use caching to speed up tests and maybe improve program structure.

I've never tried these ideas for real. They might be impractical in the wild.

Here are three GUI-oriented tests, in increasing order of goodness. The scenario has something to do with a veterinary clinic (of course). In each test, a case record is created, an animal visit is recorded, and an audit record is appended. (All the steps are necessary, because you can't record a visit until there's a case, and you can't create an audit record until there's a visit.) Normally, there can be multiple audits attached to a case. But if the first audit is marked as "nominal", it's the only one that can ever be created. If so, there should be no "Add Audit" button on the Case Management page. That's what the test checks. (It also uses the title of the page to make sure the assertion is checking the right page.)

The first test is like one you might get from a straightforward use of Watir or jWebUnit.

  def test_cannot_append_to_a_nominal_audit

    enter(:login, 'unimportant')
    enter(:password, 'unimportant')

    press('New Case')

    enter(:client, 'unimportant')
    enter(:clinic_id, '213')
    press('Record Case')

    press('Add Visit')

    enter(:diagnosis, 'unimportant')
    enter(:charges, '100')
    press('Record Visit')

    press('Add Audit')

    enter(:auditor, 'unimportant')
    enter(:variance, 'nominal')
    press('Record Audit')

    assert_page_title('Case Management')
    assert_page_has_no_button_labeled('Add Audit')

What are the problems with this test?

  • In all this code, what's important? Only two lines, which I've highlighted so you can find them easily. In real life, the important lines aren't in bold blue font, so such tests are hard to read.

  • The test is fragile in the face of change. Change the name of a field, introduce another field that has to be filled in, split a page in two: all of these will break this test and many, many others besides. Now you get to fix them all. Because they're hard to read, it's easy to fix them badly. (There are a lot of tests out there that inadvertently no longer test what they're supposed to test.)

  • The test is likely to be slow, because it drives a browser. Programmers who are used to a fast test-code-refactor cycle won't put up with that. So the tests will be run infrequently, and they'll provide information well after it'd be most valuable.

To solve the problem of fragility, some people put a library between the tests and the browser. Here's what such a test would look like:

  def test_cannot_append_to_a_nominal_audit

    login('unimportant', 'unimportant')
    new_case('unimportant', '213')
    new_visit('unimportant', '100', nil)
    new_audit('unimportant', 'nominal')

    assert_page_title('Case Management')
    assert_page_has_no_button_labeled('Add Audit')
  • The test is easier to read, but it has some problems. The fact that an audit record exists is essential to the test, whereas the existence of a visit is incidental. Yet they're given equal prominence. The use of the "unimportant" token makes the use of "nominal" stand out - that particular value must be important to this test. But what about "213" and "100"? They're not important, but there's no convenient "ignore this value" token for numbers.

  • It is more resistant to change than the previous test. If there are changes within a page, you might only have to change one library method.

    But other changes can still break a bunch of tests. In the next iteration, suppose an FDA contact record has to be added before an audit can happen. That means every test that goes directly from adding a visit to adding an audit record will become broken. Either you fix all the tests or you change new_visit to silently add an FDA contact record - which I guarantee will make for some frustrating debugging down the road.

  • It's just as slow as the previous version.

I believe such a test is still not good enough. It's still procedural - it's still of the form "do this... now this... now this... finally you can check what you care about." Here's a better test:

  def test_cannot_append_to_a_nominal_audit
    @browser.as_our_story_begins {
       we_have_an_audit_record_with(:variance => 'nominal')

    assert_page_has_no_button_labeled('Add Audit')
  • This test is declarative. It says that there must be a case with an audit record, but it doesn't say how that record's created. Moreover, it strives to be minimal, to use no word unless it's clearly related to the intention of the test. It says nothing about any of the fields that the previous tests described as "unimportant". It's even silent on the existence of case records and visits, simply assuming that whatever's required for there to be an audit record has happened. (Presumably, requirements like "you can't add an audit record unless there's been a visit" have been tested elsewhere.) All of this makes the test still easier to read.

  • The test is even more resistant to change. Because there's no sequence of steps in the test - no workflow - changes to the workflow will require localized changes in the support code, not to the tests themselves.

  • However, the test is still just as slow as the other ones, so there's room yet for improvement.

In the next installment, I'll show what the code behind the scenes looks like. Right now, I want to emphasize that all three tests do the same thing. Here's an execution log for the third test:

$ ~/src/procedural2declarative 601 $ ruby declarative-test.rb
Loaded suite declarative-test
Go to <http://app.com/app>
Enter "unimportant" into field :login
Enter "unimportant" into field :password
Press "Login"

Press "New Case"

Enter "unimportant" into field :client
Enter "213" into field :clinic_id
Press "Record Case"

Press "Add Visit"

Enter "unimportant" into field :diagnosis
Enter "100" into field :charges
Press "Record Visit"

Press "Add Audit"

Enter "nominal" into field :variance
Enter "unimportant" into field :auditor
Press "Record Audit"

Finished in 0.005032 seconds.

1 tests, 1 assertions, 0 failures, 0 errors

## Posted at 08:02 in category /testing [permalink] [top]

About Brian Marick
I consult mainly on Agile software development, with a special focus on how testing fits in.

Contact me here: marick@exampler.com.




Agile Testing Directions
Tests and examples
Technology-facing programmer support
Business-facing team support
Business-facing product critiques
Technology-facing product critiques
Testers on agile projects

Permalink to this list


Working your way out of the automated GUI testing tarpit
  1. Three ways of writing the same test
  2. A test should deduce its setup path
  3. Convert the suite one failure at a time
  4. You should be able to get to any page in one step
  5. Extract fast tests about single pages
  6. Link checking without clicking on links
  7. Workflow tests remain GUI tests
Permalink to this list


Design-Driven Test-Driven Design
Creating a test
Making it (barely) run
Views and presenters appear
Hooking up the real GUI


Popular Articles
A roadmap for testing on an agile project: When consulting on testing in Agile projects, I like to call this plan "what I'm biased toward."

Tacit knowledge: Experts often have no theory of their work. They simply perform skillfully.

Process and personality: Every article on methodology implicitly begins "Let's talk about me."


Related Weblogs

Wayne Allen
James Bach
Laurent Bossavit
William Caputo
Mike Clark
Rachel Davies
Esther Derby
Michael Feathers
Developer Testing
Chad Fowler
Martin Fowler
Alan Francis
Elisabeth Hendrickson
Grig Gheorghiu
Andy Hunt
Ben Hyde
Ron Jeffries
Jonathan Kohl
Dave Liebreich
Jeff Patton
Bret Pettichord
Hiring Johanna Rothman
Managing Johanna Rothman
Kevin Rutherford
Christian Sepulveda
James Shore
Jeff Sutherland
Pragmatic Dave Thomas
Glenn Vanderburg
Greg Vaughn
Eugene Wallingford
Jim Weirich


Where to Find Me

Software Practice Advancement


All of 2006
All of 2005
All of 2004
All of 2003



Agile Alliance Logo