Exploration Through Example

Example-driven development, Agile testing, context-driven testing, Agile programming, Ruby, and other things of interest to Brian Marick
191.8 167.2 186.2 183.6 184.0 183.2 184.6

Mon, 09 Jan 2006

Working your way out of the automated GUI testing tarpit (part 6)

part 1, part 2, part 3, part 4 part 5

Here, I dispose of another reason to run tests through the GUI: bad links and other ways of getting to pages. These bugs can be found with unit tests instead. The mechanism fits in well with business-facing test-driven design.

Let's start with a bug. In build 343, an Activity Summary page is added to the app. Links to that page are added to thirteen other pages. In build 582, someone changes the URL of the Activity Summary page and dutifully changes twelve of the thirteen pages that link to it. It's a user who finds that the thirteenth link wasn't updated.

A link-checking program won't find all such bugs because it probably can't get to all the pages of the program. So, the claim is, you should have a GUI testing tool traverse every link. Here, I'll change the sample app to show a better way.

Because I was frightened by DTML as a small child, I lean away from template languages with embedded code and toward code that generates XHTML. (We can argue the merits of the two approaches another day.)

My Renderer class is nothing fancy. A bunch of core methods generate simple XHTML. From them, I've built up more complicated methods, such as the ones used here:

   def case_display_page
      case_record = @app.current_record
      page("Case #{case_record.clinic_id}",
               p("Owner: #{case_record.client}"),

Now suppose I want to add a help link to that page, using a method called help_link_for(topic). Here's a simple implementation of that method:

   def help_link_for(topic)
      %Q{<a href="javascript:standard_popup('help?topic=#{topic}')">Help</a>}

The method generates a link to a javascript popup, but I think it should also check that the topic exists, like this:

   def help_link_for(topic)
      assert(@app.has_help_for?(topic), "Creating link to nonexistent link #{topic}.")
      %Q{<a href="javascript:standard_popup('help?topic=#{topic}')">Help</a>}

has_help_for? checks that the topic exists, using the same mechanism that the help action uses to find the help to display. Therefore, you do not need to follow the link to discover that it's bad, you merely need to generate it. Which means generating the page that contains it. Which we already do with a fast renderer unit test:

   def test_typical_case_display_page
      given_app_with {
         case_record('clinic_id' => 19600219)
      when_rendering(:case_display_page) {
         assert_page_title_matches(/^Case 19600219/)

The test doesn't explicitly check the help link, but it doesn't have to: the renderer assertion will nevertheless check it for us. Here's what will happen if the link is bad:

   2) Error:
StandardError: Programmer error. Creating link to nonexistent page "bogus_page". Please report this error to bugs@example.com.
      ./util.rb:3:in `assert'
      ./renderer.rb:106:in `help_link_for'
      ./renderer.rb:82:in `case_display_page'

(Note: I later added an explicit assertion that the help link exists because I consider it an essential part of the page. The implicit check only fails if the link exists but is bad; the explicit assertion fails if it doesn't exist at all.)

The link-creation routine checks that the particular help topic exists, but it doesn't check that "help" is the right action to get to the help pages. It's easy to ask if the app responds to an action named help. Use this code: @app.respond_to?('help'). So I could add another assertion to help_link_for, but I'd like to handle the risk of an incomplete renaming in a different way. To get there, let me start a seeming digression and fix that long-standing bug in our program (that it prompts you with a button to add an audit even when no more audits are allowed).

Here's the code that adds the button to the page:

   def add_audit_button
                                    submit('Add an Audit Record')))

The renderer could ask the app before generating the add_audit form, like this:

   def add_audit_button
      return unless @app.further_audits_allowed?

                                    submit('Add an Audit Record')))

And, since I'm changing the method anyway, I might as well have it make sure that want_add_audit_form is an action the app responds to:

   def add_audit_action
      assert(@app.respond_to?('want_add_audit_form'), ...)
      return unless @app.further_audits_allowed?

                                    submit('Add an Audit Record')))

But that's starting to bug me. I'm asking the App more and more, not telling it. Is this Feature Envy? Do I want to worry that other methods that generate this action will have to duplicate the knowledge of which checks are appropriate?

It seems to me that the renderer should hand a potential presentation to the app and ask it to apply whatever rules are relevant, but in a way that insulates the app from any knowledge of the presentation (that it'll be in XHTML, etc.). That can be done using a closure as a callback:

   def add_audit_button
      @app.fill(:template_for_want_add_audit_form) { | action |
                                          submit('Add an Audit Record')))

The App would look like this:

   def fill(template_name, *args, &block)
      self.send("fill_#{name}", *args, &block)

   def fill_template_for_want_add_audit_form(&block)
      return unless current_record.accepts_more_audits?


   def checked(action_name)
                 "#{action_name} is not a defined action.")

fill bounces the work off to a particular method. That checks whether the action is allowed by the business presentation rules. If not, it returns nil (which renders as nothing). Otherwise, it passes the correct action name to the closure (after checking that no one's renamed it out from under us) and lets that closure render away.

(Note: the renderer could call fill_template_for_want_add_audit_form directly—the same knowledge is required—but this form seemed more convenient for unit tests.)

This division of responsibility works well with test-driven design.

  1. The customer says "You shouldn't be able to add any audits if the last audit was nominal." After discussion, everyone agrees the story is to leave the "Add Audit" button off the Case Display page and update that page's help with an explanation.

  2. There's an existing test that checks everything important about the Case Display page. (It's test_typical_case_display_page, above.) A new test is written that claims the Add Audit button is missing when the last audit is nominal. Like test_typical_case_display_page, it avoids fiddly details of XHTML structure.

  3. Making the test pass is going to require some new business logic. That leads to three unit tests describing how fill responds when client code asks it to fill in a template_for_add_audit_form:

    • if there are no audits, the template is filled in (with the right action name),
    • if there's an audit with nominal variance, nil is returned instead of the filled-in template, and
    • if there are n audit records (none nominal), the template is filled in.

    None of these tests refer to text at all, much less XHTML text.

  4. Those tests are made to pass.

  5. The original test should now pass. If it doesn't, that means the renderer doesn't call the app to judge the template. How is this possible, since it's supposed to always use this mechanism to get the action name? Bad renderer! But easily fixed.

  6. The story's not done until the Customer sees the new version of the Case Display page, probably by walking through the workflow of creating a nominal audit and then observing that there's no button to create another. That might lead to tweaks of the presentation, especially those aspects not important enough to be described in a test.

  7. If the Customer wants, the same business rule can be used to check incoming actions. (Just because we don't provide a form to let people add to nominal audits doesn't mean that someone couldn't send the appropriate HTTP anyway.)

(As usual, I should note that I have not seen these ideas applied at the scale of a real app. If I ever have time to create a Giant Microbes fan site for my kids, I'll explore them further.)

At long last returning to the help popup, I can change the code that generates the link to this:

   def help_link_for(topic)
      @app.fill(:template_for_help_link, topic) { | action |
            %Q{<a href="javascript:standard_popup('#{action}?topic=#{topic}')">Help</a>}

The App code that would rule on the template would be:

   def fill_template_for_help_link(topic, &block)
                "Creating link to nonexistent help topic '#{topic}'.")

Any unit test that generated a help link would auto-check for a bad action or bad topic. It would not check whether the javascript standard_popup routine pops up a window, pops up a reasonably-sized window, pops it up somewhere not annoying, etc. That could be tested with JsUnit, Watir, or Selenium. Personally, I'd just test it by hand and trust myself to retest it if I change it.

One final note: we are still working our way out of the tarpit. I haven't stressed it in this installment, but both of the old-format tests continue to work. As always, the goal is to gradually reduce the need for slow and fragile tests.

See the code for complete details.

## Posted at 11:51 in category /testing [permalink] [top]

About Brian Marick
I consult mainly on Agile software development, with a special focus on how testing fits in.

Contact me here: marick@exampler.com.




Agile Testing Directions
Tests and examples
Technology-facing programmer support
Business-facing team support
Business-facing product critiques
Technology-facing product critiques
Testers on agile projects

Permalink to this list


Working your way out of the automated GUI testing tarpit
  1. Three ways of writing the same test
  2. A test should deduce its setup path
  3. Convert the suite one failure at a time
  4. You should be able to get to any page in one step
  5. Extract fast tests about single pages
  6. Link checking without clicking on links
  7. Workflow tests remain GUI tests
Permalink to this list


Design-Driven Test-Driven Design
Creating a test
Making it (barely) run
Views and presenters appear
Hooking up the real GUI


Popular Articles
A roadmap for testing on an agile project: When consulting on testing in Agile projects, I like to call this plan "what I'm biased toward."

Tacit knowledge: Experts often have no theory of their work. They simply perform skillfully.

Process and personality: Every article on methodology implicitly begins "Let's talk about me."


Related Weblogs

Wayne Allen
James Bach
Laurent Bossavit
William Caputo
Mike Clark
Rachel Davies
Esther Derby
Michael Feathers
Developer Testing
Chad Fowler
Martin Fowler
Alan Francis
Elisabeth Hendrickson
Grig Gheorghiu
Andy Hunt
Ben Hyde
Ron Jeffries
Jonathan Kohl
Dave Liebreich
Jeff Patton
Bret Pettichord
Hiring Johanna Rothman
Managing Johanna Rothman
Kevin Rutherford
Christian Sepulveda
James Shore
Jeff Sutherland
Pragmatic Dave Thomas
Glenn Vanderburg
Greg Vaughn
Eugene Wallingford
Jim Weirich


Where to Find Me

Software Practice Advancement


All of 2006
All of 2005
All of 2004
All of 2003



Agile Alliance Logo