Exploration Through Example

Example-driven development, Agile testing, context-driven testing, Agile programming, Ruby, and other things of interest to Brian Marick
191.8 167.2 186.2 183.6 184.0 183.2 184.6

Sun, 13 Nov 2005

Throwing tests away

In part of "When should a test be automated?", I look at tests that have broken because the intended behavior of the code changed. My urge is to fix them, but I stop myself and ask a question: if this test didn't exist in the first place, would I bother to write it? If not, I shouldn't bother to fix it. I should just delete it.

Here's an example where that practice would have led me astray.

I've been hacking away at the code in RubyFit that parses HTML. I'm changing it so that it will support Rick Mugridge's FitLibrary. To that end, I created a class, TagSlices, that splits HTML text at tag boundaries. For example, the TagSclices of foo<tag x="y">bar</tag>quux would be foo, <tag x="y">, bar, <tag>, and quux.

I'd looked at the Java implementation before starting. That code operates on a lowercased string for tag-matching, but returns chunks of the original string. In the implementation I started moving toward, maintaining the two strings was inconvenient, so I talked myself into thinking I could downcase the original string at the start and work only with that. Stupid (people sometimes do use capital letters in web pages), but I was backing away from a frustrating implementation closely modeled after the Java one - and thus un-Ruby-like and hard to get right. I was so focused on tags that I thought what was OK for them was OK for everything.

I'd generated TagSlices using a set of tests that did not reveal the bug. After I was done, I reran the unit tests for the old version. (I hadn't used them for development because they "chunked" the problem in a way that didn't fit the path I was taking.)

Here's one of those tests:

  def test_parsing
    p = Parse.from_text 'leader<Table foo=2>body</table>trailer', ['table']
    assert_equal 'leader', p.leader
->  assert_equal '<Table foo=2>', p.tag
    assert_equal 'body', p.body
    assert_equal 'trailer', p.trailer

It failed on the line marked with an arrow. I thought about that. Was the failure due to a bug? No, I'd decided it was harmless for tags to change case. Did any other assertion fail? No. Was the test completely redundant with other tests? It seemed so. So I should have thrown the test away. But I hesitated. After all, the changed behavior was a side effect, an implementation convenience. It would be just as harmless for tags to keep their case and pass the test. Maybe that wouldn't be as hard as I'd thought when I'd started. I looked at the code and it suddenly flashed on me that lowercasing the whole string wasn't harmless at all.

And, moments later, I realized that Ruby is a scripting language, after all; as such, it lives for regular expressions. Maybe in the Java world, it makes sense to search a lowercased string for "<table". In the Ruby world, it's better to search the original string for /<table/i.

So I created a test that talks specifically about case in tags and non-tag text. I made it pass. The old test passed, too. I could have thrown it away. And yet... what else might it uncover someday? So I kept it.

I shouldn't extrapolate too much from a single example, but it makes me wonder. Seven years ago, when I wrote the paper, I was solidly embedded in the testing culture, a culture of scarcity, one in which:

  • Automated tests were expensive to write because they had to go through an interface not designed for testing.

  • Programming time to fix that was almost entirely unavailable.

  • You were never anywhere close to having as many tests as you thought you needed, so the opportunity cost of fixing an old test was high.

Those assumptions are less true today. Because of that, it makes more sense to change old tests on the off chance you might find bugs or learn something. One of my other testing guidelines is to seek cheap ways to get dumb luck on your side. I'm not smart enough to do without luck. (That's not false modesty: I bet you aren't either.) Fiddling with tests is perhaps now cheap enough to be a luck generator.

(P.S. The bug would have certainly been caught by even the simplest realistic use, so it wouldn't have survived long.)

## Posted at 18:02 in category /testing [permalink] [top]

A rant: filenames

Java is a good thing. Rails is a good thing. But just as Java /perpet[ur]ated/ the horrors of StudlyCaps on those of us who like to read quickly, Rails is /perpet[ur]ating/ underscores in filenames on those of us who like to write quickly.

There are legions of examples of people not acting according to their rational self-interest. Yet another is the prevalence of filenames like webrick_server.rb over webrick-server.rb. Does an underscore take more energy to type than a dash? Yes. Does avoiding a typo require more coordination? Yes. So why this pathology?

  • C used to be the programmer's lingua franca. Since '-' in C means subtraction, variable names conventionally contain underscores. Did the trailblazer Unix programmers not realize that filenames don't have to follow the same rules?

  • Is it because VMS only allowed underscores in filenames, and VMS is just so totally cool?

Whatever the reason, we must consider the result. Thousands upon thousands of people already suffer from Emacs Pinky. To add injury to injury, those fragile pinkies must suffer additional unnecessary damage striking the shift key. How many people have been forced to switch to vi because those underscores pushed them over the edge to pinky RSI? That's a tragedy no caring person can ignore.

You have a lot to answer for, David HH. I was this close to convincing the world to use dashes, and now there's no hope.

## Posted at 18:02 in category /junk [permalink] [top]

About Brian Marick
I consult mainly on Agile software development, with a special focus on how testing fits in.

Contact me here: marick@exampler.com.




Agile Testing Directions
Tests and examples
Technology-facing programmer support
Business-facing team support
Business-facing product critiques
Technology-facing product critiques
Testers on agile projects

Permalink to this list


Working your way out of the automated GUI testing tarpit
  1. Three ways of writing the same test
  2. A test should deduce its setup path
  3. Convert the suite one failure at a time
  4. You should be able to get to any page in one step
  5. Extract fast tests about single pages
  6. Link checking without clicking on links
  7. Workflow tests remain GUI tests
Permalink to this list


Design-Driven Test-Driven Design
Creating a test
Making it (barely) run
Views and presenters appear
Hooking up the real GUI


Popular Articles
A roadmap for testing on an agile project: When consulting on testing in Agile projects, I like to call this plan "what I'm biased toward."

Tacit knowledge: Experts often have no theory of their work. They simply perform skillfully.

Process and personality: Every article on methodology implicitly begins "Let's talk about me."


Related Weblogs

Wayne Allen
James Bach
Laurent Bossavit
William Caputo
Mike Clark
Rachel Davies
Esther Derby
Michael Feathers
Developer Testing
Chad Fowler
Martin Fowler
Alan Francis
Elisabeth Hendrickson
Grig Gheorghiu
Andy Hunt
Ben Hyde
Ron Jeffries
Jonathan Kohl
Dave Liebreich
Jeff Patton
Bret Pettichord
Hiring Johanna Rothman
Managing Johanna Rothman
Kevin Rutherford
Christian Sepulveda
James Shore
Jeff Sutherland
Pragmatic Dave Thomas
Glenn Vanderburg
Greg Vaughn
Eugene Wallingford
Jim Weirich


Where to Find Me

Software Practice Advancement


All of 2006
All of 2005
All of 2004
All of 2003



Agile Alliance Logo