Sat, 12 Aug 2006
I have finished the review draft of Scripting for Testers. I am going on holiday.
Thu, 15 Jun 2006
Earlier, I wrote about sentence style tests for rendered pages. Based partly on conversations about that entry with Steve Freeman and partly on bashing against reality, I've changed the style of those tests.
Since they are about the part of the app that the user sees and since I'd like them to be readable by the product director, I found myself asking where they would come from in a business-facing-test-first world and how the product director would therefore think about them. I imagined that, sometime early on, someone makes a sketch, paper prototype, or a wireframe diagram. So I came to think that this test ought to be a textual, automatically-checkable wireframe diagram. Like this:
One interesting thing is that I put the setup for the test after the checking code. That's because the page layout seems more important.
How well does that test describe this page? (The sidebar is described in tests of its own.)
I'll let you be the judge.
Fri, 09 Jun 2006
Steve Freeman and Nat Pryce will have a paper titled "Evolving an Embedded Domain-Specific Language in Java" at OOPSLA. It's about the evolution of jMock from the first version to the current one, which is something of a domain-specific language for testing. It's a good paper.
I've been doing some work recently on an old renderer-presenter project, and I was inspired by the paper to rip out my old tests of a rendered page and replace them with tests in their style. Here's the result. It first has flexmock sentence descriptions of how the renderer uses the presenter. Then come other sentence descriptions of the important parts of the page structure.
I rather like that, today at least. It's much more understandable than my previous tests. After only a few months, I had to go digging to figure them out, but I doubt I'll have to do that for these. Moreover, I think these tests would be more-or-less directly transcribable from a wireframe diagram or sketch of a page on a whiteboard. They're also, with a little practice, reviewable by the product director.
(I'm still very much up in the air about how much automated testing how close to the GUI we should do, but this has nudged my balance toward more automated tests.)
I also remain fond of workflow tests in this style:
These workflow tests can be derived from interaction design work as easily as Fit tests are. They're less readable than Fit tests, but not impossibly code-like. These workflow tests are end-to-end. They go through HTTP (using my own browser object, rather than Watir or Selenium), into the renderer/presenter layer, down into the business logic, and through Lafcadio into MySQL.
Thu, 25 May 2006
Any time you write code that sits on top of a third party library, your code will hide some of its behavior, reveal some, and transform some. What are the testing and cost implications?
By "cost implications," I mean this: suppose subsystem USER is 1000 lines of code that makes heavy use of library LIB, and NEW is 1000 lines that doesn't (except for the language's class library, VM, and the operating system). I think we all wish that USER and NEW would cost the same (even though USER presumably delivers much more). However, even if we presume LIB is bug free, we have to test the interactions. How much? Enough so that an equal-cost USER would be 1100 lines of unentangled code? 1500? 2000? It is conceivable that the cost to test interactions might exceed the benefit of using LIB, especially since it's unlikely we're making use of all of its features.
More likely, though, we'll under-test. That's especially true
because I've never met anyone with a good handle on what we're
testing for. Tell me about a piece of fresh code, and I can
rattle off things to worry about: boundary conditions,
special values like
The result of uncertain testing is a broken promise. Given test-driven design, bug reports should fall into two categories:
The TDD promise is that there should be few type 2 real bugs. But if we don't know how to test the integration of LIB and USER, there will be many of what I call fizzbin bugs: ones where the programmer fixing them discovers that, oh!, when you use LIB on Tuesday, you have to use it slightly differently.
Since fizzbin bugs look the same to the product director or user, greater reuse can lead to a product that feels shaky. It seems to me I've seen this effect in projects that make heavy use of complex frameworks that the programmers don't know well. Everyone's testing as best they can, but end-of-iteration use reveals all kinds of annoyances.
I (at least) need a better way to think about this problems. More later, if I think of anything worth writing.
Wed, 22 Feb 2006
My "working your way out of the GUI testing tarpit" series really ought to be put into a single paper with the rough transitions smoothed over. Until that happens, if ever, what I've got will have to serve. Here's the table of contents.
Mon, 20 Feb 2006
The American Dialect Society voted "truthiness" the 2005 word of the year. It "refers to the quality of preferring concepts or facts one wishes to be true, rather than concepts or facts known to be true." For me, 2006 is turning into the year of replacing end-to-end tests with unit tests. One risk to face is that unit tests can play into truthiness. This picture illustrates the problem:
Everything seems fine here. The tests all pass. What the picture doesn't show is that the Widget tests require the strings from the Wadget to be in ascending alphabetical order. The fake Wadget dutifully does that. The Wadget tests don't express that requirement, so the real Wadget isn't coded to satisfy it. The strings come back in any old order.
Truthiness would be wishing that unit tests add up to a working system. But the truth is that those two units would add up to a system like this:
We know that those sorts of mismatches happen in real life. So we should fear unit tests.
More tests are a proper response to fear. Hence the desire to wrap the entire chain above in an end-to-end test that 'sees what the user sees'. However, such tests tend to be slow, fragile, etc. So I want to replace them with smaller tests or other methods that are fast, robust, etc., thus reducing the need for end-to-end tests to a bare minimum.
Two such methods are:
I expect there are a host of other tricks to learn (but I'm not at this moment aware of places where they're written down). What's seems to me key is to take the strategy of "something could go wrong somewhere, so here's a kind of test with a chance of stumbling over some wrongness" and replace it with (1) a host of tactics of the form "this could go wrong in places like that, so here's a specific kind of test or coding practice highly likely to prevent such bugs" and (2) a much more limited set of general tests (including especially manual exploratory testing).
P.S. I don't like the word "truthiness." It seems statements should have truthiness, not people. A question for you hepcats out there who are down with the happening slang: which is more copacetic, "that's a truthy statement" or "that's a truthish statement"?
Sat, 18 Feb 2006
A client and I were talking over how Model-View-Presenter would work for web applications. The sequence diagram to the right (click on it to get a bigger version in a new window) describes a possible interpretation. Since the part that corresponds to a View just converts values into HTML text, I'm going to call it the Renderer instead. The Renderer can be either a template language (Velocity, Plone's ZPT, Sails's Viento) or—my bias—an XML builder like Ruby's Builder.
I did a little Model-Renderer-Presenter spike this week and feel pretty happy with it. I'm wondering who else uses something like what I explain below and what implications it's had for testing. Mail me if you have pointers.
(Prior work: Mike Mason just wrote about MVP on ASP.NET. I understand from Adam Williams that Rails does something similar, albeit using mixins. So far handling the Rails book hasn't caused me to learn it. I may actually have to work through it.)
Here's the communication pattern from the sequence diagram:
What good is this? Classical Model-View-Presenter is about making the View a thin holder of whatever controls the windowing system provides. It does little besides route messages from the window system to the Presenter and vice versa. That lets you mock out the View so that Presenter tests don't have to interact with the real controls, which are usually a pain.
There's no call for that in a web app. The Renderer doesn't interact with a windowing framework; it just builds HTML, which is easy to work with. However, the separation does give us four objects (Action, Model, Renderer, and Presenter) that:
The second picture gives a hint of the kinds of checks and tests that make sense here. (Click for the larger version. Safari users note that sometimes the JPG renders as garbage for me. A Shift-Reload has always fixed it.)
More later, unless I find that someone else has already described this in detail.
Thu, 16 Feb 2006
Over at the Agile-Testing list, there's another outbreak of a popular question: are testers needed on Agile projects? To weary oldtimers, that debate is something like the flu: perennial, sneakily different each time it appears so that you can't resolve it once and be done with it, something you just have to live with.
After skimming the latest set of messages on the topic, I returned to editing a magazine article and then I had a thought that might just possibly add something.
Editors are supposed to represent readers (and others), just as testers are supposed to represent users (and others). To an even greater extent than testers, editors do exactly what the the represented people do: they read the article. And yet, you can't take J. Random Reader and expect her to be a good editor. Why not?
It seems to me that as readers we're trained to make allowances for writers. We're so good at tolerating weak reasoning, shaky construction, and muddled language that a given reader will notice only a fraction of the problems in a manuscript. A good editor will notice most of them. How?
Some of it is what "do we need testers?" discussions obsessively circle: perspective. Editors didn't write the manuscript (usually...), so their view of what it says is not as clouded by knowledge of what it should have said. Editors also do not have their ego involved in the product.
But that perspective is shared by any old reader. What makes editors special is, first, technique. I put those techniques into two rough categories:
But there's something else that editors and testers have that programmers don't have: leisure. When I'm acting as a pure reader, I intend to get through it and out the other side quickly. As an editor, there's no guilt if I linger. There's guilt if I don't. One problem that Agile projects have is a lack of slack time, down time, bench time. There's velocity to maintain—improve—and the end of the iteration looms. Agile projects are learning projects, true, but the learning is in the context of producing small chunks of business value. There's no leisure for the focus to drift from that. (I'm using "leisure" rather than "permission" because so much of the pressure is self-generated.)
My hunch is that perspective is less important than technique and leisure for producing good products. If the testing and programming roles are to move closer together (which I would like to see), the real wizards of testing technique need to collaborate with programmers to adapt the techniques to a programmer's life. (I tried to do that a few years ago. It was a disaster, cost me two friendships. Someone else's turn.) And projects need some way to introduce leisure. (Gold cards?)
Wed, 15 Feb 2006Jeffrey Fredrick about a conference:
Sat, 28 Jan 2006
Where do we stand?
I want to end this series by closing one important gap. We know that links go somewhere, but we don't know that they go to the right place, the place where the user can continue her task.
We could test that each link destination is as expected. But if following links is all about doing tasks, good link tests follow links along a path that demonstrates how a user would do her work. They are workflow tests or use-case tests. They are, in fact, the kind of design tests that Jeff Patton and I thought would be a communication tool between user experience designers and programmers. (At this point, you should wonder about hammers and nails.)
Here's a workflow test that shows a doctor entering a new case into the system.
I've written that with unusual messages, formatted oddly. Why?
Unlike this test, I think my final declarative tests really are unit tests. According to my definition, unit tests are ones written in the language of the implementation rather than the language of the business. My declarative tests are about what appears on a page and when, not about cases and cows and audits. They're unit tests, so I don't mind that they look geeky.
Workflow tests, however, are quintessential business-facing tests:
they're all about asserting that the app allows a doctor to perform
a key business task. So I'm trying to write them such that,
punctuational peculiarities aside, they're sentences someone calling
the support desk might speak.
I do that not so much because I expect a user to look at them
as because I want my perspective while writing them to be
outward-focused. That way, I'll stumble across more design
Sending all messages to an object representing a
Similarly, I'm using layout to emphasize what's most important. That's what the user can do and what, having done that, she can now do next. The actual checks that the action has landed her on the right page are less important—parenthetical—so I place them to the side. (Note also the nod to behavior-driven design.)
The methods that move around (like
As you can see, I don't check much about the page. I leave that to the declarative page tests.
That's it. I believe I have a strategy for transforming a tarpit of UI tests into (1) a small number of workflow tests that still go through the UI and (2) a larger number of unit tests of everything else.
Thanks for reading this far (supposing anyone has).
The tests I was transforming didn't do any checking of pure business logic, but in real life they probably would. They could be rewritten in the same way, though I'd prefer to have at least some such tests go below the presentation layer.
There are no browser compatibility tests. If the compatibility testing strategy is to run all the UI tests against different browsers, the transformation I advocate might well weaken it.
There are no tests of the Back button. Should they be part of workflow tests? Specialized? I don't know enough about how a well-behaved program deals with Back to speculate just now. (Hat tip to Seaside here (PDF).)
Can you do all this?
The transformation into unit tests depends on there being one place that receives HTTP requests (Webrick). Since Webrick is initialized in a single place, it's easy to find all the places that need to be changed to add a test-support feature. The same was true on the outgoing side, since there was a single renderer to make XHTML. So this isn't the holy grail—a test improvement strategy that can work with any old product code. Legacy desktop applications that have GUI code scattered everywhere are still going to be a mess.
See the code for complete details.