Exploration Through Example

I’ve been beating my head against the technical debt metaphor a bit these days. Here’s one thing that oozed out.

Preface: I’m assuming there are two kinds of debt. One is frivolous debt; the other is debt-as-investment. When debt is an investment, you have a reasonable expectation that a bulk sum of money now will allow you to start something that will, at some future time, produce excess cash greater than the cumulative interest on the debt. For example, you borrow money to buy a car because it lets you choose from more—and more kinds of—jobs. More opportunity means greater lifetime earnings than if you rode the bus while saving for a car.

Business pressure to add technical debt to a code base usually is couched in terms of debt-as-investment: “If we miss that market window, we’re out of business, so it won’t matter how scalable the code is.” (Whether the expectations are reasonable, and whether the same parties pay for the debt as get the increased returns, I leave for another day.)

Frivolous debt is different. For example, John McCain and his wife recently had USD10,000-15,000 in credit card debt at 25.99% interest. There’s no reason for them to pay that stiff an interest rate, but what the hell: she made more than USD6,000,000 in 2006, so why sweat interest charges? More commonly, though, frivolous debt does matter, and it comes from ignorance. In code, for example, technical debt can accrue because people simply don’t know guidelines like “Tell, don’t ask” or “Don’t repeat yourself“.

In what follows, “debt” means “investment debt.”

The analogy to a growing GDP

We imagine that debt-free code is without history. Suppose that, during development, classes Foo and Bar split into Foo, Bar, and Baz, and some of Foo’s methods were merged into Bar. If you were coming on the project, I should be able to explain why Foo, Bar, and Baz make sense without referring to their history.

History only comes up where there’s technical debt. It makes you say things like “there’s duplication between Foo and Bar because we didn’t realize fact X until new story Y and we haven’t time to finish merging their behavior.”

I have a sense that telling that story isn’t just a way of making excuses: it’s actually important for being able to work with the code in the presence of debt.

But, because we have this history-free ideal, we don’t pay attention to that. Maybe embedding history in the code (somehow) is a way of increasing the debt load the team is capable of supporting in perpetuity.

So, where I had been thinking about how to reduce debt, now I’m thinking about how to make it matter less. Like there are two ways to solve the problem of an insupportable national debt: you can pay it down or you can grow the economy.

I struggled using mocks in RubyCocoa. Here’s what I learned, using FlexMock.

I’m writing some tests to understand and explain the RubyCocoa interface to key-value observing. Key-value observing is yet another way for one object to observe another. (I’ll call these two objects the @watcher and the @observed.)

If I were describing a key-value observing test over the phone, I’d describe it as having four steps:

1. Set up the relationship between the two objects.

2. Change an attribute of @observed.

3. During that change, expect some @watcher method to be called. One of the arguments will be an NSDictionary that contains the values before and after the change.

4. Also expect that the value really truly has been changed.

Here’s how I’d describe it in code:

  def test_setting_in_ruby_style
    newval = 5
    during {
      @observed.value = newval
    }.behold! {
      this_change(:old => @observed.value, :new => newval)
    }
    assert_equal(newval, @observed.value)
  end

I have an idiosyncratic way of using mocks in Ruby. Notice the order in my English description above. I say what the test is doing before I say what happens during and after that action. But mock expectations have to be set up before the action. In Java (etc.) that requires steps 2 and 3 to be reversed. In Ruby, I can use blocks to put the steps in the more normal order. (You can see the definitions of during and behold! in the complete code.)

Since each of the tests sets up almost the same expectations, I hived that work off into this_change:

  def this_change(expected)
    @watcher.should_receive(:observeValueForKeyPath_ofObject_change_context).
             once.
             with(’value‘,
                  @observed,
                  on { | actuals |
                       actuals[:old] == expected[:old] &&
                       actuals[:new] == expected[:new]
                  },
                  nil)
  end

That’s a fairly typical use of FlexMock. I specify three arguments exactly, and I use code to check the remaining one. (There are components in the actuals NSDictionary that I don’t care about.)

Now I have to start to do something special. Here’s setup:

  def setup
    super
    @observed = Observed.alloc.init
    @watcher = flexmock(Watcher.alloc.init)
    @observed.addObserver_forKeyPath_options_context(
                @watcher, ‘value‘,
                OSX::NSKeyValueObservingOptionNew | OSX::NSKeyValueObservingOptionOld,
                nil)
  end

Normally, an argument to flexmock just provides a name for the mock, useful in debugging. However, key-value observing only works when the @watcher object inherits from NSObject. So I pass in an already-initialized object for FlexMock to “edit” and add mockish behavior to. It’s of class Watcher, which inherits from NSObject. But why not just use an NSObject?

I could, except for a helpful feature of key-value observing. Before each call into the @watcher, key-value observing checks if it responds to the method observeValueForKeyPath_ofObject_change_context. If not, it raises an exception.

If I just passed in an NSObject and gave FlexMock an expectation that it should_receive an observeValueForKeyPath_ofObject_change_context message, FlexMock would just attach that expectation to a list of expectations. It would not create any method of that name; rather, method_missing traps such calls to the method and then checks off the appropriate expectation.

Therefore, I have to create a Watcher class for no reason other than to contain the method:

class Watcher <  OSX::NSObject
  def observeValueForKeyPath_ofObject_change_context(
             keyPath, object, change, context)
  end
end

The method doesn’t have to actually do anything; it’s never called. Instead, FlexMock redefines it to trampoline a call to it over to message_missing.

You can find the rest of the code here: mock-example.rb.

Hope this helps.

A continual debate on the agile-testing mailing list is to what degree testers think differently than programmers and are therefore able to find bugs that programmers won’t. Without ever really resolving that question, the conversation usually moves onto whether the mental differences are innate or learnable.

I myself have no fixed opinion on the matter. That’s probably because, while vast numbers of testers are better than I am, I can imagine myself being like them and thinking like them. The same is true for programmers. In contrast, I simply can’t imagine being the sort of person who really cares who wins the World Cup or whether gay people I don’t know get married. (I’m not saying, “I couldn’t lower myself to their level” or anything stupid like that—I’m saying I can’t imagine what it would be like. It feels like trying to imagining what it is like to be a bat.)

However, I’ve long thought that security testers are a different breed, though I can’t articulate any way that they’re different in kind rather than degree. It’s just that the really good ones are transcendentally awesome at seeing how two disparate facts about a system can be combined and exploited. (A favorite example)

Bruce Schneier has an essay on security testers that I found interesting, though it doesn’t resolve any of my questions. Perhaps that’s because he said something I’ve been thinking for a while:

The designers are so busy making these systems work that they don’t stop to notice how they might fail or be made to fail, and then how those failures might be exploited. Teaching designers a security mindset will go a long way toward making future technological systems more secure.

The first sentence seems to make the second false. When I look back at the bugs I, acting as a programmer, fail to prevent and then fail to catch, an awful lot of the time their root cause wasn’t my knowledge. It’s that I have a compulsive personality and also habitually overcommit. As a result, there’s a lot of pressure to get done. The problem isn’t that I can’t flip into an adequate tester mindset, it’s that I don’t step back and take the time.

So, I suspect the interminable and seemingly irresolvable agile-testing debate should be shelved until we solve a more pressing problem: few teams have the discipline to adopt a sustainable pace, so few teams are even in a position to know if programmers could do as well as dedicated testers.

In response to a potential client, I wrote something very like the following. The interesting thing is that I’m placing more emphasis on manual exploratory testing. It’s not so much that I suddenly realize its importance as that automated business-facing tests continue to be hard to implement and adopt. More on that anon.

A short sketch of a reasonable growth path would go like this:

1. Get the programmers sold on test-driven design. How difficult that is depends mainly on how much legacy code you have (where legacy code is, as Michael Feathers says, code without unit tests). Legacy code is hard to test, so programmers don’t see the benefits of testing as quickly, so it requires that much more discipline to get over what’s always a higher hump than with greenfield code. (Michael Feathers’ Working Effectively with Legacy Code is the gold standard book, though there’s an important strategy—”strangler applications“—that’s not covered in depth. Also, I’m the track chair for a new Legacy Code track at Agile2008, I just asked Feathers to give the keynote, and he says he has “a number of surprising proposals about how to make things better”.)

   I’ve come to feel that the most important thing to get across to programmers is what it’s like to work with code built on a solid base of tests. If they understand that early on, they’ll have a clear idea of what to shoot for, which helps with the pain of legacy code. I wrote a workbook to that end.

2. At the same time, move testers away from scripted manual tests (if that’s what they’re doing) and toward a more exploratory style of manual testing. The people who are strongest on exploratory testing in Agile are Jonathan Kohl, Elisabeth Hendrickson, and Michael Bolton.

3. As programmers do more unit testing, they will become accustomed to changing their design and adding code in support of their own testing. It becomes more natural for them to do the same for the testers, allowing them to do “automation-assisted exploratory testing”. (Kohl writes about this.) I like to see some of the testers learn a scripting language to help with that. Ruby is my favorite, for a variety of reasons. I wrote a book to help testers learn it.

4. Over this period, the testers and programmers should shed most animosity or wariness they have toward each other. They’re working together and doing things to help each other. It helps a lot if they sit together.

5. Once the programmers are sold on test-driven design, they will start wishing that the product owners would supplement what they say about what they want with clear, concrete, executable examples of what they want. That is: tests, written in the language of the business. That isn’t as easy to do as we thought it would be five years ago, but it can be done more or less well. Often, the testers will find a new role as helpers to the product owners. For example, they should get involved early enough to ask questions that lead to tests that prevent bugs (which is better than discovering the bugs after you’ve paid some programmers to implement them).

6. Throughout this, some kinds of testing (like performance testing) don’t change all that much. For performance testing, I trust Scott Barber.

As a side note: I’m quite fond of the new The Art of Agile Development by Shore & Warden: enough to publicly declare that I’ll bring a copy to every team I work with. Lots of good from-the-trenches experience summarized there.

Nat Pryce wrote a nice little summary of what he’s learned about evolving an API.

An API is a user interface for programmers. That means you have to develop it like a user-interface: iteratively, adapting to how the users actually use and misuse the interface in real life […]

But… programmers focus on solutions, not goals, so their feature requests will be worded in terms of how they think they would change the API to support whatever it is they want to do. You have to engage in a lot of arduous back-and-forth to find out what they are trying to achieve […]

From Matt Heusser:

Personally, I believe that the “technical debt” metaphor is compelling. It explains the tradeoffs involved in taking shortcuts in terms that management can understand, and can take the vague and undefined and turn it into something more concrete.

At the same time, it is not a perfect metaphor, it has weaknesses, and we have a lot more to explore. For example:
- How do you measure technical debt?
- How do you communicate it?
- How do you reverse it?
- How do you avoid it?
- Is reversing or avoiding technical debt always the right thing to do?

To get to some more satisfying answers, Steve Poling and I will be organizing a peer workshop on Technical Debt on August 14/15. The event is hosted by the Calvin College CS Department and free to participants. Seating is limited to fifteen (at most, twenty) people and is by application.

The Call for Participation is available on-line:
http://www.xndev.com/downloads/WOTD_2008_CFP.rtf

The announcement is available on my blog.

If you’ve got some experiences to share, some stories to relate, or you would just like to explore the issue some more, I would encourage you to apply.

There’s one of those tagging memes going around. This one is: “grab the nearest book, open to page 123, go down to the 5th sentence, and type up the 3 following sentences.”

My first two books had pictures on p. 123.

The next three (Impro: Improvisation and the Theatre, AppleScript: the Definitive Guide, and Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life) didn’t have anything that was amusing, enlightening, or even comprehensible out of context. So I kept going, which is cheating I suppose. The last, How Designers Think, had this:

The designer’s job is never really done and it is probably always possible to do better. In this sense, designing is quite unlike puzzling. The solver of puzzles such as crosswords or mathematical problems can often recognize a correct answer and knows when the task is complete, but not so the designer.

That’s a hit. It made me realize a flaw in my thinking. You see, it reminded me of one of my early, semi-controversial papers, “Working Effectively With Developers” (referred to by one testing consultant as “the ‘how to suck up to programmers’ paper”). In its second section, “Explaining Your Job”, I explicitly liken programmers to problem solvers:

A legendary programmer would be one who was presented a large and messy problem, where simply understanding the problem required the mastery of a great deal of detail, boiled the problem down to its essential core, eliminated ambiguity, devised some simple operations that would allow the complexity to be isolated and tamed, demonstrated that all the detail could be handled by appropriate combinations of those operations, and produced the working system in a week.

Then I point out that this provides a way for testers to demonstrate value. I show a sample problem, then write:

Now, I’d expect any programmer to quickly solve this puzzle - they’re problem solvers, after all. But the key point is that someone had to create the puzzle before someone else could solve it. And problem creation is a different skill than problem solving.

Therefore, the tester’s role can be likened to the maker of a crossword or a mathematical problem: someone who presents a good, fully fleshed-out problem for the programmer to master and solve:

So what a tester does is help the programmer […] by presenting specific details (in the form of test cases) that otherwise would not come to her attention. Unfortunately, you often present this detail too late (after the code is written), so it reveals problems in the abstractions or their use. But that’s an unfortunate side-effect of putting testers on projects too late, and of the unfortunate notion that testing is all about running tests, rather than about designing them. If the programmer had had the detail earlier, the problems wouldn’t have happened.

Despite this weak 1998 gesture in the rough direction of TDD, I still have a rather waterfall conception of things: tester presents a problem, programmer solves it, we all go home.

But what that’s missing is my 2007 intellectual conception of a project as aiming to be less wrong than yesterday, to get progressively closer to a satisfactory answer that is discovered or refined along the way. In short—going back to the original quote—a conception of the project as a matter of design that’s at every level of detail and involves everyone. That whole-project design is something much trickier than mere puzzle-solving.

I used the word “intellectual” in the previous paragraph because I realize that I’m still rather emotionally attached to the idea of presenting a problem, solving it, and moving on. For example, I think of a test case as a matter of pushing us in a particular direction, only indirectly as a way of uncovering more questions. When I think about how testing+programming works, or about how product director + team conversations work, the learning is something of a side effect. I’m strong on doing the thing, weak on the mechanics of learning (a separate thing from the desire to learn).

That’s not entirely bad—I’m glad of my strong aversion to spending much time talking and re-talking about what we’ll build if we ever get around to building anything, of my preference for doing something and then taking stock once we have more concrete experience—but to the extent that it’s a habit rather than a conscious preference, it’s limiting. I’ll have to watch out for it.

Eugene Wallingford wrote a depressing post two months ago. (Note: to find the article, you’ll have to scroll down.)

I occasionally write about how students these days don’t want to program. Not only don’t they want to do it for a living, they don’t even want to learn how. I have seen this manifested in a virtual disappearance of non-majors from our intro courses[…]

Eugene point to a post from Mark Guzdial:

Mike is running a trial section of 140 students. The day that he introduced the programming aspect, a dozen students applied to drop the course. Note that: nearly 10% of the enrollment. They had to talk to Mike to drop, because of the trial nature of the section, and the common reason was that they didn’t want to do any programming in Physics.[…] Students dislike programming so much that they are dropping a required course[…]

This trend is important to me for two reasons.

1. I’ve written a book, Everyday Scripting with Ruby, to help non-programmers learn to script. That book’s outsold expectations, but I have an anecdotal impression it’s not because testers (its primary audience) are swarming to pick it up. Rather, it’s because other audiences are. (Good thing we changed the name away from Scripting for Testers.) And I’m writing another book on using Ruby to script your Mac.

2. I’ve completely failed to entice my children into learning programming. A Squeak summer camp: nix. Programming a Lego Robotics robot (in the Lego visual language or Ruby): nix. They find it boring. There are many more immediately gratifying things to do with the computer, the “wow!” factor of even game-oriented programming doesn’t compete with what you can find easily on the net, and Sims 2 and Civ IV are better at satisfying the need to compulsively twiddle with something for long periods of time.

I’ve delayed writing about this for so long because I find myself without anything useful to say. But at least you now have the links.

On the agile-testing list, I answered this question:

So for the first time in many, many years I’m not in a test management position, and I’m writing tests, automating them, etc. We’re using a tool called soapUI to automate our web services testing–it’s a handy tool, supports Groovy scripting which allows me to go directly to the DB to validate the results of a given method, etc. One feature of soapUI is centralized test scripts; I can create helper scripts to do a bunch of stuff–basically, I write the Groovy code I need to validate something and then I often find I’m moving it into a helper function, refactoring, etc.. My question is, how do you know the right balance between just mashing up the automation (ie, writing a script as an embeded test script) vs. creating the helper function and calling it?

Bret Pettichord suggested I blog my answer, so here it is:

I use these ideas and recommend them as a not-bad place to start:

1. At the end of every half-day or full-day of on-task work (depending on my mood), I’ll spend a half an hour cleaning up something ugly I’ve encountered.

2. I’ll periodically ask myself “Is it getting easier to write tests?” If it’s not, I know I should slow down and spend more effort writing helper code. I know from experience what it feels like when I’ve got good test support infrastructure—how easy writing tests can be—so I know what to shoot for.

3. I have this ideal that there should be no words in a test that are not directly about the purpose of the test. So: unless what the test tests is the process of logging in, there should be no code like:
   
create_user "sam", "password"
login "sam", "password"
...


   Instead, the test should say something shorter:
   
using_logged_in_user "sam"
...


   That tends to push you toward more and smarter helper functions.

   Unsurprisingly, I often fall short of that ideal. But if I revisit a test and have any trouble at all figuring out what it’s actually testing, I use that as a prod to make the extra effort.

What was something of a breakthrough for me was realizing that I don’t have to get it right at this precise moment. Especially at the beginning, when you don’t have much test infrastructure, stopping every twelve seconds to write that helper function you ought to have completely throws you out of the flow of what you’re trying to accomplish. I’ve gotten used to writing the wrong code, then fixing it up later: maybe at the end of the task, maybe not until I stumble across the ugliness again.

I’ve decided to use the Intellij IDEA Ruby plugin as my Ruby IDE when working on larger projects and use Aquamacs Emacs for quick work. I think you should all buy IDEA too, so that they are encouraged to make it as good a Ruby editor as it is a Java editor.

Some people use Emacs for large projects. I’m handicapped in doing that because I have an unusually bad short-term memory. So I will often want two or three windows of code plus one shell/test window open. Or at least, I will want wicked fast ways of switching to recently used files. I’ve never found an Emacs workflow that makes having more than two visible buffers comfortable. And Emacs has a more cumbersome way of flipping to recent files than IDEA, plus it doesn’t have the convenient navigational sidebar of modern tools.

I should note that my Emacs skills froze more than a decade ago, so I may be unaware of today’s goodness. In fact, looking in my .emacs file, I see this:

(setq shell-mode-hook '(lambda () (my-shell-setup) (setq mode-line-format (list "%b--" 'default-directory "---%M---%3p---%[(%m: %s)%]%-")) (local-set-key "^j" 'shell-send-input) ; stupid Bridge box ))

The “stupid Bridge box” is a terminal concentrator from around 1987.

I tried TextMate for a couple of months. I approve of the idea, but in practice I had two problems. There’s something about the logic of the key combinations that doesn’t work for me. The way similar commands are organized into related key groups (variants of key modifiers) will not stick in my head, so I was constantly trying control-shift-meta-cokebottle, getting the wrong result, trying shift-control-cokebottle, etc. (And one of the wrong key combinations I often picked—I forget which—made a not-obvious change to the file that would surprise me next time I ran the tests.) Also, the seams kept showing: globalish commands that don’t work because I’m in the sidebar not a source file, the way my tabs kept filling up with files that I’d never intentionally visited, and the like. And the current lack of window splitting is a deal-killer.

I gave NetBeans a shorter trial, and I can’t remember why I didn’t like it.

IDEA has the advantage that I use it for Java, so many of my reflexes carry over. And I always found it had some of that mystical goodness that originally made me prefer Ruby to Python: when I guessed at how something would work, most often I was right.

Still: besides the fact that an upgrade cost me USD 149 (5 euros), it has some disadvantages.

• The Ruby plugin is beta. I haven’t found too many bugs, but features are incomplete. It’s missing many of TextMate’s nice little Rails features. (But at least you don’t have to install a hack to rerun the last test from anywhere you are.)

• Because it’s a Ruby IDE wedged into a Java IDE, it may be confusing for people unfamiliar with Java (and, perhaps, Java in IDEA). For example, the General Preferences has these entries: Project JDK (”Ruby SDK 1.8.6″), Project Language Level (irrelevant for Ruby, but on my machine I could choose 1.3, 1.4, or 5.0), and Project Compiler Output (irrelevant).
  (Project setup doesn’t push Java in your face quite that much.)

• The worst is that it sometimes decides to spend a lot of memory and CPU doing something. When my pokey little MacBook (1.83Ghz, 2G memory) is trying to drive both its screen and a 24 inch external monitor, Time Machine has decided to kick in, and I’ve got various other things running (like Safari, which also likes to suck down CPU while doing nothing visible), IDEA can get slow. I’ll end up typing faster than it can echo characters.

Despite those things, I find IDEA somehow hits a sweet spot where I’m actually working with the code, not the editor (the old ready-to-hand vs. present-to-hand distinction). With other tools, I felt more friction.