In this tutorial, I’ll use sequences as trees. You can create your own kind of trees if you want, but I won’t cover that.

Here’s what we’ll be working with:

user=> (def original [1 '(a b c) 2])
#'user/original

It’s a tree whose root is a vector with three children: 1, the subtree (a,b,c) and 2. We need to convert it into some sort of data structure that allows free movement. That’s done like this:

user=> (require '[clojure.zip :as zip])
nil
user=> (def root-loc (zip/seq-zip (seq original)))
#'user/root-loc

Notice that I used the alias zip. If you refer or use clojure.zip, you’ll find yourself overwriting useful functions like clojure.core/next.

Notice also that I explicitly wrapped the tree in a seq. If you use seq-zip on an unwrapped vector, you’ll get confusing results.

At this point, I’d describe root-loc as “the location (or loc) of the root of the tomorrow tree.” I say “tomorrow tree” because it’s not a tree itself, but something that can later be converted into a tree. In reality, root-loc names both the loc and the tomorrow tree, bundled up together, but I think it most straightforward to leave the tomorrow tree implicit.

(The actual data structure is called a “zipper”, which is a decent analogy for the actual implementation but didn’t help me understand how to use the library.)

Moving around inside a tomorrow tree

With the loc, we can move around:

user=> original
[1 (a b c) 2]
user=> (zip/node (zip/down root-loc))
1

zip/down moves to the leftmost child of the current loc and returns that child’s loc. zip/node gives you the subtree of the original tree corresponding to its loc argument. It’s one of the main ways you get parts of a tree—regular lists, vectors, and the like—”out of” a tomorrow tree.

The -> macro makes movement in the tomorrow tree much easier to understand:

user=> (-> root-loc zip/down zip/right zip/node)
(a b c)
user=> (-> root-loc zip/down zip/right zip/down zip/right zip/node)
b

Nevertheless, I always wrap anything but the simplest traversals in their own functions.

Here are some other movement functions for you to try: up, left, rightmost, and leftmost. Beware of (arguable) inconsistencies in the handling of edge cases. For example, right and rightmost behave differently when moving “off the end” of a list of siblings:

user=> original
[1 (a b c) 2]
user=> (def last-one (-> root-loc zip/down zip/right zip/right))
#'user/last-one
user=> (zip/node last-one)
2
user=> (-> last-one zip/right)
nil         ; off into nothingness
user=> (-> last-one zip/rightmost zip/node)
2           ; stays put

Parts of a tree

In addition to zip/node, there are other functions for recovering parts of the original tree. All work relative to the current loc.

user=> (def b (-> root-loc zip/down zip/right zip/down zip/right))
#'user/b
user=> (zip/node b)
b
user=> (zip/lefts b)
(a)
user=> (zip/rights b)
(c)

An interesting one shows all subtrees from the root of the tree down to just above the current loc:

user=> (zip/path b)
[  (1 (a b c) 2)
      (a b c)     ]

Changing the tree

A number of functions take a current loc (and associated tomorrow tree) and produce a new loc inside a different tomorrow tree. For example, let’s delete the ‘(a b c) subtree:

user=> (def loc-in-new-tree (zip/remove (zip/up b)))
#'user/loc-in-new-tree

How does the tree represented by this new tomorrow tree differ from the original tree? We can see that with the root function, which applies zip/node to the root of the tomorrow tree:

user=> (zip/root loc-in-new-tree)
(1 2)
user=> original
[1 (a b c) 2]

Where, exactly, is the new location?

user=> (zip/node loc-in-new-tree)
1

The new loc has “backed up” from its previous version. (That’s not an exact enough description, but it’ll do for a few more paragraphs.) The other editing functions return an “unchanged” loc (except in the sense that it’s pointing into a new tomorrow tree with a changed structure): insert-left, insert-right, replace, edit, insert-child, and append-child. Try them out.

In which I reveal I’m slow

At first, I easily forgot that these functions create a new tomorrow tree and don’t really “replace” or “insert” or “edit” parts of the old one. That is:

user=> (zip/root loc-in-new-tree)
(1 2)                ; I see that I've edited the tree.
user=> (zip/root b)
(1 (a b c) 2)        ; Wait - I thought I changed the tree?!

“Well, duh”, you might say, “it’s a functional language with immutable state, so of course it doesn’t change the old tree.” You’re absolutely right, but it was surprisingly easy for old habits to sneak up on me. So, two rules:

• If you ever fail to use the return value of one of these functions, you’re doing it wrong.

• If you ever write code like this:

  (let [stashed-location (zip/whatever ...)]
     ... make "changes" ...
     ... use stashed location ...)
  

  you might be doing it wrong. Make sure that you’re not unthinkingly assuming that later changes to “the” tree are reflected in your stashed-location.

Whole-tree editing

Here’s an example of printing out a whole tree, one node at a time:

(defn print-tree [original]
  (loop [loc (zip/seq-zip (seq original))]
    (if (zip/end? loc)
      (zip/root loc)
      (recur (zip/next
                (do (println (zip/node loc))
                    loc))))))

This is an ordinary recursive loop. It visits each location in the tomorrow tree, stopping when zip/end? is true. zip/next returns a new current loc that is the next one in the tomorrow tree, where “next” means “in preorder depth-first order”. To see that, here’s what one run of the function prints:

user=> (print-tree [1 '(a (i ii iii) c) 2])
(1 (a (i ii iii) c) 2)
1
(a (i ii iii) c)
a
(i ii iii)
i
ii
iii
c
2

To make changes to the tree, add a cond. The default case should hand the current loc to zip/next. The other cases should yield a loc pointing into a changed copy of the tomorrow tree:

  (loop [loc (zip/seq-zip original-tree)]
    (if (zip/end?> loc)
      (zip/root loc)
      (recur (zip/next
          (cond (subtree-to-change? loc)
                (modify-subtree loc)
                …
                :else loc)))))

The tricky bit is making sure that modify-subtree returns a loc just before the next loc of interest (in a depth-first traversal). (It has to be before so that zip/next takes you to the interesting loc.) To get to that loc, you can use any of the movement functions (zip/next, zip/up, zip/rightmost, and so on). There’s also a zip/prev that returns the loc just before the current one.

To keep from confusing myself, I write little helper functions, each named by what it does and what loc it returns. So, for example, I have one function that glories in this name:

(defn wrap-with-expect__at-rightmost-wrapped-location [loc]
  (assert (start-of-arrow-sequence? loc))
  (let [right-hand (-> loc zip/right zip/right)
        edited-loc (zip/edit loc
                   (fn [loc] `(expect ~loc => ~(zip/node right-hand))))]
    (-> edited-loc zip/right zip/right zip/remove zip/remove)))

It takes a form like ... (f 1) => (+ 2 3) :next, with the current loc being (f 1), and turns it into this:

... (expect (f 1) => (+ 2 3)) :next

… with the current loc being at the 3 so that the zip/next returns a loc at :next. This positioning works because I use zip/remove, which returns a loc that’s “backed up” to the previous loc in a depth-first traversal. (That’s the fix to my earlier imprecision about what zip/remove returns. It’s not the previous loc at the same level, which—for the sake of a simpler explanation—I earlier allowed you to assume.)

By building and testing these little functions first, my main cond-loop is easier to get right. You can see some more examples in my test package, Midje. You can look at both the tests and the code.

However. Here we are, aren’t we?

To improve the current situation, I propose the following, roughly inspired by Consumer’s Union.

• There should be an Agile Alliance program that applies to all courses, not just those that provide certifications.

• This program will, with the Agile Alliance’s money, send spies to sign up for courses. They will anonymously write up detailed evaluations that will be posted on the Agile Alliance website.

• It will also interview a random sample of course attendees. The ones contacted just after the course will be asked for their evaluation. The ones contacted a year after the course will be asked how valuable, in retrospect, the course was. These results will also be posted on the site.

• These interviews and ratings will, over time, allow the writing of summary articles like the ones Consumer Reports Magazine runs. Expect articles about “What to Expect from a Certified ScrumMaster” or “How to select a TDD course”.

• Those running the program will pay careful attention to maintaining the trust of their readers.

It’s possible I’d participate in this. I have no desire to run it. I don’t care that much.

metavar? and form-branch? are only useful within this function. I would have preferred to write it like this:

I didn’t because I wanted those two functions tested before I relied on them. So I need a format for testing inner functions. I’ve sketched out a format and done (by hand) the transformation required to make it work.

I’ll start explaining it with this simple example:

Before dealing with the daunting complexity of multiplication, I want to test summer with something like this:

(I’m using the Midje sweet notation here. Read it as “It’s a fact that (within (outer 1 2)), (summer) produces 3.”)

To make this work, I’d redefine the defn macro for the duration of a test and have it produce a little extra metadata:

I’m stashing away a function derived from summer. It creates the same lexical environment that outer does, and returns a function to call. Here’s the macro that would expand a within, together with a sample expansion:

The expanded form is a little opaque, so here are the three steps:

1. Retrieve the environment setting function from outer’s metadata.
2. Call it with the outer arguments.
3. Within that environment, call summer.

What if a later inner function uses a previously-defined one? Like this:

One option would be to drag the earlier parts of the let into the metadata:

In that case, the test would look like this:

This test is a little not-thrilling because it’s clearer what multiplier does if you can see how it relies on summer. That could look like this:

However, dragging the previous let clauses into the metadata has to be done anyway (in case non-function values are bound), so I’d be inclined not to gild this lily.

Creating the metadata wouldn’t be too hard in the common case where the let is the outermost form in the function’s definition. I doubt I could be persuaded to care about other cases.

• It’s better to focus on the immediate problem than on convincing people of a universal truth.

• Right action does not follow from thinking correctly. Thinking correctly follows from the habit of acting correctly.

The sentence is an aphorism: terse, stylish, true, and deep. I think its proponents value the aphoristic quality as much as they do the underlying sentiment. When I watch them use it, I believe they’d be happiest if they could generate an epiphany (”a sudden, intuitive perception of or insight into the reality or essential meaning of something”). I’ll try and mimic some of the aphoristic terseness at first, even though I am explicitly not going for an epiphany.

Here goes:

1. The setting: an unproductive conversation needs to be redirected. It’s likely that the conversation is happening because some important people are unhappy. But instead of talking about how to make them happy, we’re talking about whether they are right to be unhappy. We’ve come down to—explicitly or implicitly—competing definitions of quality.

2. There is a brief lull in the conversation. Our hero speaks.

“Different people care differently.” (beat)

Because the sentence doesn’t obviously have anything to do with quality, it has a desirable “huh? where did that come from?” effect that keeps people from jumping into our hero’s pause-for-emphasis.

3. The hero speaks the following sentence with an “opening up / giving” hand movement, the sort where you bring your hand up and out, rotating it from palm down to palm up. This counters the harshness and abruptness of the first sentence: Diff-rent pee-pul judge diff-rent-ly.

“So different people will judge our product differently.” (beat)

A little rhetorical flourish in the parallelism of the two sentences. The “so” at the beginning switches the meter to more iambic, gives it a little forward momentum, opens it up into a more conversational tone. “Judge” is a strong, active word (as is “care”), and focuses attention on what people are doing that the group must react to.

4. Someone tries to jump into the pause. Our hero gains time for one more utterance with a “just one more thing” head-tilt-back, audible intake of breath, and soft palms-out rotation of the hands, forefinger up.

5. The next bit takes long enough to give our hero a chance to make contact, via hand and head movements, with the people who are bored sick of the argument and have been sitting silently while the true believers battle it out. They’re being invited to break their silence to lend support.

“I predict that if we figure out which people we care about, and figure out what they care about, and how we can know their judgments before we get surprised like we were, we can live long and happy lives without ever agreeing what quality is. Let’s give it a try, OK?”

Long—because the problem isn’t simple. A program for action. Inviting a group of arguers to become “we” again.

The whole structure of the three parts is moving from authoritarian (”a fact you must recognize is…”) and abrupt to light (”long and happy lives”) and consensus-seeking (”we”, “OK”, ending on a rising question-tone).

Quality is value to some person.
Quality is quality to some person.
Value is value to some person.

We’re not shy about saying that testers, programmers, and everyone else in the team needs to become fluent in the business domain. But we’re blasé when programming practices and programming knowledge are the exclusive province of the programmers. That hampers communication and hurts teams. By simply having people stand up, move around, and speak messages to each other, this workshop will show anyone who attends how advanced object-oriented programming and test-driven design works. Although tailored to non-programmers, programmers may find it useful too.

I used to use the constructor approach, but I increasingly find myself using the Hulk! Smash! approach. It looks like this…

The object-to-be-tested names its collaborators and gives them the values they’ll use in production:

I do this because I think of the collaborators as a static, declaration-like property of the class. (If I were ambitious, I’d convert collaborators_start_as into a class-level declaration like this:

I haven’t been that ambitious yet, as it turns out.)

Each test declares which collaborators it overrides:

Note that (in Ruby), I can even mock out classes, so I don’t have to go through contortions to pass in instances that an outside caller really would have no particular interest in:

With several of the mocking packages available to me, I wouldn’t even have to go through the indirection of putting the class ThingSource into an instance variable. I could do this:

I’m not bold enough to do that yet.

Me: You careless oaf! You burned up my lawn!

Him: I admit it. I take full responsibility. Sorry.

Me: You’re going to pay for this damage, you know. All of it.

Him: I will do that.

Me: I bet that big old maple that overhangs the house is going to have to come out. That’s going to cost you a bundle.

Him: Oh now, it doesn’t look that bad.

Me: I had a tree hit by lightning once. Didn’t look much worse than that, but carpenter ants got into it and the tree had to come out. The guys who took it out told me I should have done it right away, that the tree was worse off than it looked. I’m not waiting this time.

Him: Uh… Don’t know all that much about trees.

Me: Neither do I. Look. Let’s simplify this, keep at least some of it out of court. Let’s get someone with experience at judging these kinds of damages and have him decide, shrub by shrub, tree by tree, what you owe.

Him: Like the independent mediator written into lots of contracts.

Me: Right.

Some time passes as we dicker over who we both trust.

Me: Now, I’m pretty sure this is going to cost you over $2000…

Him: Aw, that seems high…

Me: That maple is four stories tall!

Him: Hey, I still think it’ll be alright. But OK—so long as we get that mediator guy, the one I trust.

Me: And I want you to put the money in escrow.

Him: Aw, c’mon. You know I’m good for it.

Me: Yeah, well, my trust in you is not super-high right now.

Him: Well…

Me: Look. Back out of the deal if you want. You’re going to pay one way or the other. You can do it quickly or drag it out. Your choice.

Him: OK, OK: it’s a deal.

All that remains is to add the locations of bordering cells to a given sequence of locations. A small wrinkle is that a border location may be next to more than one of the input locations, so duplicates need to be prevented. I could write the test this way:

I don’t like that. When the test fails, it’ll likely be hard to discover how the expected and actual values differ. And I bet that a failure would more likely be due to a typo in the expected value than to an actual bug. The test is just awkward.

I like this version better:

It is a clearer statement of the relationship between two concepts: a location’s neighborhood and the border of a set of locations. More pragmatically, it’s less typing. (When I first started coding in this style, it surprised me how much test setup clutter went away. I had much less need for “factories” or “fixtures” or “object mothers“.)

Given either test, the code is straightforward:

Onward

No matter how satisfying the individually-tested pieces, the whole has to work, which is why everything ends by running at least one end-to-end test. A test like the one we started with:

Because I’m making up my test notation as I go, I’ve been running all the tests manually in the REPL. Now I can run the whole file:

If you look carefully, you’ll see that the test would fail because the locations are in the wrong order. But that’s a quick fix:

Done. Ship it!

Development order - Test-down or REPL-up?

It’s often said that the Lisps are bottom-up languages, that you test out expressions in the REPL, discover good functions, and compose them into programs. A lot of people do work that way. A lot of people who use TDD to write object-oriented code also work that way: when implementing a new feature, they start at low-level objects, add whatever new code the top-level feature seems to demand of them, then use those augmented objects in the testing of next-higher-level objects.

For I guess about a year now, I’ve been experimenting with being strictly top-down in some projects. I find that leads to less churn. Too often, when I go bottom-up, I end up discovering that those low-level changes are not in fact what the feature needs, so I have to revisit and redo what I did.

I get less disrupted by rework when I go from the top down (or from user interface in). It’s not that I don’t blunder—we saw one of those in the previous installment—but those blunders seem easier to recover from.

That’s not to say that I don’t use the REPL. We saw that a little bit in this program, when I was writing neighbors. It’s perfectly sensible to do even more in the REPL. I think of the REPL as a handy tool for what XP calls a spike solution and the Pragmatic Programmers call tracer bullets. When I’m uncertain what to do next, the REPL is a tool to let me try out possibilities. So I might be stuck in a certain function and go to the REPL to see how it feels to build up parts of what might lie under it. After I’m more confident, I can continue on with the original function, test-driven, reusing REPL snippets when they seem useful.

I don’t claim everyone should work that way. I do claim it’s a valid style that you’d be wise to try.

Notation

I’m something of an obsessive about test notation, and I’ve been endlessly fiddling with a Clojure mock notation. I implemented its first version as a facade on top of clojure.contrib.mock. As I experimented, though, I found that keeping my facade up to date with my notational variants slowed me down too much, so I put the code aside until the notation settled down.

I’m pretty happy with what you’ve seen here. Are you? If so, I may start on another mock package. I’ve got the most important parts: a name and sketch of a logo. (”Midje” and someone flying safely between the sun [of abstraction without examples] and the sea [of overwhelming detail].)

Tests and code together

You can see the completed program here. It mixes up tests and code. I’ve tried that on-and-off over the years and always reverted to separate test files. This time, it’s seemed to work better. I probably want an Emacs keystroke that lets me hide all tests, though. I’d also want alternate definitions of the test macros so that I can compile them out of the production system

What next?

I’ll write a web app in this style, using Compojure.