Exploration Through Example

TL;DR: Attitudes common in the Clojure community are making that community increasingly unappealing to people like me. They may also be leading to a language less appealing to people like me than a Lisp should be.

I think people like me are an important constituency.

The dodgy attitudes come from the Clojure core, especially Rich Hickey himself, I’m sad to say. I wish the Clojure core would be more supportive of programmers not like them, and of those with different goals. Otherwise, I’m afraid Clojure won’t achieve a popularity that justifies the kind of commitment I’ve been making to it. What should I do?

I’ve caused a bit of kerfuffle on Twitter about Clojure and testing. I’ve been somewhat aggressive because I’m trying to suss out how much effort to put into Clojure over, say, the next year.

[Note to the reader: scatter “I think” and “I could be wrong” statements freely throughout the following.]

Here’s the deal.

What’s at issue is a style of working (roughly XPish). After some years of work by many, it’s entirely possible in the Ruby world to work in that style. After doing the whole Agile spokesmodel thing for a decade, that’s all I want: a place where there are enough people with my style that we can (1) find each other to work together, and (2) work together without having to re-justify our style to every damn person we meet. In Ruby, XP style is mainstream enough that we can let the product of our work speak for us. (Hence, my recently co-founded contract software company.)

I think it was at the 2nd RubyConf that Nathaniel Talbott introduced Test::Unit, a Ruby version of jUnit. I can’t be positive, but I’m pretty sure Matz (the author of Ruby) didn’t care all that much about testing or, especially, TDD. Wasn’t his style. Perhaps still isn’t. However: he did not, to my recollection, express any reaction other than his usual “People are doing things with my language! I am happy!” And while Matz gives keynotes about design and such, they are pretty obviously about how he feels about design and about how he approaches it. They are invitations, not instructions.

One of the strengths of Ruby in the early days was this air of invitation. (I don’t really follow that community closely any more.)

Rich Hickey, for all his undoubted brilliance as a language implementor and his fruitful introspection into his own design process, presents a less welcoming demeanor than Matz’s. From what I’ve seen, he is more prone, in his public speaking and writing, to instruction than to invitation.

Which is OK with me, all things being equal. But there’s a problem. Brilliant solo language designers set a tone for the community that gathers around the language, especially in the early days. We’ve seen that with Python and Ruby. (Rails too, which acts as a contrast to the “Ruby Classic” tone.) It’s happening again in Clojure, where I’d say the picture of the ideal developer is someone who, with grim determination, wrestles a hard problem to the ground and beats it into submission with a multi-issue bound copy of The Journal of Functional Programming. I’m all for people who do that! So I don’t have to! But I personally prefer something more akin to the style of Ruby’s _why, who’d sneak up on a problem and tickle it until it was laughing too hard to be a problem any more.

I didn’t always prefer the Ruby style. Just after I started my first post-graduate job, Tony Hoare’s Turing Award lecture came out, containing the famous quote:

I conclude that there are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult.

I positively thrilled to that. That was the man I wanted to be.

As time went on, though, I ran into difficulties. What if you can’t solve the problem with that first method, whether because you’re not smart enough, or the problem domain is inherently messy, or the problem keeps changing on you? What then? Do you just give up? Or do you try to find a third method?

Those difficulties, plus some timely encounters with some strange thinkers, notably Feyerabend and his Against Method, put me on my current career path. Which causes me to say this:

For someone doing core infrastructural design, such as the huge task of tricking the JVM into running a Clojure program speedily, an emphasis on purity and simplicity—one bordering on asceticism—seems to me essential. God speed Rich Hickey for that!

But when a community adopts purity, simplicity, and asceticism as core tenets—ones approaching matters of morality—you can end up with a community that makes things unpleasant for not just the impure and hedonistic, but also for just plain normal people. And they will go elsewhere.

Much as I love Lisp and admire Clojure, if it can’t grab the enthusiasm of, say, the median pretty-good Ruby or Java programmer who has the TDD habit, it’s not economically sensible for a cofounder of a new company to put as much effort into it as I have been putting.

If Clojure turns into a tool for hardcore people to solve hardcore problems, without making concessions for people with softer cores—if it remains a niche language, in other words—I should probably spend more time with Ruby. And I do think, right now, that Clojure is more likely to be a niche language than I thought last year.

So in the last few weeks, I’ve been sending out little probes, mainly to see if anyone with some star power can provisionally buy the story above. If so, I can hope to see an intentional broadening in a direction designed to shackle me and those like me more tightly into the Clojure community—and so help it succeed.

So far, meh.

If I’m sensible, a continuing “meh” response will prompt me to reduce my effort around Clojure—to resist the temptation to make another tutorial, to cut back on new Midje features (but not on fixes and the 1.3 port, I hasten to reassure its users!), and so on. If Clojure then wins big, I can jump back on the bandwagon. I will have lost some early mover advantage, but that doesn’t look to be huge anyway. (Not like it was with Agile, or would have been with Ruby if I hadn’t drifted away before Rails came out.)

I haven’t decided what to do. Reaction to this post might help. Probably the best thing would be for me to be convinced I’m nuts. Hey, it’s happened before!

A postscript on the validity of what I claim

A postscript on worries about the effect of performance on expressiveness

I’ve been asked for things to read after my “Functional Programming for Object-Oriented Programmers” course. Here are some, concentrating on the ones that are more fun to read and less in love with difficulty for difficulty’s sake.

If you were more interested in the Clojure/Lisp part of the course:

• The Little Schemer and other books by the same authors. These teach some deep ideas even though they start out very simply. You’ll either find the way they’re written charming or annoying.

• Land of Lisp is a strange book on Lisp.

• If you were totally fascinated by multimethods, you could try The Art of the Metaobject Protocol.

If you were more interested in the functional programming part of the course, the suggestions are harder. So much of the writing on functional programming is overly math-fetishistic.

• Learn You a Haskell for Great Good is another quirky book (lots of odd drawings that have little to do with the text), but it covers many topics using the purest functional language. There’s both a paper version and a free online version. The online version has better drawings (color!).

• I haven’t read Purely Functional Data Structures, though I think I have it somewhere. @SamirTalwar tweeted to me that “I’m enjoying Purely Functional Data Structures. It’s quite mathsy, but with a practical bent.” The book is derived from the author’s PhD thesis.

The approach I took in the course was inspired by the famous Structure and Interpretation of Computer Programs (free online version). Many people find the book more difficult than most of the ones above. Part of the reason is that the problems are targeted at MIT engineering students, so you get a lot of problems that involve resistors and such.

In the Agile2011 fringe, I’ll be doing a session with that title. Here’s my blurb:

One of the characteristics of Agile is that it favors the reactive over the proactive. For example, you grow the program in response to new feature requests rather than needing to know all the features in advance. Programmers regularly talk of reacting to “code smells” or paying attention to “what the code’s telling us”.

Being appropriately reactive is a skill. You’re not born knowing how to attend to subtle details, or how to react in a graceful, helpful way.

Skills can be taught. All over the world, people are learning the complementary skills of leading and following in tango, an incredibly improvisational and reactive dance. I believe the learning techniques tango instructors use can be adapted to learning other kinds of reaction—like the reactions of pair programmers to each other and to the code.

In the first part of this session, I’ll teach you some simple tango moves. We’ll concentrate both on learning the moves and also on how we’re learning the moves.

In the second part of the session, we’ll break out computers and write programs or tests (or collaborate on some other task of your choice). Will you be able to react more appropriately to your pair? Will you give off better signals? Will consciously practicing the skill of pairing help you be better at it? We’ll see!

Important note: In my experience, most people are more comfortable learning to dance with someone of the opposite sex. I’ve also noticed that there are a lot more male programmers than female programmers. I worry the title will attract only programmers, leading to a big imbalance. So I encourage everyone to encourage women at the conference to attend at least the first half of the session. They’ll get something out of it (beyond an introduction to tango).

Here is a draft blurb for a course I’d like to teach, possibly in Valladolid, Spain, in September. I’d like to see if enough people are interested in it, either there or somewhere else. (There’s a link to a survey after the description.)

Short version

• Target audience: Programmers who know Java, C#, C++, or another object-oriented language. Those who know languages like Ruby, Python, or JavaScript will already know some—but not all!—of the material.

• The goal: To make you a better object-oriented programmer by giving you hands-on experience with the ideas behind functional languages. The course will use Clojure and JavaScript for its examples. You’ll learn enough of Clojure to work through exercises, but this is not a “Learn Clojure” course.

• After the course: I expect you to return to your job programming in Java or Ruby or whatever, not to switch to Clojure. To help put the ideas into practice, you’ll get a copy of Dean Wampler’s Functional Programming for Java Developers.

Syllabus

• Just enough Clojure
• Multimethods as a generalization of methods attached to objects
• Protocols as a different approach to interfaces
• Functions as objects, objects as functions
• Mapping and folding in contrast to iteration
• Functions that create functions
• Laziness and infinite sequences
• Time as a sequence of events
• Generic datatypes with many functions versus many small classes
• Immutable data structures and programs as truth-statements about functions

Longer version

Many, many of the legendary programmers know many programming languages. What they know from one language helps them write better code in another one. But it’s not really the language that matters: adding knowledge of C# to your knowledge of Java doesn’t make you much better. The languages are too similar—they encourage you to look at problems in pretty much the same way. You need to know languages that conceptualize problems and solutions in substantially different ways.

Once upon a time, object-oriented programming was a radical departure from what most programmers knew. So learning it was both hard and mind-expanding. Nowadays, the OO style is the dominant one, so ambitious people need to proactively seek out different styles.

The functional programming style is nicely different from the OO style, but there are many interesting points of comparison between them. This course aims to teach you key elements of the functional style, helping you take them back to your OO language (probably Java or C#).

There’s a bit more, though: although the functional style has been around for many years, it’s recently become trendy, partly because language implementations keep improving, and partly because functional languages are better suited for solving the multicore problem^* than are other languages. Some trends with a lot of excitement behind them wither, but others (like object-oriented programming) succeed immensely. If the functional style becomes commonplace, this course will position you to be on the leading edge of that wave.

There are many functional languages. There are arguments for learning the purest of them (Haskell, probably). But it’s also worthwhile to learn a slightly-less-pure language if there are more jobs available for it or more opportunities to fold it into your existing projects. According that standard, Clojure and Scala—both hosted on the JVM—stand out. This course will be taught mainly in Clojure. Where appropriate, I’ll also illustrate the ideas in JavaScript, a language whose good parts were strongly influenced by the functional style.

While this particular course will concentrate on what you can do in Clojure, it’d be unfair to make you do all the work of translating these ideas into your everyday language. To help you with that, every participant will get a copy of Dean Wampler’s Functional Programming for Java Developers, which shows how functional ideas work in Java.

Your instructor

Brian Marick was first exposed to the functional style in 1983, when the accident of knowing a little bit of Lisp tossed him into the job of technical lead on a project to port Common Lisp to a now-defunct computer architecture. That led him to a reading spree about all things Lisp, the language from which the functional style arguably originated. He’s been a language geek ever since, despite making most of his living as a software process consultant. He’s the author of the popular Midje testing library for Clojure and has written two books (Everyday Scripting with Ruby and Programming Cocoa with Ruby).

For examples of Marick’s teaching style, read his explanations of the Zipper and Enlive libraries or watch his videos on Top-down TDD in Clojure and Monads.

^*The multicore problem is that chip manufacturers have hit the wall making single CPUs faster. The days of steadily increasing clock rates are over. Cache sizes are no longer the bottleneck. The big gains from clever hardware tricks have been gotten. So we programmers are getting more cores instead of faster cores, and we have to figure out how to use multiple cores in a single program. That’s a tough problem.

Enlive is a templating library for Clojure. Instead of the usual substitute-into-delimited-text approach, it works by editing node trees selected by CSS selectors. I’ve written a tutorial for it. Comments welcome.

As novice user of Hiccup, a Clojure HTML-layout format, I wanted to write a form that gathered information about N animals. Each animal had a name, a nickname, etc. I wanted to use Ring’s ring.middleware.nested-params so that I could iterate over new animals.

I had some trouble getting it to work, so perhaps this note will help someone else. Note that this applies to hiccup-0.3.5.

My first attempt was something like this:

This was derived from the description of with-group here. However, this didn’t work. Only the last text-field appears in the HTML page. [Note: I was using Noir to render pages.]

Having written a builder-style renderer like Hiccup before, I thought. “Ah. I must make a vector of the rendered functions”:

That lead to a scary exception:

java.lang.RuntimeException: java.lang.RuntimeException: java.lang.IllegalArgumentException: [:label {:for “animal1-true-name”} “Name: “] is not a valid tag name.

The solution I found was to wrap the sub-form inside another element, in this case a :p:

It’s ironic that I’d do something like that as a matter of course, except that I was exploring the API. Which led me astray.

UPDATE: Chris Granger points out “Hiccup treats any vector as a tag, but any other seq will work. Vectors have semantic meaning, the others don’t.” Instead of wrapping the separate steps in a vector, I should have used a list:

I should have realized that in the beginning (though one could argue that the two uses for a vector are distinguishable so why not make a vector work as well as a list?).

Chris also writes “In my experience the best solution is to wrap the block in (html … ). Then you never worry.” That is:

Enrique Comba Riepenhausen (@ecomba on twitter) has been demonstrating an interesting style of making a living. He travels around the world, working with companies as a programmer. When he’s done, he tells them to pay him what he was worth. That’s pretty much it.

I like strange ideas and relationships more personal than transactional, so I’d like to do this too. Here’s what I’m thinking:

• I’d like to program for you as a temporary, on-site member of the team. I prefer that to coaching these days, though the latter isn’t out of the question. I’m not bad at Clojure and Ruby¹, and my Java skills are adequate. Because I’d expect to spend a lot of time pair programming, the language isn’t nearly as much of an issue as it would be for solo work.

• I’d prefer to be in one place for one to three weeks, though longer or shorter is also possible.

• I’d like you to commit to paying for travel. (If I’m visiting more than one place on a trip, I’d divide up the responsibility.) I prefer you pay for lodging, but I’m also willing to couch surf (especially on shorter trips).

• I don’t care all that much about working on cool technology or products. I do care about the atmosphere of the team, because their mood strongly affects my own. I’m looking for teams that are optimistic, enthusiastic, context-driving and cheerfully naive in the way I talk about in this video. That rules out a large number of Enterprise teams, who so often seem incredibly burdened by the institution. (Though that does mean that the rare optimistic etc. enterprise team would be particularly tempting.)

• I have a chance to run my Top-down TDD in Clojure workshop in Oslo, Norway, in September, so I’m especially interested in European companies so that I can spend a month over there.

If you’re interested, please mail me.

¹If you’d like to see a sample of my Clojure work, here’s Midje. I can’t show you the Ruby code I’m happiest with, but you can look at Critter4Us.

I’ve been speaking about the desirability of what I call the stance of reaction in software development. Now it’s time to think about how to learn it. Since one of my inspirations for it is tango, I’ll first talk about how tango is learned. (This post may be tedious for people who don’t care about tango or teaching physical actions.)

Dawn and I took a tango lesson last night. It’s a just-beyond-beginner course concentrating on the milonga and vals, two alternate tango rhythms, but it was also teaching new figures to many of the students.

Here’s the first figure, which is one you can repeat an arbitrary number of times as you proceed sideways down the floor:

Joe and Carlotta (the instructors) began by demonstrating the figure, then explained just a few things to attend to. (Joe will usually say something about which part of the leader’s body signals the next movement to the follower. Carlotta usually talks about where and how the follower’s feet should land so as to be ready for the next step.)

What happened next was typical: everyone practiced the movement separately with Joe and Carlotta making a few comments. We moved quickly to direct practice in a mixer: you meet a partner, go sideways down the floor, separate, meet up with a new partner, and continue. Most of us aren’t confident enough to correct our partners, but it’s usually pretty easy to tell how you’ve gotten it wrong and correct it yourself. It’s also easy to tell by feel when you’ve gotten it right.

Joe and Carlotta observed, as usual. After two or three songs finished, they pointed to some things most of us could have done better. I don’t think we practiced this figure again; sometimes we do.

That typical practice session finished, they then introduced a new figure that can be combined with the first one. Instead of going backwards after the second step, the leader can take the couple to the right for one or more steps, then backward:

This figure makes it harder for the follower. After two steps, she has to be prepared either to step forward or to her left. The leader signals this mostly by just moving in the direction he wants the couple to go, which I think the follower detects mostly via her arm on his shoulder.

We went quickly into another mixer. In this one, leaders were encouraged to switch between the two figures at unexpected moments. So the followers were learning to detect the lead, and the leaders were learning to make it clear enough to be detectable.

The next figure starts at the same position but deviates in a different direction. Instead of stepping forward once, bringing the feet together, and then changing direction, the leader repeats the first two steps:

After that, he can take the couple backwards (as in the original figure), to the right (as in the second) or, if there’s room, continue diagonally forward with the first two steps.

At this point in the lesson, I think we danced several songs in the normal “line of dance” (in a rough counterclockwise circle around the room, rather than down a straight line), switching partners between songs or at most once a song. (Tango songs are usually in the 2-4 minute range.) Using the line of dance approaches non-practice dancing more closely: there’s more time to adjust to a particular partner, and also the leader must avoid more collisions than in a line. (Learning to react to events while preserving your musicality is an important part of tango.)

All that lasted about a half an hour. Then there was open practice. Dawn and I concentrated somewhat on the figures we’d just practiced, but I threw in other variations and also figures that don’t start with the same step. These figures were easy enough that we could just have fun.

Dawn and I also practice in our basement and on our driveway (which I’m sure the neighbors think is terribly cute). We concentrate on what we’ve learned recently. (All these variations are hard to remember!) We’ll often practice the fine points of steps and body orientation in slow motion, then at normal speed. We also just plain dance, though the two spaces aren’t very good for that.

There are weekly to biweekly come-one-come-all practice sessions that Dawn and I usually attend. We do the same sort of practicing as above, but we have the advantage of experienced people we can ask for tips.

After almost a year, we should probably be dancing at real dances, but they’re usually pretty crowded and we’re not so experienced in the steps that work in that kind of a crowd. Also, neither of us is a particularly social animal.

The Constant Reader is surely aware I have a childish resentment of the dominance, in software coaching, of theories that change-is-managing-a-crisis-imposed-from-the-outside. It’s not that I deny the existence of such situations; it’s that I hate the ubiquity of the theory. That’s especially so because humans are so reflexive: our theories about our behavior affect our behavior, reinforcing our theories. If the only model you have for change is crisis, then every change is seen as a crisis, and there is less room for inner-motivated change.

Over the years, I’ve convinced… oh, I’d say, about… no one of my point. So perhaps it’s time to bring out the big guns and fight metaphor with metaphor. The dominant metaphor of change is drawn from family therapy and analogizes the group in change to the family in crisis. My new metaphor of change analogizes change to the adolescent primate leaving its group.

Oops, I’ve just oversimplified:

It is no longer accurate to state that one or the other sex disperses at sexual maturity; dispersal in primates is much more complicated than that. — Katherine M. Jack, Lynn A. Isbell

Nevertheless, it’s the case that our closest animal relatives do follow patterns where the (mostly) young leave one group and, despite considerable risk, join another. The typical explanation is something deterministic like incest avoidance. The summary link above claims that explanation, too, is an oversimplification. The behavior of social primates is complex and, to my layman’s eyes, often defies simple linear “Because of X, the animal does Y” explanations.

Although it’s probably dangerous to anthropomorphize even close relatives like chimpanzees, I’ve read articles (none of which I can dig up) that tell stories of ape dispersals that invite us to attribute emotions other than fear to this huge life change. The young ape is curious about another group, finds them tantalizing. It sneaks around watching them, hesitantly getting closer and closer, scampering back to the safety of the home group if threatened, but gets increasingly brave over time. At some point, it makes a commitment and joins the group, despite a typically unfriendly early reception and often ending up as the lowest status animal.

Change here is desirable and motivated internally. Rather than a move forced out of desperation, it’s one important enough to “save up for”:

Instead, female emigration was predicted by dietary quality, suggesting that females wait to have sufficient energy reserves to buffer the high costs of dispersal. — A summary of a longer article.

I think that makes a good metaphor for the kind of change that we can seek, not avoid, in software development: change that is risky, yes, but change that we are impelled to try, because of our nature.

Are you saving up for the next change that’ll tantalize you, that you just have to try? What? You’re not expecting one? You haven’t been tantalized by the possibility of big change since you-can’t-remember-when? You are violating your primate heritage. Perhaps you should align with the Porifera.

Here’s an app that’s used to reserve teaching animals at the University of Illinois vet school:

A user selects procedures to perform by clicking in the leftmost window; animals to perform them on in the rightmost window. Clicking moves the procedure or animal to the corresponding center window and can also narrow down the possible choices. For example, some procedures can only be performed on horses. If you pick one of them, all the cows disappear from the rightmost window.

The first user interface didn’t look like this. It used three windows instead of four, and drag-n-drop instead of clicking. It was heavily tested using a mock-centric style like that of Growing Object-Oriented Software. The switch to a new interface broke all those tests. Bummer!

However, I discovered an interesting thing while dealing with the broken tests. None of them applied directly to the new user interface. However, each of them had some original purpose; each captured an issue that had been important. The exact issue was no longer important, but usually one very like it applied to the new UI. So the broken tests became something of a checklist and spur to the imagination. Rather than just fixing them, or throwing them away, I used them to ask questions: “What user intention or user safeguard does this test represent? How would that intention be exercised in the new UI? [And what test should be written?] Is there a similar danger in the new UI? [And what test would show that the user is protected from it?]”

It was something of a revelation. Allofasudden, maintaining the tests had actual value. Yes, mock-heavy tests do require more maintenance. No, that doesn’t bother me.

Being prone to grandiose statements, that’s led me to proclaim the dawning of the Third Age of Programmer Testing.

The First Age

In the first age, the vast majority of programmers thought testing was boring, pointless, beneath them, and a big waste of time. A much smaller number of programmers thought it was worth doing, although virtually no one was happy with it. I think we all realized, at least unconsciously, that something was wrong with our practice.

The Second Age

Around the beginning of this century, the practice of programmer testing took a big jump forward. Things like one-test-then-just-enough-code are, in part, mental tricks: avoiding one big chunk of testing (at either the beginning or, more usually, at the end) fools our lizard brains into thinking that we’re programming the whole time, so it knows not to be bored. We know much better how to write code that’s easy to test, avoiding tedious and time-consuming test-writing (especially setup). And so on.

Many, many programmers are still stuck in the First Age, but I don’t particularly care. I don’t have to work with them. There are plenty of Second Age folk around.

However, an interesting thing about Second Agers: the vast majority of them think test maintenance is boring, pointless, beneath them, and a big waste of time. A tiny minority (maybe just me?) embraces it.

The Third Age

Just as test-writing took a big leap forward in the past 12 or so years, I predict test-maintaining will do the same over this decade. We’ll develop mental tricks, coding and testing strategies, and tools that will make maintenance as much a thinking-about-design activity as test writing is today.