Exploration Through Example

In some way, my original post makes it seem as if normal programmers are like the drug dealer D’Angelo Barksdale eating in a fancy restaurant in “The Wire“—everyone stared, the waiter rubbed D’Angelo’s ignorance of fine-restaurant customs in his face, and D’Angelo’s introspective attitude doesn’t hide a desperate desire to leave. Well, yes (though nothing in a programmer’s life is as bad as anything in a “The Wire” character’s life), but there’s more to it.

I think Clojure already suffers from being a language for hardcore programmers to solve hardcore problems. For example, that hardcore attitude makes things like nice stack traces seem unimportant. More: fishing through stack traces can almost become a perverse right of passage. Do you have what it takes to program in Clojure, punk?

The history of programming, as I’ve lived it, has been a march where expressiveness led the way and performance followed. I came of age when the big debate was whether us new-fangled C programmers and our limited compilers could ever replace the skilled assembly-language programmer. C won the debate. When Java came out, the debate was whether a language with byte codes and garbage collection could be fast enough to be a justifiable choice over C/C++. Java won the debate. Ruby is still being held up as too slow in comparison with Java, but even the pokey MRI runtime is fast enough for a whole lot of apps. Perhaps the emerging consensus is that Twitter-esque success will require migrating from Ruby to JVM infrastructure—and that’s exactly the kind of problem you want to have.

Against that background, it’s notable that Clojure 1.3 is taking a step backward in expressiveness in the service of performance. (By this, I mostly mean that performance—calling through vars, arithmetic operations—is the default and people who are worried about rebinding and bignums have to act specially. So my super-cool introduction-to-Clojure “Let’s take the sum of the first 3000 elements of an infinite sequence of factorials” example will become less impressive because arithmetic overflow is now in the programmer’s face.) That’s worrisome. There’s not much new in 1.3 to tempt the hitherto-reluctant Java or Ruby programmer. And unless I care about performance or am a utility library maintainer, I don’t see very much in 1.3 to tempt me away from 1.2.

Now, 1.3 is not a huge hit to expressiveness, and Clojure already has pretty much all the expressive language constructs that have stood the test of time (except, I suppose, Haskell-style pattern matching). So what else is left for someone of Rich Hickey’s talents to work on but performance? Documentation?

That’s a good point, but I worry about how performance will drive the constructs available to Clojure users. For example, multimethods feel a bit orphaned because of the drive to take advantage of how good the JVM is at type dispatch. There’s a whole parallel structure of genericity that’s getting the attention. And conj is a historical example: it’s hard to explain to someone how these two things both make sense:

user=> (conj [1 2 3] 4)
[1 2 3 4]
user=> (conj (map identity [1 2 3]) 4)
(4 1 2 3)

As with arithmetic overflow, it strikes those not performance-minded (those not hardcore) as odd to have lots of generic operations on sequences except for the case of putting something on the end. “So I have to keep track of what’s a vector? In a language that creates lazy sequences whenever it gets an excuse to?”

Catering to performance increases the cognitive load a language puts on its users. Again, the hardcore programmer will take it in stride. But Lisp and functional languages are different enough that I think Clojure needs more focused attention on the not-hardcore programmers.

Now, the quick retort is that Clojure isn’t my language. It’s a gift from Rich Hickie to the world. Who am I to demand more from him? That’s a good retort. But my counter-retort is that I demand nothing. I only suggest, based on my experience with the adoption of ideas and languages. It’s hard making something. More information (from users like me) is probably better than less.

The problem with talking about the tone of the Clojure community is that almost no statement can ever be proven. Rich Hickey and I went around and around about this on Twitter. So after citing this statement of his:

If people think that spending fifty percent of their time writing tests maximizes their results—okay for them. I’m sure that’s not true for me—I’d rather spend that time thinking about my problem.

… I could say that I read a snarky “[those losers]” implicitly attached to “okay for them.” I could say that “fifty percent of your time writing tests” and “rather spend that time thinking about my problem” are complete mischaracterizations of TDD as it’s actually done. And I can say that those mischaracterizations and that snark will be aped by programmers who want to be Rich Hickey.

I could say all those things, but the statement as written isn’t false. It can even be given a generous reading. But—if I’m right—programmers like me won’t stick around to hear defensible statements defended. They’ll just look for friendlier places where they aren’t expected to deal with decade-old arguments as if they were new and insightful.

Please note that I’m not saying Rich Hickie’s design approach is wrong. I suspect for the kinds of problems he solves, it may even be true that his approach is better across all designers than TDD is.

TL;DR: Attitudes common in the Clojure community are making that community increasingly unappealing to people like me. They may also be leading to a language less appealing to people like me than a Lisp should be.

I think people like me are an important constituency.

The dodgy attitudes come from the Clojure core, especially Rich Hickey himself, I’m sad to say. I wish the Clojure core would be more supportive of programmers not like them, and of those with different goals. Otherwise, I’m afraid Clojure won’t achieve a popularity that justifies the kind of commitment I’ve been making to it. What should I do?

I’ve caused a bit of kerfuffle on Twitter about Clojure and testing. I’ve been somewhat aggressive because I’m trying to suss out how much effort to put into Clojure over, say, the next year.

[Note to the reader: scatter “I think” and “I could be wrong” statements freely throughout the following.]

Here’s the deal.

What’s at issue is a style of working (roughly XPish). After some years of work by many, it’s entirely possible in the Ruby world to work in that style. After doing the whole Agile spokesmodel thing for a decade, that’s all I want: a place where there are enough people with my style that we can (1) find each other to work together, and (2) work together without having to re-justify our style to every damn person we meet. In Ruby, XP style is mainstream enough that we can let the product of our work speak for us. (Hence, my recently co-founded contract software company.)

I think it was at the 2nd RubyConf that Nathaniel Talbott introduced Test::Unit, a Ruby version of jUnit. I can’t be positive, but I’m pretty sure Matz (the author of Ruby) didn’t care all that much about testing or, especially, TDD. Wasn’t his style. Perhaps still isn’t. However: he did not, to my recollection, express any reaction other than his usual “People are doing things with my language! I am happy!” And while Matz gives keynotes about design and such, they are pretty obviously about how he feels about design and about how he approaches it. They are invitations, not instructions.

One of the strengths of Ruby in the early days was this air of invitation. (I don’t really follow that community closely any more.)

Rich Hickey, for all his undoubted brilliance as a language implementor and his fruitful introspection into his own design process, presents a less welcoming demeanor than Matz’s. From what I’ve seen, he is more prone, in his public speaking and writing, to instruction than to invitation.

Which is OK with me, all things being equal. But there’s a problem. Brilliant solo language designers set a tone for the community that gathers around the language, especially in the early days. We’ve seen that with Python and Ruby. (Rails too, which acts as a contrast to the “Ruby Classic” tone.) It’s happening again in Clojure, where I’d say the picture of the ideal developer is someone who, with grim determination, wrestles a hard problem to the ground and beats it into submission with a multi-issue bound copy of The Journal of Functional Programming. I’m all for people who do that! So I don’t have to! But I personally prefer something more akin to the style of Ruby’s _why, who’d sneak up on a problem and tickle it until it was laughing too hard to be a problem any more.

I didn’t always prefer the Ruby style. Just after I started my first post-graduate job, Tony Hoare’s Turing Award lecture came out, containing the famous quote:

I conclude that there are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult.

I positively thrilled to that. That was the man I wanted to be.

As time went on, though, I ran into difficulties. What if you can’t solve the problem with that first method, whether because you’re not smart enough, or the problem domain is inherently messy, or the problem keeps changing on you? What then? Do you just give up? Or do you try to find a third method?

Those difficulties, plus some timely encounters with some strange thinkers, notably Feyerabend and his Against Method, put me on my current career path. Which causes me to say this:

For someone doing core infrastructural design, such as the huge task of tricking the JVM into running a Clojure program speedily, an emphasis on purity and simplicity—one bordering on asceticism—seems to me essential. God speed Rich Hickey for that!

But when a community adopts purity, simplicity, and asceticism as core tenets—ones approaching matters of morality—you can end up with a community that makes things unpleasant for not just the impure and hedonistic, but also for just plain normal people. And they will go elsewhere.

Much as I love Lisp and admire Clojure, if it can’t grab the enthusiasm of, say, the median pretty-good Ruby or Java programmer who has the TDD habit, it’s not economically sensible for a cofounder of a new company to put as much effort into it as I have been putting.

If Clojure turns into a tool for hardcore people to solve hardcore problems, without making concessions for people with softer cores—if it remains a niche language, in other words—I should probably spend more time with Ruby. And I do think, right now, that Clojure is more likely to be a niche language than I thought last year.

So in the last few weeks, I’ve been sending out little probes, mainly to see if anyone with some star power can provisionally buy the story above. If so, I can hope to see an intentional broadening in a direction designed to shackle me and those like me more tightly into the Clojure community—and so help it succeed.

So far, meh.

If I’m sensible, a continuing “meh” response will prompt me to reduce my effort around Clojure—to resist the temptation to make another tutorial, to cut back on new Midje features (but not on fixes and the 1.3 port, I hasten to reassure its users!), and so on. If Clojure then wins big, I can jump back on the bandwagon. I will have lost some early mover advantage, but that doesn’t look to be huge anyway. (Not like it was with Agile, or would have been with Ruby if I hadn’t drifted away before Rails came out.)

I haven’t decided what to do. Reaction to this post might help. Probably the best thing would be for me to be convinced I’m nuts. Hey, it’s happened before!

A postscript on the validity of what I claim

A postscript on worries about the effect of performance on expressiveness

I’ve been asked for things to read after my “Functional Programming for Object-Oriented Programmers” course. Here are some, concentrating on the ones that are more fun to read and less in love with difficulty for difficulty’s sake.

If you were more interested in the Clojure/Lisp part of the course:

• The Little Schemer and other books by the same authors. These teach some deep ideas even though they start out very simply. You’ll either find the way they’re written charming or annoying.

• Land of Lisp is a strange book on Lisp.

• If you were totally fascinated by multimethods, you could try The Art of the Metaobject Protocol.

If you were more interested in the functional programming part of the course, the suggestions are harder. So much of the writing on functional programming is overly math-fetishistic.

• Learn You a Haskell for Great Good is another quirky book (lots of odd drawings that have little to do with the text), but it covers many topics using the purest functional language. There’s both a paper version and a free online version. The online version has better drawings (color!).

• I haven’t read Purely Functional Data Structures, though I think I have it somewhere. @SamirTalwar tweeted to me that “I’m enjoying Purely Functional Data Structures. It’s quite mathsy, but with a practical bent.” The book is derived from the author’s PhD thesis.

The approach I took in the course was inspired by the famous Structure and Interpretation of Computer Programs (free online version). Many people find the book more difficult than most of the ones above. Part of the reason is that the problems are targeted at MIT engineering students, so you get a lot of problems that involve resistors and such.