Exploration Through Example

Even after all these years, I take too-big steps while coding. In less finicky languages (or finicky languages with good tool support), I can get away with it, but I can’t do that in C++ (a language I’ve been working in the past two weeks). So I’ve had to think more explicitly about taking baby steps. And that’s made me realize that there must be some refactoring literature out there that I’ve overlooked or forgotten. That literature would talk about composing small-scale refactorings into medium-scale changes. The only example I can think of is Joshua Kerievsky’s Refactoring to Patterns. I like that book a lot, but it’s about certain types of medium-scale refactorings (those that result in classic design patterns), and there are other, more mundane sorts.

Let me give an example. This past Friday, I was pairing with Edward Monical-Vuylsteke on what amounts to a desktop GUI app. Call the class in question Controller, as it’s roughly a Controller in the Model-View-Controller pattern. Part of what Controller did was observe what I’ll call a ValueTweaker UI widget. When the user asked for a change to the value (that is, tweaked some bit of UI), the Controller would observe that, cogitate about it a bit, and then perhaps tell an object deeper in the system to make the corresponding change in some hardware somewhere.

That was the state of things at the end of Story A. Story B asked us to add a second ValueTweaker to let the user change a different hardware value (of the same type as the first, but reachable via a wildly different low-level interface). The change from “one” to “two” was a pretty strong hint that the Controller should be split into three objects of two classes:

• A ValueController that would handle only the task of mediating between a ValueTweaker on the UI and the hardware value described in story A.

• A second ValueController that would do the same job for story B’s new value. The difference between the two ValueControllers wouldn’t be in their code but in which lower-level objects they’d be connected to.

• A simplified Controller that would continue to do whatever else the original Controller had done.

The question is: how to split one class into two? The way we did it was to follow these steps (if I remember right):

1. Create an empty ValueController class with an empty ValueController test suite. Have Controller be its subclass. All the Controller tests still pass.

2. One by one, move the Controller tests that are about controlling values up into the ValueController test suite. Move the code to make them pass.

3. When that’s done, we still have a Controller object that does everything it used to do. We also have an end-to-end test that checks that a tweak of the UI propagates down into the system to make a change and also propagates back up to the UI to show that the change happened.

4. Change the end-to-end test so that it no longer refers to Controller but only to ValueController.

5. Change the object that wires up this whole subsystem so that it (1) creates both a Controller and ValueController object, (2) connects the ValueController to the ValueTweaker and to the appropriate object down toward the hardware, and (3) continues to connect the Controller to the objects that don’t have anything to do with changing the tweakable value. Confirm (manually) that both tweaking and the remaining Controller end-to-end behaviors work.

6. Since it no longer uses its superclass’s behavior, change Controller so that it’s no longer a subclass of ValueController.

7. Change the wire-up-the-whole-subsystem object so that it also makes story B’s vertical slice (new ValueTweaker, new ValueController, new connection to the hardware). Confirm manually.

I consider that a medium-scale refactoring. It’s not something that even a smart IDE can do for you in one safe step, but it’s still something that (even in C++!) can easily be finished in less than an afternoon.

So: where are such medium-scale refactorings documented?

I had some thoughts about classes that wouldn’t fit into a talk I’m building about functional programming in Ruby, so I recorded them as a video.

Topics:

• Using hashes instead of classes.

• Classes as a documentation tool—specifically, as a way of making functions easy to find.

• Preferring module inclusion to subclassing (which is akin to preferring adjectives to nouns as a way of organizing the documentation of verbs). (Vaguely similar to duck-typing in Haskell.)

• Object dot notation as a more readable way of writing function composition. (Similar to the motivation for the -> macro in Clojure or type-directed name resolution in Haskell.)

I’m working on a talk called “Functional Programming for Ruby Programmers”. While doing so, I’ve somewhat changed my opinion about mutable state. Here’s my argument, for your commentary. Being new, it’s probably at best half-baked.

1. Back in the early days of AIDS (the disease), I remember a blood-supply advocate saying “You’ve slept with everyone that everyone you’ve slept with ever slept with.”

   The nice thing about immutable state is that it stays virginal and knowable. It is what it was when it was created. With mutable state, it’s more like “Your code might be infected by any code that ever touched your data before (or after!) you first hooked up with it.”

   However, I don’t personally find that a huge problem in programming, debugging, or understanding code. There are other problems I’d rather see fixed first.

2. It’s often said that “code with immutable state is easier to reason about”. I realized a long time ago that there’s some sleight-of-common-language going on there, like the way people who wanted code to be less branchy got rhetorical leverage by renaming branchiness “complexity“.

   In the claim, “reason about” is (I believe) being taken to be synonymous with “prove theorems about”. Pace a whole long trend in artificial intelligence, I don’t believe theorem proving is the basis for, nor a good analogy to, reasoning. Thus I took the “reason about” statement to be a sort of solipsistic wankery from people with roots in the theorem-proving community, not an argument that should sway me, Pragmatic Man!, who left the world of proofs-of-design-correctness around 1984.

   What I’d forgotten is that optimization is theorem proving. One can only optimize if there’s a proof that the optimized code computes the same result as the original. If you consider the desire to implement, say, lazy sequences efficiently, you see how the guarantees that immutability gives are important. Ditto for automatically spreading work to multiple cores.

3. So:

   My previous attitude was pretty much “Yeah, I have a preference for avoiding mutable state. Being immutable isn’t a huge win, but as long as I have people providing me with data structures like zippers, it’s not really any harder than fooling with mutable state. Still, I don’t see why I need an immutable language. If I don’t want to mutate my hashmaps, I just won’t mutate my hashmaps.”

   My new attitude is: “Oh! I see why I need an immutable language.”

I still don’t make a huge deal about immutability. Its benefit is greater than its cost, sure, but it’s not the Thing That Will Save Us.

Here are three Ruby functions. Each solves this problem: “You are given a starting and ending date and an increment in days. Produce all incremental dates that don’t include the starting date but may include the ending date. More formally: produce a list of all the dates such that for some n >= 1, date = starting_date + (increment * n) && date < = ending_date.

Solution 1:

Solution 2:

Solution 3 depends on a lazily function that produces an unbounded list from a starting element and a next-element function. Here’s a use of lazily:

As a lazy sequence, integers both (1) doesn’t do the work of calculating the ith element unless a client of integers asks for it, and also (2) doesn’t waste effort calculationg any intermediate values more than once.

Solution 3:

The third solution seems “intuitively” better to me, but I’m having difficulty explaining why.

The first solution fails on three aesthetic grounds:

• It lacks decomposability. There’s no piece that can be ripped out and used in isolation. (For example, the body of the loop both creates a new element and updates an intermediate value.)

• It lacks flow. It’s pleasing when you can view a computation as flowing a data structure through a series of functions, each of which changes its “shape” to convert a lump of coal into a diamond.

• It has wasted motion: it puts an element at the front of the array, then throws it away. (Note: you can eliminate that by having current start out as exclusive+increment but that code duplicates the later +=increment. Arguably, that duplicated increment-date action is wasted (programmer) motion, in the sense that the same action is done twice. (Or: don’t repeat yourself / Eliminate duplication.))

The second solution has flow of values through functions, but it wastes a lot of motion. A bunch of dates are created, only to be thrown away in the next step of the computation. Also, in some way I cannot clearly express, it seems wrong to stick the inclusive_end between the exclusive_start and the increment, given that the latter two are what was originally presented to the user and the inclusive_end is a user choice. (Therefore shouldn’t the exclusive_start and increment be more visually bound together than this solution does?)

The third solution …

• … is decomposable: the sequence of dates is distinct from the decision about which subset to use. (You could, for example, pass in the whole lazy sequence instead of a exclusive_start/increment pair, something that couldn’t be done with the other solutions.)

• … eliminates wasted work, in that only the dates that are required are generated. (Well, it does store away a first date — excluded_start — that is then dropped. But it doesn’t create an excess new date.)

• … has the same feel of a data structure flowing through functions that #2 has.

So: #3 seems best to me, but the advantages over the other two seem unconvincing (especially given that programmers of my generation are likely to see closure-evaluation-requiring-heap-allocation-because-of-who-knows-what-bound-variables as scarily expensive).

Have you better arguments? Can you refute my arguments?

I’m trying to show the virtues of a lazy functional style. Perhaps this is a bad example? [It’s a real one, though, where I really do prefer the third solution.]

Motivation

Consider a Ruby backend that communicates with its frontend via JSON. It sends (and perhaps receives) strings like this:

Let’s suppose it also communicates with a relational database. A simple translation of query results into Ruby looks like this:

(I’m using the Sequel gem to talk to Postgres.)

On the face of it, it seems odd for our code to receive dumb hashes and arrays, laboriously turn them into model objects with rich behavior, fling some messages at them to transform their state, and then convert the resulting object graph back into dumb hashes and arrays. There are strong historical reasons for that choice—see Fowler’s Patterns of Enterprise Application Architecture—but I’m starting to wonder if it’s as clear a default choice as it used to be. Perhaps a functional approach could work well:

• Functional programs focus on the flow of data through code, rather than on objects with changing state. The former seems more of a match for a typical webapp.

• It’s common in functional languages to lean toward a few core datatypes—like hashes and arrays—that are operated on by a wealth of functions. We could skip the conversion step into objects. Rather than having to deal with the leaky abstraction of an object-relational mapping layer, we’d embrace the nature of our data.

Seems plausible, I’ve been thinking. However, I’ve never been wildly good at understanding the problems of an approach just by thinking about it. It’s more efficient for me to learn by doing. So I’ve decided to strangle an application whose communication with its database is, um, labored.

I’m going to concentrate on two things:

• Structuring the code. More than a year of work on Midje has left me still unhappy about the organization of its code, despite my using Kevin Lawrence’s guideline: if you have trouble finding a piece of code, move it to where you first looked. I have some hope that Ruby’s structuring tools (classes, modules, include, etc.) will be useful.

• Dependencies. As you’ll see, I’ll be writing code with a lot of temporal coupling. Is that and other kinds of coupling dooming me to a deeply intertwingled mess that I can’t change safely or quickly?

This blog post is about where I stand so far, after adding just one new feature.
(more…)

I’ve been asked for things to read after my “Functional Programming for Object-Oriented Programmers” course. Here are some, concentrating on the ones that are more fun to read and less in love with difficulty for difficulty’s sake.

If you were more interested in the Clojure/Lisp part of the course:

• The Little Schemer and other books by the same authors. These teach some deep ideas even though they start out very simply. You’ll either find the way they’re written charming or annoying.

• Land of Lisp is a strange book on Lisp.

• If you were totally fascinated by multimethods, you could try The Art of the Metaobject Protocol.

If you were more interested in the functional programming part of the course, the suggestions are harder. So much of the writing on functional programming is overly math-fetishistic.

• Learn You a Haskell for Great Good is another quirky book (lots of odd drawings that have little to do with the text), but it covers many topics using the purest functional language. There’s both a paper version and a free online version. The online version has better drawings (color!).

• I haven’t read Purely Functional Data Structures, though I think I have it somewhere. @SamirTalwar tweeted to me that “I’m enjoying Purely Functional Data Structures. It’s quite mathsy, but with a practical bent.” The book is derived from the author’s PhD thesis.

The approach I took in the course was inspired by the famous Structure and Interpretation of Computer Programs (free online version). Many people find the book more difficult than most of the ones above. Part of the reason is that the problems are targeted at MIT engineering students, so you get a lot of problems that involve resistors and such.

Here is a draft blurb for a course I’d like to teach, possibly in Valladolid, Spain, in September. I’d like to see if enough people are interested in it, either there or somewhere else. (There’s a link to a survey after the description.)

Short version

• Target audience: Programmers who know Java, C#, C++, or another object-oriented language. Those who know languages like Ruby, Python, or JavaScript will already know some—but not all!—of the material.

• The goal: To make you a better object-oriented programmer by giving you hands-on experience with the ideas behind functional languages. The course will use Clojure and JavaScript for its examples. You’ll learn enough of Clojure to work through exercises, but this is not a “Learn Clojure” course.

• After the course: I expect you to return to your job programming in Java or Ruby or whatever, not to switch to Clojure. To help put the ideas into practice, you’ll get a copy of Dean Wampler’s Functional Programming for Java Developers.

Syllabus

• Just enough Clojure
• Multimethods as a generalization of methods attached to objects
• Protocols as a different approach to interfaces
• Functions as objects, objects as functions
• Mapping and folding in contrast to iteration
• Functions that create functions
• Laziness and infinite sequences
• Time as a sequence of events
• Generic datatypes with many functions versus many small classes
• Immutable data structures and programs as truth-statements about functions

Longer version

Many, many of the legendary programmers know many programming languages. What they know from one language helps them write better code in another one. But it’s not really the language that matters: adding knowledge of C# to your knowledge of Java doesn’t make you much better. The languages are too similar—they encourage you to look at problems in pretty much the same way. You need to know languages that conceptualize problems and solutions in substantially different ways.

Once upon a time, object-oriented programming was a radical departure from what most programmers knew. So learning it was both hard and mind-expanding. Nowadays, the OO style is the dominant one, so ambitious people need to proactively seek out different styles.

The functional programming style is nicely different from the OO style, but there are many interesting points of comparison between them. This course aims to teach you key elements of the functional style, helping you take them back to your OO language (probably Java or C#).

There’s a bit more, though: although the functional style has been around for many years, it’s recently become trendy, partly because language implementations keep improving, and partly because functional languages are better suited for solving the multicore problem^* than are other languages. Some trends with a lot of excitement behind them wither, but others (like object-oriented programming) succeed immensely. If the functional style becomes commonplace, this course will position you to be on the leading edge of that wave.

There are many functional languages. There are arguments for learning the purest of them (Haskell, probably). But it’s also worthwhile to learn a slightly-less-pure language if there are more jobs available for it or more opportunities to fold it into your existing projects. According that standard, Clojure and Scala—both hosted on the JVM—stand out. This course will be taught mainly in Clojure. Where appropriate, I’ll also illustrate the ideas in JavaScript, a language whose good parts were strongly influenced by the functional style.

While this particular course will concentrate on what you can do in Clojure, it’d be unfair to make you do all the work of translating these ideas into your everyday language. To help you with that, every participant will get a copy of Dean Wampler’s Functional Programming for Java Developers, which shows how functional ideas work in Java.

Your instructor

Brian Marick was first exposed to the functional style in 1983, when the accident of knowing a little bit of Lisp tossed him into the job of technical lead on a project to port Common Lisp to a now-defunct computer architecture. That led him to a reading spree about all things Lisp, the language from which the functional style arguably originated. He’s been a language geek ever since, despite making most of his living as a software process consultant. He’s the author of the popular Midje testing library for Clojure and has written two books (Everyday Scripting with Ruby and Programming Cocoa with Ruby).

For examples of Marick’s teaching style, read his explanations of the Zipper and Enlive libraries or watch his videos on Top-down TDD in Clojure and Monads.

^*The multicore problem is that chip manufacturers have hit the wall making single CPUs faster. The days of steadily increasing clock rates are over. Cache sizes are no longer the bottleneck. The big gains from clever hardware tricks have been gotten. So we programmers are getting more cores instead of faster cores, and we have to figure out how to use multiple cores in a single program. That’s a tough problem.

The State monad is a big jump for the monad-learner. It jumps up a level of abstraction by using functions as values. That makes it confusing because you have functions that work on functions, and it can be hard to keep track of whether a particular function is a this-kind-of-function or a that-kind-of-function. I try to tease apart the Gordian Knot thusly:

1. I start by creating a logging monad that simply logs the value of each step. This is fairly straightforward.

2. Next, I decide I want only particular steps to be logged. Steps to be logged use a log function which returns either a plain value or a wrapped value. That leads to an if statement in the “decider”.

3. Then I raise the question: can we get rid of the if statement by pushing its work down into the log function? In this section of the tutorial, I do something like follow Beck’s rules of design: make a series of undirected local changes that arrive at something globally coherent. Specifically, that first simple decision of pushing work into one place forces us to implement the full State monad (though I only show it used for building a log).

4. Finally, I use that solution to illustrate some key concepts: base values, monadic functions, monadic values, and how they’re put together to make up a monad.

I have my doubts about how well this works, particularly the 3d step. I’d value your opinion.

http://www.vimeo.com/21307543

As I mentioned earlier this year, I’m looking to make one of my decadal career shifts. Since that decision, I’ve been doing part-time contract work on a RubyCocoa application, and I’ve found it satisfying to deliver working software to people who are happy to get it. It’s also helped with the nagging dread that—while I can talk the talk about programming, testing, refactoring, and all that—I wouldn’t be able to walk the walk. It turns out I can. Although I’m slower than I’d like, I do respectable work.

In my ideal contract, I’d:

• … code in Clojure, Ruby, or Javascript. Other than that, I don’t require super-advanced or cool technology, but I do have a hankering to work on something that could somehow be the inspiration for another book. I don’t care about the domain.

• … devote 1/2 to 3/4 of my time to a single project, working with a single team, over a period of months. Some portion of that—a week or two a month—would be spent onsite. (Chicago would be the best place because it’s easily accessible by train. I live in Central Illinois.)

• … work at a sustainable pace, and be given the leeway to do a good job by my standards. I’m trying to be artisanal about my code.

  (”I want to be artisanal” might raise red flags: will I decide I know what’s needed better than those who are paying for it? My saving grace is that I have a Labrador-like eagerness to please. I want product owners to smile when they think of me.)

• … be able to stretch by occasionally going slower while I experiment with techniques. (My work on outside-in TDD in Clojure is an example.) I’m willing to be paid less in order to improve faster.

• … be in frequent contact with the people who’ll viscerally appreciate the features they get for the money they spend. That given, I don’t care whether I am working directly for a product company or as a subcontractor on behalf of a contract programming company.

• … work in an Agile style. (I almost didn’t think to include this, since I assume anyone interested in hiring me would expect or accept that. I’m not interested in a job teaching the glories of continuous integration or TDD or refactoring. I’m interesting in learning how to do them ever better, and in working with people who have the same interests.)

However, there may not be an ideal, and I don’t intend to be rigid about opportunities. I could see, for example, working with several teams at once, being someone who helps convert a daily grind into an exploration of new techniques. That’d be more like my consulting past, but I’d be more hands-on than I have in the past, involved for longer, and feel more responsible for the product.

Also: although I listed Chicago as my desired location, it has drawbacks when it comes to (1) winter and (2) helping me with my (currently somewhat faltering) attempt to learn Spanish. I wouldn’t mind working in Costa Rica, Argentina, or elsewhere in Latin America (probably for a longer continuous chunk of time onsite).

I don’t have a huge portfolio of code to show you. What I have is on Github. Critter4Us shows my Cappuccino and Ruby code. My Clojure code is limited to Midje, which is a programmer’s tool rather than an end-user project.

My email address is marick@exampler.com.

Part 1
Part 2

All that remains is to add the locations of bordering cells to a given sequence of locations. A small wrinkle is that a border location may be next to more than one of the input locations, so duplicates need to be prevented. I could write the test this way:

I don’t like that. When the test fails, it’ll likely be hard to discover how the expected and actual values differ. And I bet that a failure would more likely be due to a typo in the expected value than to an actual bug. The test is just awkward.

I like this version better:

It is a clearer statement of the relationship between two concepts: a location’s neighborhood and the border of a set of locations. More pragmatically, it’s less typing. (When I first started coding in this style, it surprised me how much test setup clutter went away. I had much less need for “factories” or “fixtures” or “object mothers“.)

Given either test, the code is straightforward:

Onward

No matter how satisfying the individually-tested pieces, the whole has to work, which is why everything ends by running at least one end-to-end test. A test like the one we started with:

Because I’m making up my test notation as I go, I’ve been running all the tests manually in the REPL. Now I can run the whole file:

If you look carefully, you’ll see that the test would fail because the locations are in the wrong order. But that’s a quick fix:

Done. Ship it!

Development order - Test-down or REPL-up?

It’s often said that the Lisps are bottom-up languages, that you test out expressions in the REPL, discover good functions, and compose them into programs. A lot of people do work that way. A lot of people who use TDD to write object-oriented code also work that way: when implementing a new feature, they start at low-level objects, add whatever new code the top-level feature seems to demand of them, then use those augmented objects in the testing of next-higher-level objects.

For I guess about a year now, I’ve been experimenting with being strictly top-down in some projects. I find that leads to less churn. Too often, when I go bottom-up, I end up discovering that those low-level changes are not in fact what the feature needs, so I have to revisit and redo what I did.

I get less disrupted by rework when I go from the top down (or from user interface in). It’s not that I don’t blunder—we saw one of those in the previous installment—but those blunders seem easier to recover from.

That’s not to say that I don’t use the REPL. We saw that a little bit in this program, when I was writing neighbors. It’s perfectly sensible to do even more in the REPL. I think of the REPL as a handy tool for what XP calls a spike solution and the Pragmatic Programmers call tracer bullets. When I’m uncertain what to do next, the REPL is a tool to let me try out possibilities. So I might be stuck in a certain function and go to the REPL to see how it feels to build up parts of what might lie under it. After I’m more confident, I can continue on with the original function, test-driven, reusing REPL snippets when they seem useful.

I don’t claim everyone should work that way. I do claim it’s a valid style that you’d be wise to try.

Notation

I’m something of an obsessive about test notation, and I’ve been endlessly fiddling with a Clojure mock notation. I implemented its first version as a facade on top of clojure.contrib.mock. As I experimented, though, I found that keeping my facade up to date with my notational variants slowed me down too much, so I put the code aside until the notation settled down.

I’m pretty happy with what you’ve seen here. Are you? If so, I may start on another mock package. I’ve got the most important parts: a name and sketch of a logo. (”Midje” and someone flying safely between the sun [of abstraction without examples] and the sea [of overwhelming detail].)

Tests and code together

You can see the completed program here. It mixes up tests and code. I’ve tried that on-and-off over the years and always reverted to separate test files. This time, it’s seemed to work better. I probably want an Emacs keystroke that lets me hide all tests, though. I’d also want alternate definitions of the test macros so that I can compile them out of the production system

What next?

I’ll write a web app in this style, using Compojure.