Exploration Through Example

Example-driven development, Agile testing, context-driven testing, Agile programming, Ruby, and other things of interest to Brian Marick
191.8 167.2 186.2 183.6 184.0 183.2 184.6

Wed, 24 Aug 2005

Still more on counterexamples

Due to conversations with Jonathan Kohl and John Mitchell, a bit more on counterexamples.

I now think that what I'm wondering about is team learning. I want to think more about two questions:

  • Say someone comes up with a counterexample, perhaps that one kind of user uses the product really differently. How is that integrated into the mindset of the team? That is, how does it become an example of an extended model of product use? (I fear too often it stays as an awkward, unintegrated counterexample.)

    Take the blocks world example. In Winston's work, he taught a computer to identify arches by giving it examples and counterexamples. (Eugene Wallingford confirms that the counterexamples were necessary.) In that world, an arch was two pillars of blocks with a crosspiece. The counterexamples included, if I remember correctly, arches without a top (just two pillars) and maybe a crosspiece balanced on a single pillar.

    It's fine and necessary for a researcher to teach a computer - or a product owner a development team - about already understood ideas like "arch". But it's even more fine when the process of teaching surprises the teacher with a new, useful, and more expansive understanding of the domain. I want more surprise in the world.

  • Is there a way to give counterexamples elevated importance in the team's routine action? So that it isn't exceptional to integrate them into the domain model?

    One thing testers do is generate counterexamples by, for example, thinking of unexpected patterns of use. What happens when those unexpected patterns reveal bugs? (When, in Bret Pettichord's definition of "bug", the results bug someone.) The bugs may turn into new stories for the team, but in my experience, they're rarely a prompt to sit down and think about larger implications.

    An analogy: that's as if the refactoring step got left out of the TDD loop. It is when the programmer acts to remove duplication and make code intention-revealing that unexpected classes arise. Without the refactoring, the code would stay a mass of confusing special cases.

    Sometimes - as in the Advancer example I cite so compulsively - the unexpected classes reflect back into the domain and become part of the ubiquitous language. So perhaps that reflection is one way to make incorporating counterexamples routine. We tend to think of the relationship between product expert and team as mainly directional, one of master to apprentice: the master teaches the apprentice what she needs to know. Information about the domain flows from the master to the apprentice. There's a conversation, yes, but the apprentice's part in the conversation is to ask questions about the domain, to explain the costs of coping with the domain in a certain way, to suggest cheaper ways of coping - but not to change the expert's understanding of the domain. Perhaps we should expect the latter.

    Put another way: suppose we grant that a project develops its own creole - its own jargon - that allows the domain expert(s) and technical team to work effectively with each other. Something to keep casual track of would be how many nouns and verbs in the creole originated in the code.

## Posted at 08:02 in category /ideas [permalink] [top]

Mon, 22 Aug 2005

More on counterexamples

Andy Schneider responded to my counterexamples post with the following. I think they're neat ideas.

  1. I express project scope in terms of what the project is delivering and what it is not delivering. I learnt to do this in 1994, after listening to a bunch of people interpret my scope statements in different ways, depending on what they wanted to read into them. On the surface it seems daft to list all the things a project is not, it'd be a long list. However, there is always some obvious set of expectations you know you're aren't going to fill and some obvious confusions. I use those to draw up my 'Is Not' Lists.

  2. I'm writing a lot of papers laying down design principles for common architectural scenarios, trying to get some re-use at the design level and also trying to improve productivity by having the boring stuff already sorted for 80% of the cases. I communicate my principles with a narrative text within which the user 'discovers' the principles (which I highlight so it can be read by consuming the principles). At the end of the paper I normally write a section labelled something like implications. Here i walk through a set of counter-examples that describe practices that contradict the principles. This gets people to think about the implications of what's being said. Creates me a bunch of work working through the feedback, as these sections always elicit more feedback than the rest. If I didn't provide counter-examples no one would consider the space not covered or excluded by the principles.

    So, I've learnt it is useful, I have seen the fact it gets people to think about what something is not and the feedback from people is always better for it. In many ways it is opposite to a politician's approach, where they avoid counterexamples because they want you to read into their words what you want to. They don't want you to consider the space not covered or excluded.

(Reprinted with permission.)

## Posted at 08:47 in category /ideas [permalink] [top]

Mon, 08 Aug 2005


In my thinking about tests as examples, I've been thinking of them as good examples:

The right system behaves like this. And like this. And don't forget this.

But what about counterexamples?

A system that did this would be the wrong system. And so would a system that did this.

There's some evidence that differences are important in understanding.

  • The linguist Ferdinand de Saussere taught that meaning of the word "boat" isn't "a small vessel for travel on water." Rather the meaning of "boat" is generated by contrast with other words like "ship", "raft", "yawl", "statue of a boat", etc. (Derrida would later go on to make perhaps too much of the fact that there's no limit to the recursion, since all those other words are also defined by difference.)

  • In the early '70s, Patrick Winston wrote a program that learned the concept of "arch" from a series of examples and "near misses". My copy of his book has long since gone to the place paperclips, coathangers, and individual socks go, so I can't check if the near-miss counterexamples merely improved the program or were essential to its success.

  • My kids are now of the age (nine and ten) where they ask for dictionary-like definitions of words. But when they were younger, they more obviously learned by difference: they'd point at something, give the wrong name, then accept the correction without further discussion. ("Duck." "No, that's a goose." "Dog." "Yes, a nice dog.") Presumably the counterexamples helped with that amazing burst of vocabulary young kids have.

So what about those times when the programmer proudly calls the product owner over to show the newest screen and watches her face fall just before she says, "That's not really what I had in mind"? Or those times when a small group is talking about a story and a programmer pops up with an idea or a supposed consequence that's wrong? That's an opportunity to - briefly! - attend to what's different about the way two people are thinking.

Does anyone make explicit use of counterexamples? How? What have you learned?

## Posted at 20:05 in category /ideas [permalink] [top]

Thu, 28 Oct 2004

"Methodology work is ontology work" posted

Now that I've presented my paper at OOPSLA, I can post it here (PDF).

Here's the abstract:

I argue that a successful switch from one methodology to another requires a switch from one ontology to another. Large-scale adoption of a new methodology means "infecting" people with new ideas about what sorts of things there are in the (software development) world and how those things hang together. The paper ends with some suggestions to methodology creators about how to design methodologies that encourage the needed "gestalt switch".

I earlier blogged the extended abstract.

This is one of my odd writings.

## Posted at 09:37 in category /ideas [permalink] [top]

Sat, 24 Jul 2004

Methodology work is ontology work

I've had a paper accepted at OOPSLA Onward. I had to write a one-page extended abstract. Although I can't publish the paper before the conference, it seems to me that the point of an abstract is to attract people to the session or, before then, the conference. So here it is. I think it's too dry - I had to take out the bit about bright cows and the bit about honeybee navigation - but brevity has its cost.

(As you can guess from the links above, the paper is a stew of ideas that have surfaced on this blog. I hope the stew's simmered enough to be both tasty and nourishing.)

I argue that a successful switch from one methodology to another requires a switch from one ontology to another. Large-scale adoption of a new methodology means "infecting" people with new ideas about what sorts of things there are in the (software development) world and how those things hang together. The paper ends with some suggestions to methodology creators about how to design methodologies that encourage the needed "gestalt switch".

In this paper, I abuse the word "ontology". In philosophy, an ontology is an inventory of the kinds of things that actually exist, and (often) of the kinds of relations that can exist between those things. My abuse is that I want ontology to be active, to drive people's actions. I'm particularly interested in unreflective actions, actions people take because they are the obvious thing to do in a situation, given the way the world is.

Whether any particular ontology is true or not is not at issue in the paper. What I'm concerned with is how people are moved from one ontology to the other. I offer two suggestions to methodologists:

  1. Consider your methodology to be what the philosopher of science Imre Lakatos called "a progressive research programme." Lakatos laid out rules for such programmes. He intended them to be rules of rationality, but I think they're better treated as rules of persuasion. Methodologies that follow those rules are more likely to attract the commitment required to cause people to flip from one system of thought to another (from one ontology to another) in a way that Thomas Kuhn likened to a "gestalt switch".

  2. It's not enough for people to believe; they must also perceive. Make what your methodology emphasizes visible in the world of its users. In that way, methodologies will become what Heidegger called ready-to-hand. Just as one doesn't think about how to hold a hammer when pounding nails, one shouldn't think about the methodology, its ontology, and its rules during the normal pace of a project: one should simply act appropriately.

Methodologies do not succeed because they are aligned with some platonic Right Way to build software. Methodologies succeed because people make them succeed. People begin with an ontology - a theory of the world of software - and build tools, techniques, social relations, habits, arrangements of the physical world, and revised ontologies that all hang together. In this methodology-building loop, I believe ontology is critical. Find the right ontology and the loop becomes progressive.

## Posted at 13:06 in category /ideas [permalink] [top]

Thu, 11 Mar 2004

Telling the code to tell you something

Two related ideas:

  1. Chad Fowler builds on an idea from Dave Thomas to show dynamically-typed code that documents the types it probably wants. Dave's idea was that the first type a method's called with is probably the right type. So, in an aspectish way, you point at the method, tell it to remember how it's called and to complain if later calls are different. Chad's idea is that a method can just as easily record the information in a way that lets you create documentation about the types of method arguments.

    Chad's idea of using the results to inform an IDE is particularly clever.

  2. Agitator is a new testing tool that semi-intelligently floods a program with data and records interesting facts about what happens. (Full disclosure: I've received some money from the company that sells it.) Kevin Lawrence tells a story of how the desire to simplify the results Agitator produced for some code resulted in a small class hierarchy replacing much more code smeared all over the place. The end result and the feel of the process is the standard test-driven refactoring story, but the "impulsive force" was interestingly different.

    (See also a different story, from Jeffrey Fredrick. To me, it's a story about how Agitator's large number of examples hinted that there's a bug somewhere and that maybe an assertion right over there would be a good way to trap it.)

The common thread is quickly instrumenting your program, running a lot of ready-to-hand examples through it, getting some output that's not too noisy, and deriving value from that. Not a new idea: how long have profilers been around? But now that "listening to what the code is trying to tell us" is marginally more respectable (I blithely assert), we should expand the range of conversation. It shouldn't be only about the code's static nature, but also about its dynamic nature. And perceiving the dynamic nature should be as simple, as semi-automatic, even as rawly perceptual as, say, noticing duplication between two if statements.

## Posted at 20:30 in category /ideas [permalink] [top]

About Brian Marick
I consult mainly on Agile software development, with a special focus on how testing fits in.

Contact me here: marick@exampler.com.




Agile Testing Directions
Tests and examples
Technology-facing programmer support
Business-facing team support
Business-facing product critiques
Technology-facing product critiques
Testers on agile projects

Permalink to this list


Working your way out of the automated GUI testing tarpit
  1. Three ways of writing the same test
  2. A test should deduce its setup path
  3. Convert the suite one failure at a time
  4. You should be able to get to any page in one step
  5. Extract fast tests about single pages
  6. Link checking without clicking on links
  7. Workflow tests remain GUI tests
Permalink to this list


Design-Driven Test-Driven Design
Creating a test
Making it (barely) run
Views and presenters appear
Hooking up the real GUI


Popular Articles
A roadmap for testing on an agile project: When consulting on testing in Agile projects, I like to call this plan "what I'm biased toward."

Tacit knowledge: Experts often have no theory of their work. They simply perform skillfully.

Process and personality: Every article on methodology implicitly begins "Let's talk about me."


Related Weblogs

Wayne Allen
James Bach
Laurent Bossavit
William Caputo
Mike Clark
Rachel Davies
Esther Derby
Michael Feathers
Developer Testing
Chad Fowler
Martin Fowler
Alan Francis
Elisabeth Hendrickson
Grig Gheorghiu
Andy Hunt
Ben Hyde
Ron Jeffries
Jonathan Kohl
Dave Liebreich
Jeff Patton
Bret Pettichord
Hiring Johanna Rothman
Managing Johanna Rothman
Kevin Rutherford
Christian Sepulveda
James Shore
Jeff Sutherland
Pragmatic Dave Thomas
Glenn Vanderburg
Greg Vaughn
Eugene Wallingford
Jim Weirich


Where to Find Me

Software Practice Advancement


All of 2006
All of 2005
All of 2004
All of 2003



Agile Alliance Logo