MoB-FAT 1: Controller channels
I am going to write a document called “The Mechanics of Business-Facing Automated Tests”. I’ll be posting drafts of the different chunks here, hoping that you’ll help me improve it. I’m interested in improvements to the ideas and the flow of explanation rather than line-by-line or word-by-word improvements. Thanks.
The sample application is only a realistic sketch. It pretends to be the server part of a system used by travel agents. For my own convenience, I wrote the application in Ruby, but I’m treating it just as I would were it in Java. The tests are written in Ruby for the reasons I gave earlier.
In normal use, the application is a server that communicates with a client and with airline servers. The server communicates to the client via XML messages over a network. To communicate with airline servers, it drops files on disk to be picked up by FTP. Airline servers deliver files in the same way.
Note: in the mockup, I didn’t bother with a database. Should I? It won’t change the implementation much, and it will make it harder for people to run the code at home. But it’s important for examples like these to feel realistic.
To be affordable, tests must have more control than is available in a fielded application. My tests could drive the existing client, but (1) many GUI clients are hard to drive programmatically, and (2) it will be much easier to create the desired XML myself than to poke at the client to generate the right XML indirectly. So that’s what I did. I need to be conscious of what that means, though. Suppose I have a mistaken idea of what kind of XML the client produces. Since the tests and the server will share the mistake, especially if the server is written test-first, a bug will go unnoticed until someone connects the real client to the server.
There is always a tradeoff between the cost of tests and their bug-finding power. There is no industry-standard best practice balance point: every team and product needs to find its own. My personal preference is to err on the side of reduced cost but try to discover mismatches quickly through manual exploratory testing of the whole system. If I keep finding mismatch bugs, I know I need to pay a higher testing cost to have business-facing tests be more completely end-to-end. (Or, preferably, the team should figure out some way to prevent mismatches.)
[Footnote: The Agile attitude is shown by doing a little less than the minimum you can imagine working, then letting reality drive you to add more process, technology, or whatever. It’s much easier to add when you’ve done too little than remove when you’ve done too much. (I think I learned this from either Alistair Cockburn or Jim Highsmith.)]
Since the airline server is third-party software, it can’t be changed to add control. So it needs to be replaced. I chose to replace it with a separate process, the fake airline server. That server is a separate process that looks just like the real one to the server under test. The test tells it what to do through a controller channel, in this case using Ruby’s remarkably easy to use remote procedure call library.
[Footnote: Another alternative would be to have the test itself fake being both the client and the airline server. (That’s like what Michael Feathers calls a self shunt.) That’s probably the easier solution to code in most cases. I’m doing it this way to make it clearer what substitutes for what (and because managing another process is pretty easy in Ruby.)]
I needed to do still more for control. Let’s assume that the server under test drops files for the airline server at ten-minute intervals. I don’t want to wait that long for a test to run, nor do I want to write test support code that checks to see if a new file is available. So I added a controller channel to tell the server “drop your file to airline X right now.”
Controller channels are often controversial, the objection being that I’m now not testing the exact code or configuration that will be put into production. That’s another example of the same tradeoff as before. It is possible that controller channels will introduce or hide bugs. However, consider the alternative: tests that are harder to write. If they’re harder to write, there’ll either be fewer of them (which means that bugs will be missed) or there will be fewer features in the product (because each acceptably tested feature takes longer to create). My usual approach is to use controller channels freely until there’s actual evidence that they’re causing harm, not just worries about it. Again, that means that these automated tests needs to be supplemented with manual ones.
Controller channels are also used to peek into the server under test. Suppose, for example, that a test wanted to know some information that’s available only indirectly: by making six different queries to the server and adding up information from each of them. It’s likely that the information is much more readily available inside the server, so a side channel could be used to ask the server for it directly.
When using controller channels, don’t forget to ask whether what your test needs might be useful to someone else. Perhaps a system administrator trying to work through a problem with an airline would like a way to force the server to drop files at will. It ought to be the case that features available through a controller channel will migrate over to the published interface. This is in keeping with the nature of business-facing tests: they talk about the product in the language of business. Making what business says about the product (1) easy to accomplish and (2) easy to see that it’s been accomplished will better “tune” the product code to the needs of the business.
April 26th, 2007 at 11:15 pm
The trade off around “tests that are harder to write” is an interesting one, in part because what is “harder” is non-static. Brian you know that I deal primarily (but not exclusively) with developer/unit testing, and there’s always that trade-off about how “unit-y” to make tests. Typically tests that are more system-y are “easier to write” but slower to execute. Tests that are more unit-y (in the Michael Feathers sense of not using exernal resources) are “harder to write” but much much faster to execute.
But my view is that the harder is a temporary thing. Once you’ve jumped that hurdle into the harder modality, the tests are no longer harder — they just require a type of thought that is less natural than the “easier” tests. But the benefit of the “harder” style is persistent. The faster tests remain faster.
That said, I usually push people to start with what is easy. I think writing automated tests by itself is enough of an unnatural act that it is worth becoming practiced at whatever form you can manage before trying to expand your scope.
(I’m frequently annoyed by the question on the TDD mailing list “how to I TDD multithreaded code?” because I very much doubt these people are testing all that they can and have only the multithreaded-ness left to test.)
April 27th, 2007 at 12:50 pm
We tend to do a good job with unit testing and with full system testing. However, the integration tests we develop in between these tests that map to your use case tended for a while to be either not useful or counterproductive. Problems frequently turned up when developer X wanted to write a test rig to test their new code via the exposed IPC mechanism. Given that they just implemented a (well unit-tested) interface in their favorite language using their favorite classes, they naturally wrote the test app with the same technology. What usually happened next was that when other developers went to integrate with this code, they used the test process as a model, but other found that their favorite classes that were supposed to do the same thing, especially when crossing languages, turned out to not work _quite_ the same. As such, we now require that when folks do write test apps for their code, they write their test app using different classes and/or a different language than they used to write their real code. We’ve found that this practice hasn’t been too much extra work, and that this has greatly reduced the number of times the test clients cause confusion, discontent, and bug cover-ups.
April 28th, 2007 at 10:13 am
[…] Bella definizione di Brian Marick, su che cos’è l’attitudine all’agilità: The Agile attitude is shown by doing a little less than the minimum you can imagine working, then letting reality drive you to add more process, technology, or whatever. It’s much easier to add when you’ve done too little than remove when you’ve done too much. (I think I learned this from either Alistair Cockburn or Jim Highsmith #. […]
April 28th, 2007 at 1:01 pm
Jeffrey: you’re right about the way the cost drops over time. There’s an even bigger problem with business-facing tests: often the startup cost is borne by a programmer for the benefit of a tester. (The programmer writes the Java fixture, for example, that the tester uses.) That’s one of the reasons you get big unwieldy tests that know a lot of irrelevant detail: there’s no one to factor the detail out of them. That’s an additional benefit to writing the tests in a language, like Ruby, that has something of a shallower learning curve, is less cumbersome for the things tests do a lot of (string processing, stashing things in dictionaries/hashes), and is less persnickety: the testers can do much more of the fixture work themselves.
Timm: that’s an interesting story, thanks. I’d like to hear more details, perhaps with an example of the kind of mismatches you saw.
April 29th, 2007 at 6:49 pm
I’m not clear about whether we’re talking about entirely the same thing here, but let me give an example of what seems to me to be a similar situation.
I’ve had to add credit-card processing to several websites over the past few years. Usually we get some client code that talks to the credit card processing server and a couple of accounts to log in to a test version and a production version of this server. We can run our code against a test server, but this is generally painful for serveral reasons: it’s an external dependency that’s not always available, it’s slow, and the test server keeps state that you need to know about. (For example, the first time in a month when you run a transaction with a given transaction ID, the charge will be successful; it will then fail for the rest of the month.)
My solution is generally to build a mock server (as you’ve done above) that implements the same interface as my code that talks to the real server, but accepting special codes for testing. On this, for example, all “real” credit card numbers will have the charge denied, but the credit card number “valid-card” (obviously not valid in the real world) will always allow the charge. These are the tests that the business users use, both in the automated ruby tests and on the staging server. (It’s important that the “valid” credit card numbers in this test system would always fail on the real system, as this isolates the testing code in the mock server, and ensures that the client code is exactly the same in both testing and production.)
I then write a separate set of tests, used by developers and deployment engineers, that tests our interface code talking to the credit card processing company’s test server, to ensure that we’re talking to it correctly and that it returns the results we think it does. These are run on a regular basis to validate that we’re talking to the credit card processing company correctly.
May 2nd, 2007 at 10:40 am
Yup, that’s what I’m talking about. It’s fairly common. A bit less common is the idea of adding controller channels directly into the product under test.
May 2nd, 2007 at 10:40 am
[…] Earlier in the series: Controller channels […]