Embedded vs. independent testers
Bruce Daley posts on how most humans are biased to think they’re less error-prone than they are. As far as I know, that’s a claim solidly based in empirical research. (See also Bruce Schneier’s The Psychology of Security.) From this, he concludes:
Given the nature of their work, software developers and software programmers suffer more from the illusion of knowledge and the illusion of control than most other professions, making them particularly subject to over-looking mistakes in their own code. Which is why software needs to be tested independently.
However. Consider the graph below.
Here, the programmer and independent tester start testing at the same time. (Bad programmer! Bad!) The programmer starts out with more knowledge of the app than the tester (the line marked P/+), but she also has a large amount of cognitive bias (P/-) and lacks testing skill. That makes her miss bugs her knowledge would otherwise allow her to find (the area under the red line). Moveover, her biases seem to be pretty impervious to evidence.
The tester starts out with less knowledge, but has no (relevant) cognitive biases at all. Also, his testing skill lets him ramp up his bug finding pretty fast—but it still takes him a while to overcome her advantage.
Which do you want doing the testing? If you’re shipping at time A, it looks like the programmer has the edge. (Compare the shaded areas under the curve.)
We could expect that advantage to erode over time. If the ship date is farther out, the independent tester would have an advantage, as this graph shows:
Even when all that matters is bug count, the decision is not straightforward, especially since it’s based on information you can’t know until after you’ve decided. (How long will it take the tester to get up to speed? How many and what kind of bugs will the programmer miss?)
On most projects, there are lots of other factors to consider.
So I encourage people not to make the assertion the post’s author does.
March 10th, 2008 at 5:28 pm
It is hard for me to argue with the man who wrote the book on Ruby Testing, but it seems the more knowledge the programmer has the more likely she is to make a mistake based on the illusion of knowledge. As the tester gains knowledge in your second graph he too will be more and more inclined to overlook mistakes. Of course it is much better for both the programmer and the tester to be looking for bugs - which correct me if I am wrong - is one of the key benefits of agile development.
March 11th, 2008 at 8:43 am
Hi Brian
How would you draw the curve for a tester that sits very close to the developers, and has a lot of knowledge of the app, and testing skills?
Regards,
Rikard
March 15th, 2008 at 4:40 pm
Are these models verified? I’m wary that we’re just assuming that the independent tester will have a steeper slope (purely by virtue of independence).
March 24th, 2008 at 5:54 pm
Bruce: I agree that the illusion of knowledge is a real effect, but I doubt the programmer’s greater ability to miss bugs would overcome the advantage of greater starting true knowledge - at least in that early period.
Rikard: It seems to me that embedded testers do fall prey to the illusion of knowledge (they get infected by some of the programmers’ beliefs in things that ain’t so), but they’re also quicker at learning things that are so - and that they can exploit to find bugs.
Jason: No, they’re not verified, but my experience seems to have always been that I start out finding few bugs because I don’t know enough about the app yet, then my productivity goes zooming up as I find my footing, and eventually the slope flattens out when I’ve fully exploited my knowledge, am not learning more very fast, and - ideally - have found most of what there is to find.
March 28th, 2008 at 9:36 am
Brian, to me it seems like an empirical approach to this problem might help to tease out other factors lurking here.
One detail that popped out to me is that “knowledge” is static in this model. I would think that over the course of development the programmer will be generating more code that the independent tester will then need to learn (i.e. new knowledge). The rate of generation of new knowledge by the programmer will vary from project to project, and the ability to absorb new knowledge by the tester will also vary. There might be other kinds of bias as well, for instance if both the tester and the programmer both have a similar training in the specific domain being tested this might create additional bias that would not be eliminated by an external tester since the assumptions would be shared by both parties.
It also seems that close association between the tester and the programmer would increase the rate of knowledge absorption for the tester, but this close association might also potentially cause the independent tester to absorb the biases of the programmer. So it may be that there would be a third red line representing tester bias called T/- which would increase as the tester learns the code base and domain area.