Test planning documents
I’m going to start copying my replies to questions on agile-testing here.
I am interested in hearing how people that don’t think a Test Strategy should exist go about answering the following questions and when they do it. […]
Number of testers required—including what skills they need (i.e. (Performance, Security, Domain Knowledge, Technical Knowledge) and where they will come from (current employee’s, contractors etc)
Environments, especially for new projects. What is going to be needed, how much will it cost, when will it be required. (Some hardware can take 6-8 weeks to source)
Data—what data will be used, where will it come from, privacy concerns
External/Dependant requirements—for example will real users be involved in UAT/Usability Testing? When will they be involved? (If it is “general” public this needs time to organise, if they are in another country/offsite do you want someone to be there to monitor/oversee what they are doing). Is a usability lab going to be used? Does the product need to be benchmarked by an external company? When? etc.
Software/Tools—if anything needs to be purchased this should be highlighted.
You’ve given a set of questions that need to be answered. My bias would be to answer them at the last possible minute. Let’s take usability testing, and assume it’s a scarce resource. It takes two months to set it up. You’ll only be able to do it twice.
You want to do the first session early enough that it tells you that you’re heading down some grossly wrong path before you’re too far along. But if you do it too early, the information you get may be made partially irrelevant because the product changes direction or conception for whatever reason—and now you’ve blown one of only two opportunities.
As Cem Kaner says, you know less at the beginning of a project than you ever will again. So at the beginning, I’d ask whether our best guess is that we’ll need usability testing after the second month. If we actually think it will be closer to the start of month four, why should we say anything more than “we don’t need to start arranging it yet”?
The end of the first month would be a good time to re-evaluate that guess. Do we now think it would be better after month three?—if so, we should schedule it now. Or will it be better after month five?— good thing we didn’t already schedule the lab, since it would at the very least be an annoyance to reschedule.
And so on.
At any given moment in the project, you’ll have answers that are OK but not as good as they could be, exactly as you’ll have product that’s OK but not as good as it could be. You’ll have written the answers down so that you don’t forget them. If there’s reasoning that you think likely will be forgotten by the next time you revisit the question, you should write that down too. These things you write down—information to help yourself—will not make sense to anyone not privy to the conversations that led to them.
Some people outside the conversations will not be happy with that:
-
They won’t understand “Usability—two months lead time, est. iteration 10″ and they’ll want the answer written out in more detail. Easy enough.
-
They’ll want your reasoning written down so that they can check it or at least reassure themselves that you’ve made a sensible effort. They want to know why they should trust you, and talking to you doesn’t suffice.
-
They’ll probably want to know—and maybe even in writing—why you can’t predict the entire testing future of the project now.
It’s important to realize that the work of making them happy is of no benefit to you. They are asking you to do work—work that costs money—for the benefit of an external stakeholder (themselves). My basic attitude in such situations is “talk to the product owner.” She is the person/role who’s responsible for deciding whether any chunk of work x is a better expenditure of money right now than whatever else I could be doing. It’s her job to reconcile the irreconcilable differences among all the interest groups who care about the product. So when she adds it as a story to the iteration, I’ll do it.
July 19th, 2007 at 1:47 pm
“Let’s take usability testing, and assume it’s a scarce resource.”
Steve Krug’s _Don’t Make Me Think_ and Nielsen [http://www.useit.com/alertbox/20000319.html] suggest that we can do less with fewer people than we imagine, to good effect.
regards
Bob
July 20th, 2007 at 1:55 am
Indeed! On my 10 years testing experience my testing never benefit from sharing any testing planning documents with stakeholders: they never care to read them careful enough. I also prefer “talk to the product owner” and experts to learn alternative paths, but select among them dynamically as the project goes on.
I’ve faced two issues however. Do you? How do You solve/avoid them?
1)Stakeholders (executives) love to be in control. They don’t like such a dynamics. Going against them you risk to be fired, don’t you? Being too independent you loose their support (no money for tools, training, conferences, etc.)
2)How about morale of test team: they don’t know what to expect, and how to prepare (i.e. read some articles/books on usability). Do you also just talk to them and how much tell them?
July 23rd, 2007 at 8:00 am
Ainars:
1) If it’s that important to them, I give them what they want but tell them that if the team is working well, they should soon stop wanting that. The legitimate reason for wanting to see these planning documents is because they’ve traditionally not been able to know how things are going. An Agile project ought to have so much visibility - be so practically exhibitionist in showing its status - that there’s no need for indirect kinds of reassurance.
That doesn’t work for illegitimate reasons: command and control personality, signaling status by forcing people to do silly things, etc.
2) I think the whole team should be able to explain what’s going on because - to at least some extent - the whole team is involved in the ongoing planning. For example, people are going to be discussing, once per iteration, when the usability testing is going to happen. So people will know what to expect because they’ll be continually reminded of it.
July 26th, 2007 at 5:35 am
All good points but:
The saying ‘When the sh!t hits the fan…’ a test strategy is a good doc to have around
August 2nd, 2007 at 2:16 pm
My company has just recenlty moved to using agile development, and as a QA team lead in our prior waterfall environment, I find myself currently trying to create a usable test plan that can be presented as supporting documentation for the acceptance criteria. However, I’ve been trying to avoid making it any kind of “sign-off” document. Instead, I see it more as a helpful document to get everyone on the same page of what is and isn’t going to be tested. With the complexity of our product, it’s very easy for a developer to forget (or for a product owner to never know) certain functional areas that would need to be tested or regressed for a specific story.
I’m not sure this document will be needed in the future as we refine our agile processes, but the information contained in the test plan, if not the traditional usage, is still valuable right now and can hopefully help spread a little testing mentality to the whole team.