Test Technique – Writing a test plan document
Warning – 3/4 finished thought.
In the event of an unknown or unexpected system behavior in the software world – the test plan documents mean nothing.
When we develop our testing strategy we do two things – outline what we WILL test AND what we are NOT going to test. The latter is always one of the hardest to figure out, explain and do effectively. A close relationship with development is required. Trust from both sides is crucial. Expert knowledge of the system and how changes affect it is required. 95% of the time this process works but other times it backfires. It is usually the unknowns that screw us.
For project X a lead tester wrote a test plan document to satisfy project stakeholders. The document brought up lots of good questions and discussion. The one section that got a fair amount of discussion was the “Not going to test” section. We explicitly stated that PERFFORMANCE would NOT be TESTED. We would eyeball the performance (web based UI connected to our 4 year old core server technology) and not provide a formal analysis of request/response times. Why take this approach? We had done load testing with a shipping release build 1 month prior. We got benchmarks, provided this info to development and discussed the risks. Everything looked good. There was no reason for any of us to suspect performance problems. Everyone agreed “There should be no reason we would have performance problems”. All project stakeholders “signed off” on the test plan.
What do you think the single biggest problem found while testing?
PERFORMANCE!! (It would be hard for me to make my point if it wasn’t)
Deceived by our own test plan document. Our own analysis. Our own agreements. Our own judgements!! We spent nearly 4 weeks of development and testing investigating performance problems.
Everyone has a plan – until they get punched in the face. – Mike Tyson, Boxer.
There are two directions we could have gone from here. 1) Stick with the test plan. After all – we all agreed. 2) Devise a new plan of attack and throw the original document out the window. In exploring these options I knew that we would ultimately adapt and devise a new plan of attack but I decided to explore option number 1 a little bit. If we stuck with the document what are some things we could do to make it work?
Ask for more time?
This is the usual approach. You can ask for it - no problem there. Wanting something and expecting it are two different things (Weinberg). The chances of getting it in this context were slim to none. Why? The product release was tied to a conference show it would be released to the public. So there was no moving the date. I suppose we could work late nights and weekends.
Ask for more people?
All the other testers were on other projects. The priority of the project with the performance problem was highest so we could have shuffled people around. All our other projects were high priority as well. The overhead in training them on the new product, explaining known issues, getting hardware etc seemed like too much to deal with. It would take away from the testing we needed to get done.
Ask for more hardware?
It wouldn’t arrive in time to allow us to be productive.
Do a CYA (Cover your As*)
Use the the test plan to say “We said we weren’t going to test for performance and because everyone signed off – we aren’t going to do it now.” Well we could have done that but it’s not a road that would have lead us to a productive place. It might have been a way to get management’s attention. I might have been looking for a new job right now.
What did we actually do?
We adapted. We worked hard and got it done. We worked weekends and late nights. We split our time as best we could between performance and doing system regression tests in other areas of the product. We continued to be clear what we weren’t getting to because of this testing. When the product was released we didn’t stop testing. We kept covering areas that didn’t have as much attention as they should have. Did we find things that we missed because we were focused on performance? You bet. Did we address the important issues and decide if we can live with it or not? Yup.
“You have to be fast on your feet and adaptive or else a strategy is useless.” Charles de Gaulle
In this context – having a sticking to the test plan document could have would have deceived the entire project community. To the best of my knowledge the test plan document was not referenced or looked at by anyone after the initial review meetings. It provided very little utility to the test team afterwards. For fun I calculated the cost in terms of time to create the document. It came out to around $3,000.
One interesting thought is “Why did we put time into writing this document in the first place”.
Well it communicates the ideas behind our testing to the project team. Everyone “feels good” when the “testing” is written in a document. I’ve tried not writing a document and instead communicate our strategy verbally. This doesn’t seem good enough for the project team. It seems that there is some safety or comfort of having it one document. After this experience it feels more like a CYA (cover your ass) type of document than ever before.
Have you been in this type of situation on a software project? When have you been deceived by your own work? What approach did you try? How did you deal with it?
