In the event of an unknown or unexpected system behavior in the software world - the test plan documents mean nothing.
When we develop our testing strategy we do two things - outline what we WILL test AND what we are NOT going to test. The latter is always one of the hardest to figure out, explain and do effectively. A close relationship with development is required. Trust from both sides is crucial. Expert knowledge of the system and how changes affect it is required. 95% of the time this process works but other times it backfires. It is usually the unknowns that screw us.
For project X a lead tester wrote a test plan document to satisfy project stakeholders. The document brought up lots of good questions and discussion. The one section that got a fair amount of discussion was the “Not going to test” section. We explicitly stated that PERFFORMANCE would NOT be TESTED. We would eyeball the performance (web based UI connected to our 4 year old core server technology) and not provide a formal analysis of request/response times. Why take this approach? We had done load testing with a shipping release build 1 month prior. We got benchmarks, provided this info to development and discussed the risks. Everything looked good. There was no reason for any of us to suspect performance problems. Everyone agreed “There should be no reason we would have performance problems”. All project stakeholders “signed off” on the test plan.
What do you think the single biggest problem found while testing?
Michael Hunter, of Braidy Tester fame, posted a challenge on his blog. He showed a parking sign photo and asked how many different ways it could be interpreted. I decided to take 45 minutes during one of our weekly test team meetings and try it out. What the hell does this have to do with testing you might ask? Well - thanks for asking. Here is how it relates for me
Idea generation - we always need to generate new ideas for testing
Clarification - requirements are mostly ambiguous and this was practice at clarifying them.
Sharing and discussion of ideas - “It could be interpreted this way” which lead to someone else saying “Yes and this way too” - Team work, supportive attitude with a focus on sharing.
Time boxing so we don’t get carried away - Important in testing and just about anything else
Coming up with alternate solutions/problem solving - It’s good to be seen as a problem solver instead of “Bearer of Bad News”
Needless to say we had a great time doing this activity. I time boxed the idea generation to 15 minutes and then had about 15 minutes of discussion. We used index cards to write down our ideas. We went around the room and each person read an idea from their card. We ran into some duplicate ideas but that was ok - we weren’t focused on having every idea be unique. The puprose was to put on our “thinking caps”
Now it’s easy to take an idea and tear it to shreds. To quote Edward De Bono in The Thinking Course “..critical destuction of one hypothesis has never produced a better one. It is creativity that produce the better hypothesis.” As a follow up challenge I asked the team to come with a sign that wasn’t ambigous. We explored some great ideas and had a great discussion. A few of the ideas and designs came out to be really clear and (mostly) unambiguous (I’ll have to get them and post it with this article.) The discussion was going so well that it actually went over time and into lunch.
Here are the results which were summarized by Aqiqul Hoda and Michael Hetmanczuk. The participants were Adam White, Alan Walker, Ali Khan, Aqiqul Hoda, Christy Gnanapragasam, Herb Bal, Joseph Kubik, Michael Hetmanczuk, Mortaza Abhari, Thomas Yook and Zhe Chen
1. Can park for 2 hrs from Mon to Sat between 7AM to 6PM.
2. If you have Zone 4 Permit you could park as long as you want.
3. No Limits of parking on Sundays and Holidays.
4. No Parking for Zone 4 permit vehicles.
5. No parking on Sun and holidays.
6. No parking in the Night.
7. Zone 4 permit vehicles parking only between 6 PM to 7 AM and Sun and
holidays.
8. 2 hr parking this side of street.
9. 2 hr parking both side of street.
10. 2 hr parking from this sign onward.
11. Parking at 6PM allowed can go past 6 PM for 2 hrs.
12. Parking once a day only.
13. Zone permits vehicles parking only after hrs but not on sun and
holidays.
14. Zone permit sign maynot be related to parking.
15. 2 hours limit parking between 7am and 6pm on days except Sunday and
Holiday. Above has exception by Zone 4 permit means Zone 4 permit could
park at anytime
16. Board number 2: hour parking
17. Is the #2 the sign ID or does it indicate 2 hours?
18. Can we park during other ours or only 7 AM - 6 PM
19. Where is the sign?
20. Does it mean I can park from 6 AM - 7 PM?
21. Hol? What qualifies as a holiday?
22. Does this apply to bikes?
23. Except sun/holiday means on Sunday and holidays - no parking at all
24. On sunday/holiday you can park all day
25. On sunday/holiday the 2 hour limit is lifted, but parking is only
allowed from 7 AM to 6 PM.
26. Zone 4 permit means you can park all the time.
27. Zone 4 permit means you can park 7 AM - 6 PM.
28. Except Zone 4 permit means you can’t park at all if you have a zone
4 permit.
29. Except Sun/Holiday: means only those with a zone 4 permit can park
on sun/holiday.
30. Could be interpreted as 1 hour parking. The #2 could be something
else, i.e. street number, parking spot.
31. Except sun-holiday: starts on sunday, ends on a holiday.
32. What is a zone 4 permit? Do I automatically get one?
33. Does that mean I can park long on sunday-holiday?
34. Maximum of 2 hour parking allowed between 7 AM and 6 PM except
sunday or holiday.
35. If you have zone 4 permit pass you can park the car anytime.
36. Within this zone, a maximum of 4 vehicles are permitted to park.
37. The #2 indicates it is a second sign that indicates “hour parking”
zone between 7 AM and 6 PM except on sundays, holidays.
Here is a tool that no tester should be without - BBTest Assistant made by blueberry software. I love this piece of software. It helps you show bugs that aren’t “reproducible”.
One time we logged a bug about performance of a certain part of our app. We were dismissed and told we were being impatient and too picky. We created a BBtest assistant movie of the perfomance showing the delay the user would go through. Guess what happened after that? Yup - the bug got fixed. The proof is in the video….
Knowing what’s going on during a testing session is important. Some of the tools outlined in this video help you see the bugs sneaking around your computer.
I thought I would post some videos from the CAST 2007 testing conference. CAST stands for Conference of the Association of Software Testing. I was lucky enought to attend this past summer. At the conference there was a testing competition. The basic idea was we were given software and asked to test it. We had access to the developer and business owner (who happened to be the same person) as well as access to a bug tracking tool.
We could submit bugs using the tool and also gain extra points for doing a video presentation of some of the bugs as well as the techniques we used.
I wanted to toot my own horn here and say that we killed the competition. Nah - We pwned it!!!! My team (Team Canada) won first prize based on our bugs, techniques and test report. Unfortunately we couldn’t claim the $1500 cash prize because one person on our team was linked with the conference (in a very loose way). Thanks again Paul!!!!!
I wanted to post some of the videos that I was in during the competition. Here is the first one - talking about
