In the event of an unknown or unexpected system behavior in the software world - the test plan documents mean nothing.
When we develop our testing strategy we do two things - outline what we WILL test AND what we are NOT going to test. The latter is always one of the hardest to figure out, explain and do effectively. A close relationship with development is required. Trust from both sides is crucial. Expert knowledge of the system and how changes affect it is required. 95% of the time this process works but other times it backfires. It is usually the unknowns that screw us.
For project X a lead tester wrote a test plan document to satisfy project stakeholders. The document brought up lots of good questions and discussion. The one section that got a fair amount of discussion was the “Not going to test” section. We explicitly stated that PERFFORMANCE would NOT be TESTED. We would eyeball the performance (web based UI connected to our 4 year old core server technology) and not provide a formal analysis of request/response times. Why take this approach? We had done load testing with a shipping release build 1 month prior. We got benchmarks, provided this info to development and discussed the risks. Everything looked good. There was no reason for any of us to suspect performance problems. Everyone agreed “There should be no reason we would have performance problems”. All project stakeholders “signed off” on the test plan.
What do you think the single biggest problem found while testing?
Michael Hunter, of Braidy Tester fame, posted a challenge on his blog. He showed a parking sign photo and asked how many different ways it could be interpreted. I decided to take 45 minutes during one of our weekly test team meetings and try it out. What the hell does this have to do with testing you might ask? Well - thanks for asking. Here is how it relates for me
Idea generation - we always need to generate new ideas for testing
Clarification - requirements are mostly ambiguous and this was practice at clarifying them.
Sharing and discussion of ideas - “It could be interpreted this way” which lead to someone else saying “Yes and this way too” - Team work, supportive attitude with a focus on sharing.
Time boxing so we don’t get carried away - Important in testing and just about anything else
Coming up with alternate solutions/problem solving - It’s good to be seen as a problem solver instead of “Bearer of Bad News”
Needless to say we had a great time doing this activity. I time boxed the idea generation to 15 minutes and then had about 15 minutes of discussion. We used index cards to write down our ideas. We went around the room and each person read an idea from their card. We ran into some duplicate ideas but that was ok - we weren’t focused on having every idea be unique. The puprose was to put on our “thinking caps”
Now it’s easy to take an idea and tear it to shreds. To quote Edward De Bono in The Thinking Course “..critical destuction of one hypothesis has never produced a better one. It is creativity that produce the better hypothesis.” As a follow up challenge I asked the team to come with a sign that wasn’t ambigous. We explored some great ideas and had a great discussion. A few of the ideas and designs came out to be really clear and (mostly) unambiguous (I’ll have to get them and post it with this article.) The discussion was going so well that it actually went over time and into lunch.
Here are the results which were summarized by Aqiqul Hoda and Michael Hetmanczuk. The participants were Adam White, Alan Walker, Ali Khan, Aqiqul Hoda, Christy Gnanapragasam, Herb Bal, Joseph Kubik, Michael Hetmanczuk, Mortaza Abhari, Thomas Yook and Zhe Chen
1. Can park for 2 hrs from Mon to Sat between 7AM to 6PM.
2. If you have Zone 4 Permit you could park as long as you want.
3. No Limits of parking on Sundays and Holidays.
4. No Parking for Zone 4 permit vehicles.
5. No parking on Sun and holidays.
6. No parking in the Night.
7. Zone 4 permit vehicles parking only between 6 PM to 7 AM and Sun and
holidays.
8. 2 hr parking this side of street.
9. 2 hr parking both side of street.
10. 2 hr parking from this sign onward.
11. Parking at 6PM allowed can go past 6 PM for 2 hrs.
12. Parking once a day only.
13. Zone permits vehicles parking only after hrs but not on sun and
holidays.
14. Zone permit sign maynot be related to parking.
15. 2 hours limit parking between 7am and 6pm on days except Sunday and
Holiday. Above has exception by Zone 4 permit means Zone 4 permit could
park at anytime
16. Board number 2: hour parking
17. Is the #2 the sign ID or does it indicate 2 hours?
18. Can we park during other ours or only 7 AM - 6 PM
19. Where is the sign?
20. Does it mean I can park from 6 AM - 7 PM?
21. Hol? What qualifies as a holiday?
22. Does this apply to bikes?
23. Except sun/holiday means on Sunday and holidays - no parking at all
24. On sunday/holiday you can park all day
25. On sunday/holiday the 2 hour limit is lifted, but parking is only
allowed from 7 AM to 6 PM.
26. Zone 4 permit means you can park all the time.
27. Zone 4 permit means you can park 7 AM - 6 PM.
28. Except Zone 4 permit means you can’t park at all if you have a zone
4 permit.
29. Except Sun/Holiday: means only those with a zone 4 permit can park
on sun/holiday.
30. Could be interpreted as 1 hour parking. The #2 could be something
else, i.e. street number, parking spot.
31. Except sun-holiday: starts on sunday, ends on a holiday.
32. What is a zone 4 permit? Do I automatically get one?
33. Does that mean I can park long on sunday-holiday?
34. Maximum of 2 hour parking allowed between 7 AM and 6 PM except
sunday or holiday.
35. If you have zone 4 permit pass you can park the car anytime.
36. Within this zone, a maximum of 4 vehicles are permitted to park.
37. The #2 indicates it is a second sign that indicates “hour parking”
zone between 7 AM and 6 PM except on sundays, holidays.
I was playing around with a visual method of designing test - classification trees. I was bored of the same ole’ same ole’ you know. RazorCat is the only software that I know of that visually depicts classification trees. (although I haven’t looked that hard)
As I was signing up I got this error message. What would you do in this case?
There is no hardware id in the form - ahhhh - I’m going insane It’s almost like the error message that said “Press the any key”
Here is a bug that didn’t impact me very much but I can easily think of a situation when it might.
I was looking for job descriptions to use as an example for a task I was doing. My search of QA had returned 504 results which was fine….until I couldn’t see them all.
In this context that’s fine - but what if I was filtering resumes here. The last 4 candidates wouldn’t get seen. They might be my ideal candidates!!
As I was writing this post I decided to see if I could reproduce this. I couldn’t get it with the “QA” query as it appears that there are only 495 postings related to that.
I tried another generic term “Analyst” - it returned 1821 hits and I could only get to 1000 of these. I used the go to page function - page 40 is the final page and there is no next button so now i’m either missing 821 potential jobs or the stats being displayed are wrong. Take your pick
In retrospective - if I hadn’t been laid off in 2001 - I probably wouldn’t have become a software tester or if I did it would have been a very different path.
It’s pretty cool to be in a national newspaper. I’ve already had a linked in request and a friend from university, who resides in Bermuda, get in touch with me. I didn’t expect either of these things to happen. It’s all about the network!!!
I’ve got a problem I’m trying to solve and I need your input. We have a lab of 150 servers and are about to double the size. We need a tool to help with the scheduling of the servers to projects and people. We can’t seem to locate such a tool even though I’m sure we aren’t the first ones with this problem.
We haven’t been able to find any tool that solves this problem. Do you know of something handles this? Maybe there is an existing tool that is used for project management that could be adapted to this use? I just don’t know of any.
Perhaps my Google searches need refining “hardware resource scheduling”, “hardware tracking” turned up nothing of use.
Do you know of anyone who might know someone who has experience with solving this type of problem? If you do – I’d love to hear from you
My fiance gave me a 3rd generation ipod for christmas. Its pretty sweet. I like the coverflow feature. I spent waaaay too much time download covers and updating album names.
Today was the first day I took the ipod out for a run. One thing I love about the old is one hand navigation. I had no reason to believe this one would be different. But….it is.
Let’s do a little usability experiment. If you have a new ipod then hold it in your right hand and toggle the position of the lock.
Now switch it to you left hand and toggle the position of the lock.
Is there a difference? Yup.
Is it harder to do with your left hand than your right? Definitely
Does it make one handed operation of the ipod impossible? Yes - unless you employ both hands or do some fancy manipulating with your left hand.
You might have guessed that I’m left-handed. I can hear all the right handers saying in their sympathetic tone - suck it up.
Normally I do but this little quirk seems silly coming from a company that is supposed to be leading edge in design and usability.
From my vantage point this is a huge usability bug. I can no longer operate my nano with one hand.
I wonder if apple has any left handers on their design/usability/test team
To follow up my post on weak measurement being better than no measurement I decided to write some things I think can affect the ability of a development team to turn bugs around and hence make the use of the metrics from past releases not useful.
- Starting testing too early
- Starting testing too late
- Incorrect Risk Assessments by testers/developers/product managers
- Misunderstood Features by testers, developers, project team
- Constantly changing features (code churn/ greater risk of regression bugs)
- Inexperienced and/or Immature coders/testers
- Vacations/Sick days/Personal days/emergencies/snow days/Halo 3/WOW was released yesterday days
- Too many bugs carried over from last release.
- Unknown changes being checked in
- Misunderstood Features
- 3rd party applications cause problems
- Software licenses run out for development/test software.
- Busted Build process
- Product won’t install
- No business spec
- No Technical Spec
- Not enough hardware
- Incorrect hardware
- Not enough power to run the hardware
- Hardware/hardrives die unexpectedly (Does this ever happen expectedly?)
- Not enough People
- No Unit Tests
- Incorrect Time Estimates (oxymoron?)
- Miscommunication of vital pieces of information
- Features not cast/frozen/signed off on or otherwise stopped changing
- No requirements/Unclear requirements
- Bug system goes down.
- Internet connection goes down
- Power goes out
- Key stakeholders not available
- More time spent in meetings than fixing/testing software
- Customer problems take senior resources away.
- Meeting Interruptions/Coding interruptions/testing interruptions
Why do humans insist on predicting things that are inherently unpredictable?
We can’t predict the stock market. Why not? We have years and years of past data. Data is the key right? If data allows us to predict the future then it’s simple - build a model of the market, test it on past data, and get rich. Right? Hmmm - something is flawed in that argument. I’m thinking that it might be a little more complex than that. I know that people build their lives around trying to do this - my experience tells me that very few of them ever become rich.
Jump over to software. There are people who believe it is possible to predict the number of bugs in an upcoming release of software based on the number of in the past release. The logic goes like this “We have 5 years of data in the bug system- so we should be able to predict the number of upcoming bugs in the next release. We can even predict how quickly we will be able to fix those bugs based on the time to close the bug.”Does this argument sound familiar? Why is it that people believe this - but don’t believe they can predict the stock market? Or worse they believe this and believe that the stock market can be predicted.
In software I often get asked how long the bug fix cycle of the upcoming release is going to take. I actually have no idea how long it will take and I don’t want to spend my time figuring it out. There are too many variables and too many factors to take into account to make any prediction useful.
The person asking usually appeals to authority or my ego saying “You’ve been around here long enough - you should have an idea” My answer is always - it depends. “Oh yeah - on what?” is the usual response Well - It depends on the number of bugs we find and how long it takes to fix them. Ask development how long it will take them to fix the bugs we find.
“Well we have 5 years of data in the bug database. How long did it take to fix the bugs we’ve already addressed?”
My point is that it doesn’t matter how long it took in the past. What matters is how long it’s going to take to fix the ones in the future.
“Well we can just average out the fix times across releases. Remove the high times and as well those done really quickly so we have a good statistical average.”
Taking the average of fix times defeats the purpose. Why would you remove the ones that took a really long time? Those are the ones you should pay attention to. They also won’t help you in predicting the fix times of the future. Do you really believe that the past data on bugs will help you predict the bugs in the future?
“Well -In the absence of a good measure I’d rather use a weak (read: bad) measure than nothing at all.”
Never give a number to a bureaucrat. Such is the state of (some) software businesses.
Here is a tool that no tester should be without - BBTest Assistant made by blueberry software. I love this piece of software. It helps you show bugs that aren’t “reproducible”.
One time we logged a bug about performance of a certain part of our app. We were dismissed and told we were being impatient and too picky. We created a BBtest assistant movie of the perfomance showing the delay the user would go through. Guess what happened after that? Yup - the bug got fixed. The proof is in the video….
It’s almost like the error message that said “Press the any key”