Michael Hunter, of Braidy Tester fame, posted a challenge on his blog. He showed a parking sign photo and asked how many different ways it could be interpreted. I decided to take 45 minutes during one of our weekly test team meetings and try it out. What the hell does this have to do with testing you might ask? Well - thanks for asking. Here is how it relates for me
Idea generation – we always need to generate new ideas for testing
Clarification – requirements are mostly ambiguous and this was practice at clarifying them.
Sharing and discussion of ideas – “It could be interpreted this way” which lead to someone else saying “Yes and this way too” - Team work, supportive attitude with a focus on sharing.
Time boxing so we don’t get carried away - Important in testing and just about anything else
Coming up with alternate solutions/problem solving – It’s good to be seen as a problem solver instead of “Bearer of Bad News”
Needless to say we had a great time doing this activity. I time boxed the idea generation to 15 minutes and then had about 15 minutes of discussion. We used index cards to write down our ideas. We went around the room and each person read an idea from their card. We ran into some duplicate ideas but that was ok – we weren’t focused on having every idea be unique. The puprose was to put on our “thinking caps”
Now it’s easy to take an idea and tear it to shreds. To quote Edward De Bono in The Thinking Course “..critical destuction of one hypothesis has never produced a better one. It is creativity that produce the better hypothesis.” As a follow up challenge I asked the team to come with a sign that wasn’t ambigous. We explored some great ideas and had a great discussion. A few of the ideas and designs came out to be really clear and (mostly) unambiguous (I’ll have to get them and post it with this article.) The discussion was going so well that it actually went over time and into lunch.
Here are the results which were summarized by Aqiqul Hoda and Michael Hetmanczuk. The participants were Adam White, Alan Walker, Ali Khan, Aqiqul Hoda, Christy Gnanapragasam, Herb Bal, Joseph Kubik, Michael Hetmanczuk, Mortaza Abhari, Thomas Yook and Zhe Chen
1. Can park for 2 hrs from Mon to Sat between 7AM to 6PM.
2. If you have Zone 4 Permit you could park as long as you want.
3. No Limits of parking on Sundays and Holidays.
4. No Parking for Zone 4 permit vehicles.
5. No parking on Sun and holidays.
6. No parking in the Night.
7. Zone 4 permit vehicles parking only between 6 PM to 7 AM and Sun and
holidays.
8. 2 hr parking this side of street.
9. 2 hr parking both side of street.
10. 2 hr parking from this sign onward.
11. Parking at 6PM allowed can go past 6 PM for 2 hrs.
12. Parking once a day only.
13. Zone permits vehicles parking only after hrs but not on sun and
holidays.
14. Zone permit sign maynot be related to parking.
15. 2 hours limit parking between 7am and 6pm on days except Sunday and
Holiday. Above has exception by Zone 4 permit means Zone 4 permit could
park at anytime
16. Board number 2: hour parking
17. Is the #2 the sign ID or does it indicate 2 hours?
18. Can we park during other ours or only 7 AM – 6 PM
19. Where is the sign?
20. Does it mean I can park from 6 AM – 7 PM?
21. Hol? What qualifies as a holiday?
22. Does this apply to bikes?
23. Except sun/holiday means on Sunday and holidays – no parking at all
24. On sunday/holiday you can park all day
25. On sunday/holiday the 2 hour limit is lifted, but parking is only
allowed from 7 AM to 6 PM.
26. Zone 4 permit means you can park all the time.
27. Zone 4 permit means you can park 7 AM – 6 PM.
28. Except Zone 4 permit means you can’t park at all if you have a zone
4 permit.
29. Except Sun/Holiday: means only those with a zone 4 permit can park
on sun/holiday.
30. Could be interpreted as 1 hour parking. The #2 could be something
else, i.e. street number, parking spot.
31. Except sun-holiday: starts on sunday, ends on a holiday.
32. What is a zone 4 permit? Do I automatically get one?
33. Does that mean I can park long on sunday-holiday?
34. Maximum of 2 hour parking allowed between 7 AM and 6 PM except
sunday or holiday.
35. If you have zone 4 permit pass you can park the car anytime.
36. Within this zone, a maximum of 4 vehicles are permitted to park.
37. The #2 indicates it is a second sign that indicates “hour parking”
zone between 7 AM and 6 PM except on sundays, holidays.
To follow up my post on weak measurement being better than no measurement I decided to write some things I think can affect the ability of a development team to turn bugs around and hence make the use of the metrics from past releases not useful.
- Starting testing too early
- Starting testing too late
- Incorrect Risk Assessments by testers/developers/product managers
- Misunderstood Features by testers, developers, project team
- Constantly changing features (code churn/ greater risk of regression bugs)
- Inexperienced and/or Immature coders/testers
- Vacations/Sick days/Personal days/emergencies/snow days/Halo 3/WOW was released yesterday days
- Too many bugs carried over from last release.
- Unknown changes being checked in
- Misunderstood Features
- 3rd party applications cause problems
- Software licenses run out for development/test software.
- Busted Build process
- Product won’t install
- No business spec
- No Technical Spec
- Not enough hardware
- Incorrect hardware
- Not enough power to run the hardware
- Hardware/hardrives die unexpectedly (Does this ever happen expectedly?)
- Not enough People
- No Unit Tests
- Incorrect Time Estimates (oxymoron?)
- Miscommunication of vital pieces of information
- Features not cast/frozen/signed off on or otherwise stopped changing
- No requirements/Unclear requirements
- Bug system goes down.
- Internet connection goes down
- Power goes out
- Key stakeholders not available
- More time spent in meetings than fixing/testing software
- Customer problems take senior resources away.
- Meeting Interruptions/Coding interruptions/testing interruptions
It’s been a while since my last post. I’ve got lots to write about and not enough time to do it.
The one thing I do want to mention is the upcoming CAST conference which will be held in Toronto this year. Some of the best minds in software testing will be at this conference. Speakers like Michael Bolton, Robert Sabourin, Jerry Weinberg, Cem Kaner. Not to mention I’ll be doing a presentation as well
Why do humans insist on predicting things that are inherently unpredictable?
We can’t predict the stock market. Why not? We have years and years of past data. Data is the key right? If data allows us to predict the future then it’s simple – build a model of the market, test it on past data, and get rich. Right? Hmmm – something is flawed in that argument. I’m thinking that it might be a little more complex than that. I know that people build their lives around trying to do this – my experience tells me that very few of them ever become rich.
Jump over to software. There are people who believe it is possible to predict the number of bugs in an upcoming release of software based on the number of in the past release. The logic goes like this “We have 5 years of data in the bug system- so we should be able to predict the number of upcoming bugs in the next release. We can even predict how quickly we will be able to fix those bugs based on the time to close the bug.”Does this argument sound familiar? Why is it that people believe this – but don’t believe they can predict the stock market? Or worse they believe this and believe that the stock market can be predicted.
In software I often get asked how long the bug fix cycle of the upcoming release is going to take. I actually have no idea how long it will take and I don’t want to spend my time figuring it out. There are too many variables and too many factors to take into account to make any prediction useful.
The person asking usually appeals to authority or my ego saying “You’ve been around here long enough – you should have an idea” My answer is always – it depends. “Oh yeah – on what?” is the usual response Well – It depends on the number of bugs we find and how long it takes to fix them. Ask development how long it will take them to fix the bugs we find.
“Well we have 5 years of data in the bug database. How long did it take to fix the bugs we’ve already addressed?”
My point is that it doesn’t matter how long it took in the past. What matters is how long it’s going to take to fix the ones in the future.
“Well we can just average out the fix times across releases. Remove the high times and as well those done really quickly so we have a good statistical average.”
Taking the average of fix times defeats the purpose. Why would you remove the ones that took a really long time? Those are the ones you should pay attention to. They also won’t help you in predicting the fix times of the future. Do you really believe that the past data on bugs will help you predict the bugs in the future?
“Well -In the absence of a good measure I’d rather use a weak (read: bad) measure than nothing at all.”
Never give a number to a bureaucrat. Such is the state of (some) software businesses.
Here is a tool that no tester should be without – BBTest Assistant made by blueberry software. I love this piece of software. It helps you show bugs that aren’t “reproducible”.
One time we logged a bug about performance of a certain part of our app. We were dismissed and told we were being impatient and too picky. We created a BBtest assistant movie of the perfomance showing the delay the user would go through. Guess what happened after that? Yup – the bug got fixed. The proof is in the video….
Knowing what’s going on during a testing session is important. Some of the tools outlined in this video help you see the bugs sneaking around your computer.
I thought I would post some videos from the CAST 2007 testing conference. CAST stands for Conference of the Association of Software Testing. I was lucky enought to attend this past summer. At the conference there was a testing competition. The basic idea was we were given software and asked to test it. We had access to the developer and business owner (who happened to be the same person) as well as access to a bug tracking tool.
We could submit bugs using the tool and also gain extra points for doing a video presentation of some of the bugs as well as the techniques we used.
I wanted to toot my own horn here and say that we killed the competition. Nah – We pwned it!!!! My team (Team Canada) won first prize based on our bugs, techniques and test report. Unfortunately we couldn’t claim the $1500 cash prize because one person on our team was linked with the conference (in a very loose way). Thanks again Paul!!!!!
I wanted to post some of the videos that I was in during the competition. Here is the first one – talking about
How do you load test an application when the application makes remote calls to Windows machines, uses WMI to collect and return information and stores the info in a central database?
We can fake it by calling each machine more than once but this isn’t exactly the same. I can tell you the other ways we’ve tried to solve the problem creatively but I’d rather not get into that right now.
I’m looking for access to 4000-5000+ machines. The machines can be physical or virtual. They can be windows or linux. I’d prefer more windows than linux. A user account that has admin type privileges is required so that we can read the data off the machines (although someone else can type in the password so we don’t have to know it). The system has to collect data for 1-2 months.
On a side note – there are other types of applications out there that do this type of stuff. Does anyone know how they load test?
If you know of anyone who might know someone that would have access to this many machines or perhaps work on a product like cirba, or microsoft systems center (formerly MOM), or VMware’s consolidation planning tool – I’d love to talk to them (even if we make a competing product )
I’d like to show you a bug in iTunes. I’ve created a video to show this. This is the first time I’ve used a video to describe a bug in software (outside of my company). We do it all the time inside our company. Let me know what you think of this.
Picture puzzles are something I used to love doing. Life magazine recently released an entire magazine devoted to picture puzzles (find the differences). Until I saw this magazine I thought these puzzles were only for kids. As I flipped through the magazine I noticed the puzzles were more difficult than I remembered from my childhood.
I did some looking at home and found Life has some examples online
