Adam White

Here is a tool that no tester should be without - BBTest Assistant made by blueberry software. I love this piece of software. It helps you show bugs that aren’t “reproducible”.

One time we logged a bug about performance of a certain part of our app. We were dismissed and told we were being impatient and too picky. We created a BBtest assistant movie of the perfomance showing the delay the user would go through. Guess what happened after that? Yup - the bug got fixed. The proof is in the video….

Here is my bb test assistant video from CAST

Knowing what’s going on during a testing session is important. Some of the tools outlined in this video help you see the bugs sneaking around your computer.

I thought I would post some videos from the CAST 2007 testing conference. CAST stands for Conference of the Association of Software Testing. I was lucky enought to attend this past summer. At the conference there was a testing competition. The basic idea was we were given software and asked to test it. We had access to the developer and business owner (who happened to be the same person) as well as access to a bug tracking tool.

We could submit bugs using the tool and also gain extra points for doing a video presentation of some of the bugs as well as the techniques we used.

I wanted to toot my own horn here and say that we killed the competition. Nah - We pwned it!!!! My team (Team Canada) won first prize based on our bugs, techniques and test report. Unfortunately we couldn’t claim the $1500 cash prize because one person on our team was linked with the conference (in a very loose way). Thanks again Paul!!!!!

I wanted to post some of the videos that I was in during the competition. Here is the first one - talking about

Direct link to video

There are more to follow - so stay tuned. I may even post the videos of the others on the team if they give me permission.

Here’s a problem I need help solving.

How do you load test an application when the application makes remote calls to Windows machines, uses WMI to collect and return information and stores the info in a central database?

We can fake it by calling each machine more than once but this isn’t exactly the same. I can tell you the other ways we’ve tried to solve the problem creatively but I’d rather not get into that right now.

I’m looking for access to 4000-5000+ machines. The machines can be physical or virtual. They can be windows or linux. I’d prefer more windows than linux. A user account that has admin type privileges is required so that we can read the data off the machines (although someone else can type in the password so we don’t have to know it). The system has to collect data for 1-2 months.

On a side note - there are other types of applications out there that do this type of stuff. Does anyone know how they load test?   

 If you know of anyone who might know someone that would have access to this many machines or perhaps work on a product like cirba, or microsoft systems center (formerly MOM), or VMware’s consolidation planning tool - I’d love to talk to them (even if we make a competing product )

I’d like to show you a bug in iTunes. I’ve created a video to show this. This is the first time I’ve used a video to describe a bug in software (outside of my company). We do it all the time inside our company. Let me know what you think of this.

ITunes Delete bug

Picture puzzles are something I used to love doing. Life magazine recently released an entire magazine devoted to picture puzzles (find the differences). Until I saw this magazine I thought these puzzles were only for kids. As I flipped through the magazine I noticed the puzzles were more difficult than I remembered from my childhood.

I did some looking at home and found Life has some examples online

 http://www.life.com/Life/picture_puzzle/

Along the way I found this bug 

Read the rest…

I like cooking. It’s one of my passions. I also like software testing. It too is a passion of mine. While cooking one night I noticed that I was using things that resembled heuristics I use in testing. I don’t usually use a recipe to cook - whatever is in the fridge will do. I just “go with it”. The results are usually pretty good. Some disasters have ensued - but not enough to make me follow a recipe. The creativity in doing this really gets me pumped. I experience some sort of flow or zone when putting ingredients together – taste a bit of this, add a bit of that, stir, taste again, repeat. While testing I experience similar feelings.

Read the rest…

Testing starts as soon as we have a build to test. If it’s checked in - it gets tested even before the features are deemed “Feature Complete”

Now - how do we track the bugs found during this stage of development? Up until our most recent proejct we’ve always used the bug tracking system.

Now weve been asked to stop using the bug system because it adds too much overhead. The explanation given is “A developer is working on a feature and you log bugs on it when they know it isn’t working…then at the end they have to go back through the list and re-test everything you logged to make sure it’s fixed. It adds too much to their plate.”

My first reaction to this was one of fear and anger. Fear because our development culture appears to moving towards information hiding. To me, as a tester, not logging bugs is hiding information. One of the main goals of testing at PlateSpin is provide timely information to project stakeholders. Without having the bugs logged in a central place for all to see it becomes more difficult to get the information to everyone. Also, for most testers, not logging bugs is equivalent to developers not checking in their code for several days. I wonder what kind of reaction I would get if I asked development to stop checking code in after code freeze?

Why the sudden aversion from development to the bug system?

In the previous release we started testing a feature that was clearly not ready. By the time we were done we had accumulated about 200 bugs on a particular feature. This added a significant overhead for the developers responsible for that feature. The aversion is understandable to a degree. We made an “amateur” tester mistake. We used a tornado to knock over a straw house.

After being asked to not use the bug system I stated my case that I thought this was a bad idea and gave my reasons why. Other testers did the same. We decided to go the route the developers wanted and not use the bug tracking system. Instead we said we would send out a daily (running) list of issues.

During a pre-release status meeting somone said “It would be good if we could number these bugs so we could just refer to the numbers”.

“I know a system that we can use to do that - and it doesn’t require any effort to use.”

Someone else suggested that the testers could simply use Word instead of notepad to write the list of issues. This would give them access to the handy numbered list. (Notice who had to change their work)

Then someone suggested “It would be really nice if we could have these issues in one central place”

“I have just the system for that” - I replied.

Then it was requested testers put the bugs in a project tracking spreadsheet. This sheet had the exact same headings as the bug tracking system. Hmmm - very curious indeed.

This is where I drew the line. We told the developers that if they wanted the information in the spreadsheet that we didn’t mind. We just weren’t going to do it for them. If they wanted to enter it - they were more than welcome to.

How many issues do you think got copied into the spreadsheet?

We haven’t hit our feature complete date yet so I can’t say how this little experiment will work out. We’ve had some developers complain to us for not logging bugs on their code. When they ask for screenshots and logs we have to say we don’t have them becuase have no place to put them and reference them to a bug.

I’m really curious to see how things go with our experiment. Our backlog is already 100+ bugs. The day of feature complete is going to a fun bug logging day!

I mentioned above that we gave reasons for using the bug system instead of a spreadsheets - here are some of them.

• Multiple people cannot edit the spreadsheet at one time. (Lets not tell them about shared workbooks in excel. It will be our secret - ok?)
• One must continually look the spreadsheet to see changes in priority etc
• No running log of notes and troublshooting steps. The bug database is often helpful when it comes time to write documentation and
• knowledgebase articles
• No automatic notification system when a bug status changes or when a new issue is opened. 
• No way to add screenshots, diagnostics, or other attachments. Having some, or all, of these items can significantly speed up the amount of time it takes to fix an issue.

Got any other ones we can use?