When to Start Logging Bugs
September 14, 2007 3:06 am TestingTesting starts as soon as we have a build to test. If it’s checked in - it gets tested even before the features are deemed “Feature Complete”
Now - how do we track the bugs found during this stage of development? Up until our most recent proejct we’ve always used the bug tracking system.
Now weve been asked to stop using the bug system because it adds too much overhead. The explanation given is “A developer is working on a feature and you log bugs on it when they know it isn’t working…then at the end they have to go back through the list and re-test everything you logged to make sure it’s fixed. It adds too much to their plate.”
My first reaction to this was one of fear and anger. Fear because our development culture appears to moving towards information hiding. To me, as a tester, not logging bugs is hiding information. One of the main goals of testing at PlateSpin is provide timely information to project stakeholders. Without having the bugs logged in a central place for all to see it becomes more difficult to get the information to everyone. Also, for most testers, not logging bugs is equivalent to developers not checking in their code for several days. I wonder what kind of reaction I would get if I asked development to stop checking code in after code freeze?
Why the sudden aversion from development to the bug system?
In the previous release we started testing a feature that was clearly not ready. By the time we were done we had accumulated about 200 bugs on a particular feature. This added a significant overhead for the developers responsible for that feature. The aversion is understandable to a degree. We made an “amateur” tester mistake. We used a tornado to knock over a straw house.
After being asked to not use the bug system I stated my case that I thought this was a bad idea and gave my reasons why. Other testers did the same. We decided to go the route the developers wanted and not use the bug tracking system. Instead we said we would send out a daily (running) list of issues.
During a pre-release status meeting somone said “It would be good if we could number these bugs so we could just refer to the numbers”.
“I know a system that we can use to do that - and it doesn’t require any effort to use.”
Someone else suggested that the testers could simply use Word instead of notepad to write the list of issues. This would give them access to the handy numbered list. (Notice who had to change their work)
Then someone suggested “It would be really nice if we could have these issues in one central place”
“I have just the system for that” - I replied.
Then it was requested testers put the bugs in a project tracking spreadsheet. This sheet had the exact same headings as the bug tracking system. Hmmm - very curious indeed.
This is where I drew the line. We told the developers that if they wanted the information in the spreadsheet that we didn’t mind. We just weren’t going to do it for them. If they wanted to enter it - they were more than welcome to.
How many issues do you think got copied into the spreadsheet?
We haven’t hit our feature complete date yet so I can’t say how this little experiment will work out. We’ve had some developers complain to us for not logging bugs on their code. When they ask for screenshots and logs we have to say we don’t have them becuase have no place to put them and reference them to a bug.
I’m really curious to see how things go with our experiment. Our backlog is already 100+ bugs. The day of feature complete is going to a fun bug logging day! 
I mentioned above that we gave reasons for using the bug system instead of a spreadsheets - here are some of them.
- Multiple people cannot edit the spreadsheet at one time. (Lets not tell them about shared workbooks in excel. It will be our secret - ok?)
- One must continually look the spreadsheet to see changes in priority etc
- No running log of notes and troublshooting steps. The bug database is often helpful when it comes time to write documentation and
- knowledgebase articles
- No automatic notification system when a bug status changes or when a new issue is opened.
- No way to add screenshots, diagnostics, or other attachments. Having some, or all, of these items can significantly speed up the amount of time it takes to fix an issue.
Got any other ones we can use?
Adam Goucher :
Date: September 14, 2007 @ 12:41 pm
“Testing starts as soon as we have a build to test.” - Okay, you know I disagree with this one. Testing should begin as soon as the developer starts working on the code. Are not their unit tests considered, erm, testing? How about when the CI build kicks off? The integration tests get run? The inspection tools run? Only *then* do you have a ‘build’ to test. But you know this sales pitch already.
As for logging bugs before the feature is complete, when I ran the shop at points this is what we did
- once the developer had ’something’ to test, we would poke it, but not log bugs.
- issues would be recorded in the wiki for future reference. we were there to assist the developer in identifying any gaps they missed
- once the feature was ‘complete’, we then we would unleash hell on the feature and log bugs
Now, this worked well because the apps were built in layers which helped facilitate this.
- first the gui was written, with the backend logic was stubbed out so we could check validation rules, etc
- after that the database stuff was readied so we could test the upload / integrity of it
- now the stubs were replaced with real logic
-adam
Rikard Edgren :
Date: September 24, 2007 @ 3:56 am
Some more possible reasons to use the bug system:
* It is easier to use, train and maintain one system instead of several.
* The testers will probably describe the issues better in a bug system, since it is more official. In a Wiki or spread sheet, it is easier to enter a oneliner that is difficult to understand for developers, or impossible to reproduce.
* If developers/project managers don’t use the unofficial list, the testers eventually needs to do the tracking for them.
* If metrics are interesting, it could be easier with one system, or more difficult (a new field for phase of testing might be needed.)
On the other hand, besides the overhead argument, there are communication and feelings that often points to not using the bug system.
I usually let each developer decide in which way to report and when to start testing for their functionality, until we reach Feature Complete.
/Rikard
Eric Jacobson :
Date: November 1, 2007 @ 2:31 pm
I think the real problem here is the insistence on testing a feature before the dev is finished writing it. Sure, it’s possible, fun, and even valuable. But when there are an infinite number of tests to execute, it may be more valuable to execute those tests in areas getting no attention from other testers. The other testers I speak of are the devs trying to complete their code on said feature. Yes, devs are testers too!
Your metaphor about “using a tornado to knock over a straw house” is great! I think as testers, we provide more value to the project by hitting the stone houses (i.e., code complete features) with tornados. This is where the skilled tester provides the most value.
“Follow-on bugs” is a term I think I’ve seen Michael Hunter use to describe bugs that occur downstream due to an upstream “blocking bug”. In many cases follow-on bugs get fixed indirectly when the upstream bug is fixed. I’m a firm believer in this heuristic, and it hopefully keeps my focus in the right place. Logging 200 bugs on a broken feature seems a little extreme…I guess that may be what you are saying.
If someone (who matters) on your project thinks certain chunks of bugs are time wasters, they are probably right. I agree with you that logging these bugs outside your bug tracking system is also a time waster. This is completely silly, and I have seen it as well. But if you focus on more valuable bugs, the whole notion of the extra dumbed-down bug tracking system is no longer necessary.