Adam White

One is the one thing that needs to fall into place before testers can start operating the product? The build has to be successful!

One day I was looking at some statistics about our build process.

Project 1

Month fo Feb - 80% pass

Month of March  - 72% pass

As I looked at these stats I was having a hard time deciding what action to take because of them. I said outloud ”I  wish I could see these numbers in a visual way - like on a calander or something. We could put a big green check mark when the build passes and a big red X when it fails.

John Sterne, a student from Waterloo, who has done 4 or 5 work terms with us, decided to run with my “I wish” statement.

Much to my surprise I was able to get a better feel for the state of the build and it’s impact on my team by looking at the following graphics

80 % pass rate or 20% failure rate

72% pass rate or 27 % failure rate (6/22)

Which one communicates more powerfully to you?

If you like the pictures - then take a guess at how the testing for this product was going

Here is another example why.

Problem or no problem?

My laptop blue screen with a .sys file error. Upon rebooting an IE window was opened automatically by the system suggesting that the file could be corrupt or be from maleware. The window suggest I install Windows Live OneCare. I decided to do this.

OneCare told me that I had to defrag my drive. I found this curious so I ran the defrag tool that came with the system.

Question for you - problem or no problem? How do I go about sovling this inconsistency?

Recently I had lunch with a group of testers here in Toronto.  The topic of bug tracking, support ticket tracking and backend systems in general came up.  Each person at the the table had a story to tell about their bug tracking systems or their support ticket tracking systems. Most of the stories weren’t positive. From what I recall every person at the table felt that at least one (or more) of systems their company used didn’t support their work. They weren’t being as effecient and effective as they could be based on their experience with other tools

Why is it that companies do this?

The tools the people at the table were forced asked to use came from above in an attempt to “bring systems together” and to “have one central spot to access everything” of interest.

The general trend seemed to be that this goal was achieved. The information wasall in one spot but the tool did not help in achieving the goal of getting things done quickly and effeciently. One person at the table had a story about a particular test case database that had a front end that so horrible and time consuming to use that the testers wrote automation to populate test cases db from a flat file. Sound like a tool that supports effective and effecient testing doesn’t it?

What’s the solution?

Each person at the table had their own way of dealing with the problem.  Solutions ranged from “Well - it’s the system they told me to use so I guess I have to use it” to something like “Yeah - X is the corporate standard but we still use Y anyway. They don’t complain too much about it as long as we give them the metrics they want.”

How would you go about bringing this type of problem to light? This is a problem that involves making your team more productive and effecient. It might also involve changing the “coroporate standard”.

How would you/do you approach this type of thing?

So far based on what I have learned from my exploration and my emails with Tiho I have been able to create an unprioritized charter list.

• Test for max characters in text boxes
• Research share feedback functionality
• Research labels under review my feedback.
• Explore download CSV functionality
• Explore Expand all and collapse all
• Create a workflow diagram of all the dialogs and their functionality.
• Find a way to do some automatic data input and exercise the workflow.
• Explore error messages
• Test SSL
• Run through the install process for promoting code to production
• Try to migrate the system to a new server
• Learn more about the enterprise version of the product

Adam Goucher has a great idea for a blog post - Reward systems for testers. What metrics do you use to track a tester’s success?  I found myself commenting on it and formulating an idea I could post on my own . The methods I describe below can be used to reward any team. I use it with the testers that report to me as well the escalations developers I directly manage and teams that I don’t directly manage like the support team.

Using metrics to reward people

Using metrics to reward people can get you into the slippery slope of people gaming the system.  So one tester logged 10 bug and the other 5.  Who is the better tester? Need more info right?  Thought so.  4 bugs slipped out in one area and 1 in a different area.  Who is the better tester? What if it’s the same tester working on both areas? What if one of those areas was not tested because the dev team said it “there were no changes”?  Doesn’t seem fair does it? It doesn’t seem reasonable to me.

My approach

I find the best way to reward people is with appreciation right at the time they do something you like!

Here is how I go about this.

I wrote a list of expectations that I have of the team and a list of what they can expect from me as their leader. I reviewed this list with the team on multiple occasions. During the week I watch, listen and observe the teams interactions with their peers on the team and in other departments (development, support, product management, project management etc.) If they do something I like I note it down on a printed list of my expectations.

Sometimes I approach them immediately afterwards and tell them what I liked about the interaction and sometimes I wait until our weekly team meeting. Sometimes I read off the list things I observed that I liked and sometimes I don’t. I read the energy of the room and from myself and decide if it’s the best time to deliver appreciation.

I also randomly give out movie gift cards to go along with my appreciation as well. I also let teams testing a product leave early on Friday or come in late (depending on the circumstances).

When new projects come up I give people who have done good work and met my expectations a chance to run with the new stuff.

Flaws/Drawbacks with this approach

First problem is the random gift card approach. Some people tend to like a specific algorithm to receive a reward. “If I do X I receive Y”.  This can lead some people to feel left out because they aren’t receiving monetary recognition for their efforts while others are.

Second sometimes there is a really good interaction that I miss because I’m in a meeting or away from my teams area. Most of the time if it’s a really good interaction multiple people on the team will tell me about it. Sometimes the person involved will tell me directly. I also enjoy hearing these stories as well.

Side note: Wanting to see these interactions is one of the reasons I refuse to sit in an office that is away from my team. I would miss out on all the rich context!

Third - there are times when some people don’t receive appreciation for activities done throughout the week. Why? Well it could be because of the reason stated above, or any other of a multitude of reasons.

Despite the flaws I do it anyway

Why? Because I’ve seen the power of rewarding people with appreciation. On one instance I took an employee aside, looked him in the eyes said “Ron - I appreciate you for helping me run the team while I was away on vacation. It really meant a lot to me and people are telling me you did a great job. I like hearing that” Ron’s eyes started to tear up. I don’t think I could have given him a monetary reward that would have had the same impact as that simple sentence. Was it tough for me to do? No.  Did it cost me a lot? No did it have an impact that I wasn’t expecting? YES. Ron’s productivity went through the roof.

Rewards people takes effort

It’s up to me to know what motivates each person individually. I don’t feel there is a reward system you can put in place that blankets every tester on the team. For some people it’s money, for others it is intellectually stimulating work, for others it is camaraderie.

Having a blanket system or method doesn’t seem reasonable to me. It’s easy to do - anyone can pull metrics out of the bug tracking system to reward people. This method is not exactly fair and doesn’t take into consideration the nuances of things that are important to you

Keep in mind

The main thing to remember when rewarding your team is to be reasonable, pay attention to they are doing AND tell them when you notice it. Money helps but it doesn’t always work for everyone. My method is a lot of work but has paid off for me in the support and dedication of my team.

Let’s recap what’s been done so far:

• Discovered Rypple and started testing it out. 
• Applied theories from the Heuristic Test Strategy Model. Took notes on what I observed
• Asked stakeholders what is important to them using CRUSPIC STMPL as a guide.
• Went back and forth with the stakeholder.
• Started a charter list.

Here is how I could approach Rypple going forward with what I’ve learned so far

• Explore the project environment using HSTM as a guide
• Talk to Tiho face to face to get more insight 
• Develop a rich set of testing ideas using a variety of techniques. 
• Pull from my RST learning, my JItlearning and model the program using different visual tecniques - mind maps, white-boards etc.
• Put these ideas in a safe place. My blog seems like a good place so far

So have I provided value to Rypple? In response to my last post I got this from Rypple.

Read the rest…

I’m going to jump back to Rypple for a bit. From my last Rypple post I’ve got multiple jumping off points for a deeper discussion with Tiho. I’m sure there are things that I’ve missed that would be important to them. I’d love to sit down with him and brainstorm. I would go more into the capabilities with him and see what is important.

For reliability/robustness I would have to clarify with Tiho what this means to him. If I were to follow the heuristic test strategy model it guide me to ask if the product will work well and resist failure in all required situations. It goes deeper to talk about error messages and data integrity. I would explore the error messages in the program and ways for invoking them. If I had an architecture diagram I might be able to find ways to invoke these error messages individually and make my reliability testing more thorough. I could even make a mock up diagram based on what I know and ask him to fill it in. Then I could start asking what would happen if certain things failed or didn’t work properly.

I find it interesting that reliability and robustness got combined together. I take reliability to be the way a system performs over time. Robustness is resilience to failure and when failure does happen how the product handles it. A quick idea might explore doing some kind of denial of service attack on Rypple or corrupt the database in some way to see how it handles failure. Another idea would be to use perlclip to insert some really long strings and see if I can past their validation.

Usability is also important. For this I would have to focus on my opinion and present it. I can also poll the people I send message to through Rypple to see what they think of the usability of the program.

On the security side I’ve discovered something I didn’t know - they have an SSL site. There is also mention of an enterprise version. They take steps to prevent sql injection, cross site scripting, and use a state of the art facility to store the data. This is a gold mine for more test ideas. I also have to do more research into security exploits and ways to test SaaS apps for this.

Installability is not perceived to be a problem since they are SaaS. I found this intriguing as I’m not familiar with installation processes for SaaS. They will have to have some method/process of getting the code from dev to production.

I need more info on Testability. Tiho mentions that there are testing harness in place. This is encouraging. What would help me more effectively is to know what kind of logging they have for error messages. Are the logs primarily on the web server? Are there any other types of hooks/harnesses? What special tools do the developers use when debugging?

Maintainability in the test strategy model talks about how economical it is to build, fix and enhance the product. For me this ties closely into the support process and patching process. How easily can the support team, or at Rypple’s stage in the game probably the developers directly, diagnose and fix problems. What’s the turn around time? How readable is the code? If Rypple experiences a growth spurt will the developers coming in say “Wow - this is great code and it’s so easy to understand” or will they say “Wow - this is the worst code I’ve ever seen”.

For portability Tiho mentions that they picked a stack that would allow them to easily move to another hosting provider. I wonder what easy means to them and how “easy” it really is. I wonder how important this might be in the grand scheme of things. If relationships with the provider go sour then it will probably be a pretty big deal.

Localization. There isn’t much to explore since it’s not a focus right now. I think this was perfectly acceptable especially given the phase they are in. If I was a tester at Rypple it would be something I would be heavily encouraging the development team to integrate in a way that allows all the static text strings to be in separate files - especially error messages. This has the potential to save them lots of time and pain in terms of making the product support different languages. We went through this ordeal at PlateSpin and it required a significant re-work and re-test of the product.

At this point to help keep me focused I’m starting a spreadsheet of possible charters. Stay tuned for that.

I found a company called Idée inc.

 who makes really neat products for searching images. Their products are  cool because they don’t rely soley on tags to find images. Instead they use attribuates of the actual image. I must say it seems rather magical and mysterious to me.

Here is how one product, TinEye

, it works according to their website

“Every day TinEye’s spiders crawl the web for additional images. Using sophisticated pattern recognition algorithms, TinEye creates a unique and compact digital signature or ‘fingerprint’ for each one and adds it to the index.

When you want to find out where an image is being used on the web, you submit it to TinEye. The attributes of the image are analyzed instantly, and its fingerprint is compared to the fingerprint of every single image in the TinEye search index. The result? A detailed list of any websites using that image, worldwide”

They also have something called Idée Lab where you can try out their technologies yourself

http://labs.ideeinc.com/

I wondered a couple of things about this

1) Exactly what attributes are they looking at? What goes into the making of these picture signatures or “digital fingerprints”?

2) If I were a tester working for Idée how would I test their applications?

I will post some of my thoughts and notes.

To review the first session - I explored the product and took free-style notes while doing it. I came up with a list of ideas that i can explore further. I also came up with an issues list that I would like to speak to someone about.

For my second session I uncovered a lot of information but I didn’t operate their product. How can this be testing? What did I do?

I emailed one the folks at Rypple and asked what kind of information they would be interested in. I used the quality related information from James’ Heuristic Test Strategy Model. The quality heuristic can be remembered with the mnemonic CRUSPIC STMPL. What does that stand for?
It’s most of the ility type words that people talk about in the context of software. This mnemonic has become stuck in my head - when James and Michael present it they often tell stories that might help you remember it. They also encourage people to make up their own mnemonic as well.

CRUSPIC STMPL stands for:

Capability
Reliability
Usability
Security
Performance
Installability
Compatibility

Supportability
Testability
Maintainability
Portability
Localizability

I wasn’t expecting much of a response. I was pleasantly surprised. I got a response from Tiho who is listed on Linked In as Star Software Developer at Rypple. Here is his response.

Read the rest…