Adam White

Adam Goucher has a great idea for a blog post - Reward systems for testers. What metrics do you use to track a tester’s success?  I found myself commenting on it and formulating an idea I could post on my own . The methods I describe below can be used to reward any team. I use it with the testers that report to me as well the escalations developers I directly manage and teams that I don’t directly manage like the support team.

Using metrics to reward people

Using metrics to reward people can get you into the slippery slope of people gaming the system.  So one tester logged 10 bug and the other 5.  Who is the better tester? Need more info right?  Thought so.  4 bugs slipped out in one area and 1 in a different area.  Who is the better tester? What if it’s the same tester working on both areas? What if one of those areas was not tested because the dev team said it “there were no changes”?  Doesn’t seem fair does it? It doesn’t seem reasonable to me.

My approach

I find the best way to reward people is with appreciation right at the time they do something you like!

Here is how I go about this.

I wrote a list of expectations that I have of the team and a list of what they can expect from me as their leader. I reviewed this list with the team on multiple occasions. During the week I watch, listen and observe the teams interactions with their peers on the team and in other departments (development, support, product management, project management etc.) If they do something I like I note it down on a printed list of my expectations.

Sometimes I approach them immediately afterwards and tell them what I liked about the interaction and sometimes I wait until our weekly team meeting. Sometimes I read off the list things I observed that I liked and sometimes I don’t. I read the energy of the room and from myself and decide if it’s the best time to deliver appreciation.

I also randomly give out movie gift cards to go along with my appreciation as well. I also let teams testing a product leave early on Friday or come in late (depending on the circumstances).

When new projects come up I give people who have done good work and met my expectations a chance to run with the new stuff.

Flaws/Drawbacks with this approach

First problem is the random gift card approach. Some people tend to like a specific algorithm to receive a reward. “If I do X I receive Y”.  This can lead some people to feel left out because they aren’t receiving monetary recognition for their efforts while others are.

Second sometimes there is a really good interaction that I miss because I’m in a meeting or away from my teams area. Most of the time if it’s a really good interaction multiple people on the team will tell me about it. Sometimes the person involved will tell me directly. I also enjoy hearing these stories as well.

Side note: Wanting to see these interactions is one of the reasons I refuse to sit in an office that is away from my team. I would miss out on all the rich context!

Third – there are times when some people don’t receive appreciation for activities done throughout the week. Why? Well it could be because of the reason stated above, or any other of a multitude of reasons.

Despite the flaws I do it anyway

Why? Because I’ve seen the power of rewarding people with appreciation. On one instance I took an employee aside, looked him in the eyes said “Ron – I appreciate you for helping me run the team while I was away on vacation. It really meant a lot to me and people are telling me you did a great job. I like hearing that” Ron’s eyes started to tear up. I don’t think I could have given him a monetary reward that would have had the same impact as that simple sentence. Was it tough for me to do? No.  Did it cost me a lot? No did it have an impact that I wasn’t expecting? YES. Ron’s productivity went through the roof.

Rewards people takes effort

It’s up to me to know what motivates each person individually. I don’t feel there is a reward system you can put in place that blankets every tester on the team. For some people it’s money, for others it is intellectually stimulating work, for others it is camaraderie.

Having a blanket system or method doesn’t seem reasonable to me. It’s easy to do – anyone can pull metrics out of the bug tracking system to reward people. This method is not exactly fair and doesn’t take into consideration the nuances of things that are important to you

Keep in mind

The main thing to remember when rewarding your team is to be reasonable, pay attention to they are doing AND tell them when you notice it. Money helps but it doesn’t always work for everyone. My method is a lot of work but has paid off for me in the support and dedication of my team.

You’ll notice in my email exchanges with Rypple I’m digging out more project related information without even trying.  The heuristic test strategy model for the project environment has another mnemonic – CIDTESTD.

An easy way to remember this is to remember the phrase “Kid tested – mother approved.” or in this case “CIDTESTD – mother approved”

It stands for

• Customers
• Information
• Developer Relations
• Test Team
• Environment and Tools
• Schedule
• Test Items
• Deliverables

So what are some of the nuggest I have discovered about the project from my interactions with Rypple?

Customers

• I’ve discovered the product is in a private beta that you sign up for and someone at Rypple has to manually approve. This was a point of contention for one person who left a comment on a blog. Maybe it’s not clear enough to them that it’s a private beta?
• I have figured out that the target audience is people who are in the position to give and receive feedback. There seems to be a lot of talk about Gen-Y and gap between them and “older generation”

Information

• I’ve learned that they use an Agile process of some sort but subscribe to a particular flavor of scrum/agile implementation
• I will need to dig in deeper into the history of the product. I’d also look into any 3rd party software they are using. I’d review the bug lists for those products and see if any of the issues can affect the product.

Developer Relations

• It appears as though most developers are friendly and open to feedback if my interactions with Tiho are any indication. I’ve been able to get good, quick feedback from him consistently.
• Security is an area where I pick up on something important

Test Team

• I don’t know anyone in particular on the test team right now but I do know they have test harnesses.
• I am starting to realize that I don’t have any expertise in web security testing. There are a couple of approaches to fixing that 1) Read up and practise testing for security related issues 2) Call on people I know to help guide me 3) Hire someone else to do the security testing.

Equipment and Tools

• It sounds like they can set up a development environment right on their own desktop boxes.
• They have an offsite hosting facility.
• I’m creating matrices and checklists for myself as I go
• I really need to get information on how the developers debug problems and the tools they use.
• They support all major browsers and versions
• They “have plenty more to come when it comes to integrating with other online services”. Got some stuff find out there.

Schedule

• My arbitrary schedule is to be done my test plan creation, risk and coverage matrices and charter list and top 10 issues by Tuesday March 24th, 2009.
• There is a test harness that helps with the regression testing of the weekly code drop

Test Items

• For the scope of test items I’m working that out with Rypple as I work through this process.
• The product is available.
• I am unsure how volatile it is.
• I don’t have any access to the internal workings to know what has a been added lately.
• I can see from the public blog that they just added a connector to facebook.
• The product appears to be reliable enough for me to effectively test it.
• They are doing weekly drops of the code so volatility doesn’t seem like that much of a concern. I guess it depends on whether I’m testing in the context of scrum or not.

Deliverables

• I’m making written reports of my whole process as I go. If I were working for Rypple I would probably not make all this information available in written format. Most of it would be verbal, brainstorming, modeling etc.
• If I worked for Rypple I would see that part of my deliverables were given to other teams – mainly the list top 10 issues a critic might say needs to be given to support and services.

Discovering these things about the project environment and then probing them more might lead me to better information and charters later on.

Let’s recap what’s been done so far:

• Discovered Rypple and started testing it out. 
• Applied theories from the Heuristic Test Strategy Model. Took notes on what I observed
• Asked stakeholders what is important to them using CRUSPIC STMPL as a guide.
• Went back and forth with the stakeholder.
• Started a charter list.

Here is how I could approach Rypple going forward with what I’ve learned so far

• Explore the project environment using HSTM as a guide
• Talk to Tiho face to face to get more insight 
• Develop a rich set of testing ideas using a variety of techniques. 
• Pull from my RST learning, my JItlearning and model the program using different visual tecniques – mind maps, white-boards etc.
• Put these ideas in a safe place. My blog seems like a good place so far

So have I provided value to Rypple? In response to my last post I got this from Rypple.

(more…)

I’m going to jump back to Rypple for a bit. From my last Rypple post I’ve got multiple jumping off points for a deeper discussion with Tiho. I’m sure there are things that I’ve missed that would be important to them. I’d love to sit down with him and brainstorm. I would go more into the capabilities with him and see what is important.

For reliability/robustness I would have to clarify with Tiho what this means to him. If I were to follow the heuristic test strategy model it guide me to ask if the product will work well and resist failure in all required situations. It goes deeper to talk about error messages and data integrity. I would explore the error messages in the program and ways for invoking them. If I had an architecture diagram I might be able to find ways to invoke these error messages individually and make my reliability testing more thorough. I could even make a mock up diagram based on what I know and ask him to fill it in. Then I could start asking what would happen if certain things failed or didn’t work properly.

I find it interesting that reliability and robustness got combined together. I take reliability to be the way a system performs over time. Robustness is resilience to failure and when failure does happen how the product handles it. A quick idea might explore doing some kind of denial of service attack on Rypple or corrupt the database in some way to see how it handles failure. Another idea would be to use perlclip to insert some really long strings and see if I can past their validation.

Usability is also important. For this I would have to focus on my opinion and present it. I can also poll the people I send message to through Rypple to see what they think of the usability of the program.

On the security side I’ve discovered something I didn’t know – they have an SSL site. There is also mention of an enterprise version. They take steps to prevent sql injection, cross site scripting, and use a state of the art facility to store the data. This is a gold mine for more test ideas. I also have to do more research into security exploits and ways to test SaaS apps for this.

Installability is not perceived to be a problem since they are SaaS. I found this intriguing as I’m not familiar with installation processes for SaaS. They will have to have some method/process of getting the code from dev to production.

I need more info on Testability. Tiho mentions that there are testing harness in place. This is encouraging. What would help me more effectively is to know what kind of logging they have for error messages. Are the logs primarily on the web server? Are there any other types of hooks/harnesses? What special tools do the developers use when debugging?

Maintainability in the test strategy model talks about how economical it is to build, fix and enhance the product. For me this ties closely into the support process and patching process. How easily can the support team, or at Rypple’s stage in the game probably the developers directly, diagnose and fix problems. What’s the turn around time? How readable is the code? If Rypple experiences a growth spurt will the developers coming in say “Wow – this is great code and it’s so easy to understand” or will they say “Wow – this is the worst code I’ve ever seen”.

For portability Tiho mentions that they picked a stack that would allow them to easily move to another hosting provider. I wonder what easy means to them and how “easy” it really is. I wonder how important this might be in the grand scheme of things. If relationships with the provider go sour then it will probably be a pretty big deal.

Localization. There isn’t much to explore since it’s not a focus right now. I think this was perfectly acceptable especially given the phase they are in. If I was a tester at Rypple it would be something I would be heavily encouraging the development team to integrate in a way that allows all the static text strings to be in separate files – especially error messages. This has the potential to save them lots of time and pain in terms of making the product support different languages. We went through this ordeal at PlateSpin and it required a significant re-work and re-test of the product.

At this point to help keep me focused I’m starting a spreadsheet of possible charters. Stay tuned for that.

I found a company called Idée inc.

 who makes really neat products for searching images. Their products are  cool because they don’t rely soley on tags to find images. Instead they use attribuates of the actual image. I must say it seems rather magical and mysterious to me.

Here is how one product, TinEye

, it works according to their website

“Every day TinEye’s spiders crawl the web for additional images. Using sophisticated pattern recognition algorithms, TinEye creates a unique and compact digital signature or ‘fingerprint’ for each one and adds it to the index.

When you want to find out where an image is being used on the web, you submit it to TinEye. The attributes of the image are analyzed instantly, and its fingerprint is compared to the fingerprint of every single image in the TinEye search index. The result? A detailed list of any websites using that image, worldwide”

They also have something called Idée Lab where you can try out their technologies yourself

http://labs.ideeinc.com/

I wondered a couple of things about this

1) Exactly what attributes are they looking at? What goes into the making of these picture signatures or “digital fingerprints”?

2) If I were a tester working for Idée how would I test their applications?

I will post some of my thoughts and notes.

To review the first session – I explored the product and took free-style notes while doing it. I came up with a list of ideas that i can explore further. I also came up with an issues list that I would like to speak to someone about.

For my second session I uncovered a lot of information but I didn’t operate their product. How can this be testing? What did I do?

I emailed one the folks at Rypple and asked what kind of information they would be interested in. I used the quality related information from James’ Heuristic Test Strategy Model. The quality heuristic can be remembered with the mnemonic CRUSPIC STMPL. What does that stand for?
It’s most of the ility type words that people talk about in the context of software. This mnemonic has become stuck in my head – when James and Michael present it they often tell stories that might help you remember it. They also encourage people to make up their own mnemonic as well.

CRUSPIC STMPL stands for:

Capability
Reliability
Usability
Security
Performance
Installability
Compatibility

Supportability
Testability
Maintainability
Portability
Localizability

I wasn’t expecting much of a response. I was pleasantly surprised. I got a response from Tiho who is listed on Linked In as Star Software Developer at Rypple. Here is his response.

(more…)

I have to give hats off to John Creson – a tester at Autodesk. When he arrived at the testers lunch today he brought a person from his support team! WHAT!!! A support person hanging out with a bunch of testers?? Can’t be. Really?

It’s true – it’s a real life story of teams working together. My last post because was too close to lunch for him to gaming the system

Hats of to John for expanding our testing circle to include people from his support team.

Way to go John!

I’m about to let the cat out of the bag. Some of you know it already – some of you don’t. 

If you really care about quality work on improving people’s relationship to your software. Don’t stop there – take it to the next level. Work on improving how the people who work on your software interact with each other.

Get support and test working together – really working together. Have people switch roles for a couple of days – walk a mile in the other guys shoes. Have them shadow each other. Get your testers and developers together (if they aren’t already). And GAASSP.

Get your developers and your support reps working together.

Don’t just get them exchanging emails. That’s not going to help. Sitting in on meetings together doesn’t do much either. Have developers start taking support calls. Writing KB articles, reviewing calls with support, dealing with irate customers.

I bet you might have a team of services guys out “in the real world”, flying around to customers, helping them out, fixing issues, working with support. Why not hook them up with the testers and developers too? Don’t tell me they don’t have time to help improve Quality.

Like magic better people interactions make quality problems disappear

If these teams really work together I can promise you something magical will start to happen. Your quality problems will start to disappear. Magically. Poof!! Gone.

Why? Because you have increased the value you provide to your customers. Quality is NOT an attribute of software. You can’t touch it, you can’ count it. Quality is a relationship between the user and the software. So keep working on it.

Here’s the thing. If your customers find a bug and you deal with it in a timely and appropriately manner – your customers will more than likely not be upset. If you have a skilled support person they will take ownership and let the customer know their issue is being looked into. They can follow up with the customer – even if you don’t have a fix yet. That’s crazy talk man!! Make the customer feel like their issue is important and follow up even if we don’t have fix!! Pffft.

Does this mean that customers won’t get mad? No.

Does this mean your support guys won’t want to hang up on rude customers? No.

But it does mean your customers might be happier when it comes to renew their support and services contract because they got value out of it. They aren’t just a number anymore – the are a real, living human being that someone at your company cares about.

The metrics game

Once you start you are going to run into a problem   (more…)

So here’s the thing. I’m on vacation for a month. My boss told me to disconnect from work completely – no emails, no blackberry, no nothing. So I did. I’ve been trying not chat with team members on msn. etc, etc, etc. And I’ve been successful at that (for the most part).

I failed at one thing though. I can’t disconnect from testing. It’s in my blood. It’s like a drug. I crave it. I get a rush when I’m learning, when I get in that tester mindset, and I’m interact with intellectually stimulating people.  People who think differently and don’t follow the herd.  Those who say wait a minute let’s look at what’s going on here. Testing is so much a part of what I do and what I’m so passionate about.

Ok Boss I can disconnect from day to day project demands and release schedules – Fine. But disconnect from testing? Forget it!! I can’t. Even with our new house I’m being a tester all the time if you take Weinberg’s definition “A tester is someone who knows things can be different”. House ownership has definitely taught me that things can be different.

Here are some examples of how I can’t get away from testing and i’m learning how things can be different.

I spent most of yesterday reading “The Gift of Time”. Then I spent some time thinking about James Bach’s

recent post on Quality is Dead. Do I agree with him or not? At first yes – but then no.  Still deciding. I can argue it either way. Maybe there is a blog post there for me.

Last night I had dinner with Rob Sabourin who is in town teaching his course for a client. If you ever need someone to get you pumped about testing just give Rob a shout! Having dinner with Rob is just so damn interesting. I know the conversation is interesting and valuable when I’ve started taking notes on the table. It’s even more interesting when I tear that paper off and bring it home with me for later reference.  Who else do you know of that can link software testing to Dr. Seuss. Who else can stand up on a stage and do a presentation called “What baseball taught me about software testing” or “What the the looney tunes taught me about software testing”. Rob can! He inspired me to write about Curious George as a tester. I got rejected by a magazine so I gave up on it.  I’m going to dust that piece off and put it on my blog.

Last year I was telling Rob about the link between my training at second city improv and testing. He told me I should write about it – which I did. I wrote up some thoughts but my inner critic got in my way and I didn’t put any more effort into it. I think it’s time to kick that critic in the ass – thanks Rob!

The energy Rob brings to the testing community is great. I’ve had the priviledge of attending his JIT (Just in Time testing) course as an auditor and also to bring him in to my company to do work on visual modeling. Great stuff. If you want to give your testers a kick of energy – have them do visual modeling with Rob. I promise they won’t look at things the same way afterwards.  My team definitely did not.

I found Michele Smith’s blog  from a link on James’ site. I liked what I saw so I read her recent posts. I found a link to pre-order Secrets of a Buccaneer Scholar, James’ new book from Amazon.  So I put that in my basket. Thanks Michele!

From her site I also found Michael Bolton ‘s article/presentation called “T he Two Futures Of Software Testing“.  I spent the whole morning devouring it, thinking about what he is saying.  I know Michael is at Novell this week at their Quality Summit 2009 and teaching RST afterwards. I hope it’s going well for him. 

I’ve been dusting off my testing skills with the help of Rypple and TinEye.

I’m having lunch tomorrow with a bunch of testers in Toronto. I love these conversations!!!

I’m so pumped about testing or should I say knowing things can be different. I often get this way when I’m at conferences where I can think about testing without day to day project demands.  I’ve got the better part of a whole month left to work on my testing skills! I feel like I am in a very fortunate place

The purpose of this entire exercise is to show how I would apply the heuristic test strategy model developed by James Bach

What follows in the post is the notes from my first testing session intended to show show my thought process during my exploration.

*Warning – I’m stepping out on a limb here. I’m putting my work out there for review. Keep in mind they are my notes and may not make sense to you. They aren’t edited for spelling mistakes, grammar or anything else. The notes were taking as a stream of consciousness while I was testing.

If you want to know more about session based testing check out the this write up

To accompany my notes I have a 60 minute video taken with BB Test assistant. I also have a few screenshots that I took. I will try to find a way to post these so that i’m mindful of bandwidth constraints.

# Charter: Explore Rypple functionality

# Tester: Adam K White

# Length: Normal

# Charter vs opportunity
100% charter

# Set-up
0

# Notes
Clicked link in email for account activation. Typed in a 52 character password. After putting in my password again to verify I was asked to invite contacts I was prompted with a page that asked me to watch a video to get started. It showed import contacts on this page as well. I would think import contacts should have been on the invite page for those who are familar with the functionality.

The main functionality seeems to be
- Get feedback
- Review feedback
- Give Feedback
- My Network

In the top frame we have
- Welcome, Adam
- Help
- Give Us feedback
- Logout

I will explore the functionalities within the get feedback area.

Get feedback has 3 sections

Section 1
What is your question?

Max Characters appear to be 140. The training video said 100. This is inconsistent. I would imagine they found that 100 characters is not enough. The text box that appears to allow 2 lines of text to accomodate 140 characters. I would assume 70 characters per line. Will test later.

There is an icon of lightbulb below the “What is your question text box” with a + sign at the bottom right of it. A mouse over shows a tool top of suggested questions. Click the button expands a list of canned questions.

Section 2
What is your question about?

Text box has the following text
<Select up to 3 attibutes, seperated by a comma. E.g. leadership, negotiating, communication>

There is a similar light bulb icon which upon clicking reveals key words such as
leadership, integrity, presentation, attitude, communication, more attributes

Clicking more attibutes reveals
delegation, vision, strategy, listening, reliability less attributes

Noticed the the L in Less attibutes in capitilized. Problem? Don’t remember if the m in more attributes is capitalized. Clicking less attributes confirms that it is. Should it be? Something says to me that is should be. It’s consistent within itself but it violates something else. Will think about it.

No character max displayed
Section 3

How do you want to ask?

Two radio buttons
- Ask advisers through email
- Get a unique URL

There are two blue icons that seem to indicate information.

Just noticed them in the other sections as well.

Clicking gives a tooltip/fly out of information.

Text box is bigger than previous two. Appears to be 3 lines. No character max displayed

Can createa group from these advisors as well. Interesting functionality. I wonder how many people can be in a group of advisers.

—–

At the bottom of the 3 main sections there is a personalize your request, preview and create feedback request.
Personalize your request
- Shows a 3 line text box. Max appears to be 440 characters

Below the text box there is an arrow that reminds me of the undo arro in word or something similar to a back button in a browser. Mouse over shows a fly out saying “Reset personal message”. Seems to fit my expectations.

Preview
- Click preview says that need to enter at least one attribute. Thought this was strange. I expected it to ask me to enter a question first. Scrolling up I found that I had a question in there.

Removed question and clicked preview. Program prompted me to type a question. That meets my expecations.
Going to sumbit a question now.

Typed question “How would you test this program?”

Attributes “testing, listening, delegation”

I wonder if there is character max on the attributes – test for later

Unchecked rate these attributes

Started typing the emails of people i added earlier during invite session. They show up in a drop down while typing. Meets my expecations. If they didn’t show up I’d be disappointed.

Typed a personal message “I’d like your help in exploring this program. I’m testing it out for fun – not for profit. ”

Click Preview Shows a dialog. First line is a red circle with a white exclamation point in it. Text to the right of it says “Heads up. You’re one of two people being asked”

That’s odd – since I didn’t typ in my own email. Investigate later

Sumbitted my question. Dialog shows submitting and then success I think. Don’t rememeber and wasn’t taking a video.

Final page shows the date, my question, advisers asked, attributes, labels.

Not sure what this label business is about – investigate later.

Drop down box with “select action” is visible. text in box says add advisors, close request, send reminder.

There is a button called share feedback – will investigate later

There is a page divider and below it shows

No responses so far with links Download CSV, Expand All, Collapse all. Then button with Share feedback and back to listing.

These buttons seem to be duplicates of the ones from the top of the frame. Might make more sense once I’ve asked more questions and gotten feedback.

Going to stop this session now, clean up my notes and organize my findings

While typing up notes I noticed that I’m now in the review my feedback section.
# Issues/Things to follow up on
- Not being given the option to import contact right away after activating. I might have missed it. I have no way to reset my account to “new”. As a tester this would a useful tool to have. Often times I want to set my account back to a known state. Unfortunetly this doesn’t always work properly either.

Click Preview shows red circle with a white exclamation point in it. Text to the right of it says “Heads up. You’re one of two people being asked”. I wasn’t one of the two people being asked


# Test ideas
- Test for max characters in text boxes
- Research share feedback functionality
- Research labels under review my feedback.
- Explore download CSV functionality
- Explore Expand all and collapse all
- Create a workflow diagram of all the dialogs and their functionality.
- Find a way to do some automatic data input and exercise the workflow.

From my test ideas I have a set of new charters to explore. It took me roughly 60 minutes to get this basic list of ideas. From there there are a few different directions I can take. I can discuss the issues I found and I could prioritize my test ideas with them. Before I run off and operate the product more I can also jump off on a different thread. Stay tuned….