Adam White

So far based on what I have learned from my exploration and my emails with Tiho I have been able to create an unprioritized charter list.

• Test for max characters in text boxes
• Research share feedback functionality
• Research labels under review my feedback.
• Explore download CSV functionality
• Explore Expand all and collapse all
• Create a workflow diagram of all the dialogs and their functionality.
• Find a way to do some automatic data input and exercise the workflow.
• Explore error messages
• Test SSL
• Run through the install process for promoting code to production
• Try to migrate the system to a new server
• Learn more about the enterprise version of the product

Adam Goucher has a great idea for a blog post - Reward systems for testers. What metrics do you use to track a tester’s success?  I found myself commenting on it and formulating an idea I could post on my own . The methods I describe below can be used to reward any team. I use it with the testers that report to me as well the escalations developers I directly manage and teams that I don’t directly manage like the support team.

Using metrics to reward people

Using metrics to reward people can get you into the slippery slope of people gaming the system.  So one tester logged 10 bug and the other 5.  Who is the better tester? Need more info right?  Thought so.  4 bugs slipped out in one area and 1 in a different area.  Who is the better tester? What if it’s the same tester working on both areas? What if one of those areas was not tested because the dev team said it “there were no changes”?  Doesn’t seem fair does it? It doesn’t seem reasonable to me.

My approach

I find the best way to reward people is with appreciation right at the time they do something you like!

Here is how I go about this.

I wrote a list of expectations that I have of the team and a list of what they can expect from me as their leader. I reviewed this list with the team on multiple occasions. During the week I watch, listen and observe the teams interactions with their peers on the team and in other departments (development, support, product management, project management etc.) If they do something I like I note it down on a printed list of my expectations.

Sometimes I approach them immediately afterwards and tell them what I liked about the interaction and sometimes I wait until our weekly team meeting. Sometimes I read off the list things I observed that I liked and sometimes I don’t. I read the energy of the room and from myself and decide if it’s the best time to deliver appreciation.

I also randomly give out movie gift cards to go along with my appreciation as well. I also let teams testing a product leave early on Friday or come in late (depending on the circumstances).

When new projects come up I give people who have done good work and met my expectations a chance to run with the new stuff.

Flaws/Drawbacks with this approach

First problem is the random gift card approach. Some people tend to like a specific algorithm to receive a reward. “If I do X I receive Y”.  This can lead some people to feel left out because they aren’t receiving monetary recognition for their efforts while others are.

Second sometimes there is a really good interaction that I miss because I’m in a meeting or away from my teams area. Most of the time if it’s a really good interaction multiple people on the team will tell me about it. Sometimes the person involved will tell me directly. I also enjoy hearing these stories as well.

Side note: Wanting to see these interactions is one of the reasons I refuse to sit in an office that is away from my team. I would miss out on all the rich context!

Third – there are times when some people don’t receive appreciation for activities done throughout the week. Why? Well it could be because of the reason stated above, or any other of a multitude of reasons.

Despite the flaws I do it anyway

Why? Because I’ve seen the power of rewarding people with appreciation. On one instance I took an employee aside, looked him in the eyes said “Ron – I appreciate you for helping me run the team while I was away on vacation. It really meant a lot to me and people are telling me you did a great job. I like hearing that” Ron’s eyes started to tear up. I don’t think I could have given him a monetary reward that would have had the same impact as that simple sentence. Was it tough for me to do? No.  Did it cost me a lot? No did it have an impact that I wasn’t expecting? YES. Ron’s productivity went through the roof.

Rewards people takes effort

It’s up to me to know what motivates each person individually. I don’t feel there is a reward system you can put in place that blankets every tester on the team. For some people it’s money, for others it is intellectually stimulating work, for others it is camaraderie.

Having a blanket system or method doesn’t seem reasonable to me. It’s easy to do – anyone can pull metrics out of the bug tracking system to reward people. This method is not exactly fair and doesn’t take into consideration the nuances of things that are important to you

Keep in mind

The main thing to remember when rewarding your team is to be reasonable, pay attention to they are doing AND tell them when you notice it. Money helps but it doesn’t always work for everyone. My method is a lot of work but has paid off for me in the support and dedication of my team.

You’ll notice in my email exchanges with Rypple I’m digging out more project related information without even trying.  The heuristic test strategy model for the project environment has another mnemonic – CIDTESTD.

An easy way to remember this is to remember the phrase “Kid tested – mother approved.” or in this case “CIDTESTD – mother approved”

It stands for

• Customers
• Information
• Developer Relations
• Test Team
• Environment and Tools
• Schedule
• Test Items
• Deliverables

So what are some of the nuggest I have discovered about the project from my interactions with Rypple?

Customers

• I’ve discovered the product is in a private beta that you sign up for and someone at Rypple has to manually approve. This was a point of contention for one person who left a comment on a blog. Maybe it’s not clear enough to them that it’s a private beta?
• I have figured out that the target audience is people who are in the position to give and receive feedback. There seems to be a lot of talk about Gen-Y and gap between them and “older generation”

Information

• I’ve learned that they use an Agile process of some sort but subscribe to a particular flavor of scrum/agile implementation
• I will need to dig in deeper into the history of the product. I’d also look into any 3rd party software they are using. I’d review the bug lists for those products and see if any of the issues can affect the product.

Developer Relations

• It appears as though most developers are friendly and open to feedback if my interactions with Tiho are any indication. I’ve been able to get good, quick feedback from him consistently.
• Security is an area where I pick up on something important

Test Team

• I don’t know anyone in particular on the test team right now but I do know they have test harnesses.
• I am starting to realize that I don’t have any expertise in web security testing. There are a couple of approaches to fixing that 1) Read up and practise testing for security related issues 2) Call on people I know to help guide me 3) Hire someone else to do the security testing.

Equipment and Tools

• It sounds like they can set up a development environment right on their own desktop boxes.
• They have an offsite hosting facility.
• I’m creating matrices and checklists for myself as I go
• I really need to get information on how the developers debug problems and the tools they use.
• They support all major browsers and versions
• They “have plenty more to come when it comes to integrating with other online services”. Got some stuff find out there.

Schedule

• My arbitrary schedule is to be done my test plan creation, risk and coverage matrices and charter list and top 10 issues by Tuesday March 24th, 2009.
• There is a test harness that helps with the regression testing of the weekly code drop

Test Items

• For the scope of test items I’m working that out with Rypple as I work through this process.
• The product is available.
• I am unsure how volatile it is.
• I don’t have any access to the internal workings to know what has a been added lately.
• I can see from the public blog that they just added a connector to facebook.
• The product appears to be reliable enough for me to effectively test it.
• They are doing weekly drops of the code so volatility doesn’t seem like that much of a concern. I guess it depends on whether I’m testing in the context of scrum or not.

Deliverables

• I’m making written reports of my whole process as I go. If I were working for Rypple I would probably not make all this information available in written format. Most of it would be verbal, brainstorming, modeling etc.
• If I worked for Rypple I would see that part of my deliverables were given to other teams – mainly the list top 10 issues a critic might say needs to be given to support and services.

Discovering these things about the project environment and then probing them more might lead me to better information and charters later on.

I need YOUR

help in providing a better value to YOU

. I started my recent posts with many goals – one of which was to demonstrate how I might go about providing value to Rypple and Idée as a tester using the Heuristic Test Strategy Guide as a tool in my process.

I got a few comments on my early posts and then participation tapered off. This got me wondering – am I providing value to you as a reader? Before I go ahead with more posts I’d like to know what I can do better to serve you. My hope is to turn this series of blog posts into a springboard for demonstrating one way to provide value.

How am I doing with demonstrating one of my stated goals – providing value using HSTM? I’d appreciate hearing from you.

We could even use Rypple. It would be a great test and I might get some valuable feedback and so would they!

Adam

Let’s recap what’s been done so far:

• Discovered Rypple and started testing it out. 
• Applied theories from the Heuristic Test Strategy Model. Took notes on what I observed
• Asked stakeholders what is important to them using CRUSPIC STMPL as a guide.
• Went back and forth with the stakeholder.
• Started a charter list.

Here is how I could approach Rypple going forward with what I’ve learned so far

• Explore the project environment using HSTM as a guide
• Talk to Tiho face to face to get more insight 
• Develop a rich set of testing ideas using a variety of techniques. 
• Pull from my RST learning, my JItlearning and model the program using different visual tecniques – mind maps, white-boards etc.
• Put these ideas in a safe place. My blog seems like a good place so far

So have I provided value to Rypple? In response to my last post I got this from Rypple.

Read the rest of this entry »

I’m going to jump back to Rypple for a bit. From my last Rypple post I’ve got multiple jumping off points for a deeper discussion with Tiho. I’m sure there are things that I’ve missed that would be important to them. I’d love to sit down with him and brainstorm. I would go more into the capabilities with him and see what is important.

For reliability/robustness I would have to clarify with Tiho what this means to him. If I were to follow the heuristic test strategy model it guide me to ask if the product will work well and resist failure in all required situations. It goes deeper to talk about error messages and data integrity. I would explore the error messages in the program and ways for invoking them. If I had an architecture diagram I might be able to find ways to invoke these error messages individually and make my reliability testing more thorough. I could even make a mock up diagram based on what I know and ask him to fill it in. Then I could start asking what would happen if certain things failed or didn’t work properly.

I find it interesting that reliability and robustness got combined together. I take reliability to be the way a system performs over time. Robustness is resilience to failure and when failure does happen how the product handles it. A quick idea might explore doing some kind of denial of service attack on Rypple or corrupt the database in some way to see how it handles failure. Another idea would be to use perlclip to insert some really long strings and see if I can past their validation.

Usability is also important. For this I would have to focus on my opinion and present it. I can also poll the people I send message to through Rypple to see what they think of the usability of the program.

On the security side I’ve discovered something I didn’t know – they have an SSL site. There is also mention of an enterprise version. They take steps to prevent sql injection, cross site scripting, and use a state of the art facility to store the data. This is a gold mine for more test ideas. I also have to do more research into security exploits and ways to test SaaS apps for this.

Installability is not perceived to be a problem since they are SaaS. I found this intriguing as I’m not familiar with installation processes for SaaS. They will have to have some method/process of getting the code from dev to production.

I need more info on Testability. Tiho mentions that there are testing harness in place. This is encouraging. What would help me more effectively is to know what kind of logging they have for error messages. Are the logs primarily on the web server? Are there any other types of hooks/harnesses? What special tools do the developers use when debugging?

Maintainability in the test strategy model talks about how economical it is to build, fix and enhance the product. For me this ties closely into the support process and patching process. How easily can the support team, or at Rypple’s stage in the game probably the developers directly, diagnose and fix problems. What’s the turn around time? How readable is the code? If Rypple experiences a growth spurt will the developers coming in say “Wow – this is great code and it’s so easy to understand” or will they say “Wow – this is the worst code I’ve ever seen”.

For portability Tiho mentions that they picked a stack that would allow them to easily move to another hosting provider. I wonder what easy means to them and how “easy” it really is. I wonder how important this might be in the grand scheme of things. If relationships with the provider go sour then it will probably be a pretty big deal.

Localization. There isn’t much to explore since it’s not a focus right now. I think this was perfectly acceptable especially given the phase they are in. If I was a tester at Rypple it would be something I would be heavily encouraging the development team to integrate in a way that allows all the static text strings to be in separate files – especially error messages. This has the potential to save them lots of time and pain in terms of making the product support different languages. We went through this ordeal at PlateSpin and it required a significant re-work and re-test of the product.

At this point to help keep me focused I’m starting a spreadsheet of possible charters. Stay tuned for that.

I found a company called Idée inc.

 who makes really neat products for searching images. Their products are  cool because they don’t rely soley on tags to find images. Instead they use attribuates of the actual image. I must say it seems rather magical and mysterious to me.

Here is how one product, TinEye

, it works according to their website

“Every day TinEye’s spiders crawl the web for additional images. Using sophisticated pattern recognition algorithms, TinEye creates a unique and compact digital signature or ‘fingerprint’ for each one and adds it to the index.

When you want to find out where an image is being used on the web, you submit it to TinEye. The attributes of the image are analyzed instantly, and its fingerprint is compared to the fingerprint of every single image in the TinEye search index. The result? A detailed list of any websites using that image, worldwide”

They also have something called Idée Lab where you can try out their technologies yourself

http://labs.ideeinc.com/

I wondered a couple of things about this

1) Exactly what attributes are they looking at? What goes into the making of these picture signatures or “digital fingerprints”?

2) If I were a tester working for Idée how would I test their applications?

I will post some of my thoughts and notes.

To review the first session – I explored the product and took free-style notes while doing it. I came up with a list of ideas that i can explore further. I also came up with an issues list that I would like to speak to someone about.

For my second session I uncovered a lot of information but I didn’t operate their product. How can this be testing? What did I do?

I emailed one the folks at Rypple and asked what kind of information they would be interested in. I used the quality related information from James’ Heuristic Test Strategy Model. The quality heuristic can be remembered with the mnemonic CRUSPIC STMPL. What does that stand for?
It’s most of the ility type words that people talk about in the context of software. This mnemonic has become stuck in my head – when James and Michael present it they often tell stories that might help you remember it. They also encourage people to make up their own mnemonic as well.

CRUSPIC STMPL stands for:

Capability
Reliability
Usability
Security
Performance
Installability
Compatibility

Supportability
Testability
Maintainability
Portability
Localizability

I wasn’t expecting much of a response. I was pleasantly surprised. I got a response from Tiho who is listed on Linked In as Star Software Developer at Rypple. Here is his response.

Read the rest of this entry »

I have to give hats off to John Creson – a tester at Autodesk. When he arrived at the testers lunch today he brought a person from his support team! WHAT!!! A support person hanging out with a bunch of testers?? Can’t be. Really?

It’s true – it’s a real life story of teams working together. My last post because was too close to lunch for him to gaming the system

Hats of to John for expanding our testing circle to include people from his support team.

Way to go John!

I’m about to let the cat out of the bag. Some of you know it already – some of you don’t. 

If you really care about quality work on improving people’s relationship to your software. Don’t stop there – take it to the next level. Work on improving how the people who work on your software interact with each other.

Get support and test working together – really working together. Have people switch roles for a couple of days – walk a mile in the other guys shoes. Have them shadow each other. Get your testers and developers together (if they aren’t already). And GAASSP.

Get your developers and your support reps working together.

Don’t just get them exchanging emails. That’s not going to help. Sitting in on meetings together doesn’t do much either. Have developers start taking support calls. Writing KB articles, reviewing calls with support, dealing with irate customers.

I bet you might have a team of services guys out “in the real world”, flying around to customers, helping them out, fixing issues, working with support. Why not hook them up with the testers and developers too? Don’t tell me they don’t have time to help improve Quality.

Like magic better people interactions make quality problems disappear

If these teams really work together I can promise you something magical will start to happen. Your quality problems will start to disappear. Magically. Poof!! Gone.

Why? Because you have increased the value you provide to your customers. Quality is NOT an attribute of software. You can’t touch it, you can’ count it. Quality is a relationship between the user and the software. So keep working on it.

Here’s the thing. If your customers find a bug and you deal with it in a timely and appropriately manner – your customers will more than likely not be upset. If you have a skilled support person they will take ownership and let the customer know their issue is being looked into. They can follow up with the customer – even if you don’t have a fix yet. That’s crazy talk man!! Make the customer feel like their issue is important and follow up even if we don’t have fix!! Pffft.

Does this mean that customers won’t get mad? No.

Does this mean your support guys won’t want to hang up on rude customers? No.

But it does mean your customers might be happier when it comes to renew their support and services contract because they got value out of it. They aren’t just a number anymore – the are a real, living human being that someone at your company cares about.

The metrics game

Once you start you are going to run into a problem   Read the rest of this entry »