Adam White

Just read an interesting article called In favor of tough criticism .

It’s a bit of a long read but well worth it. It’s especially timely as I am discussing testing dashboards with collegues and peers. I enjoyed the comments and I especially like the conclusion.

“The future of critical exchange stands at a crossroads. The increased reliance on faint praise, along with the rise of anonymity online, threatens to enervate the free flow of ideas in academe…. It is time for literary scholars to question their critical affiliations, to question behavior that encourages conformity over nonconformity; faint praise over pointed criticism; anonymity over transparency. Telling a colleague “You’re wrong” shows more compassion and collegiality than remaining silent—or hiding behind a cloak of anonymity.

We need to grow thicker critical skin. Why? Because critical behavior that always results in a chorus of affirmation is nothing more than conformity; because allowing views to persist that need to be challenged is nothing less than critical mediocrity; and because failure to tell our colleagues what we truly think about their work is simple dishonesty. A reshaped critical culture will help build a more robust, honest, and transparent academy.”

Even though the writing is focused on the academic world I think the ideas have usefulness outside that context. It made me think about the implications at work for direct reports, peers and bosses as well as colleagues in the technology industry.

What’s your thoughts on the situation?

- Critique (even if it’s harsh) to help your friend, employee, colleague, boss make a better argument or take the approach of “If you don’t have anything nice to say don’t say anything”?

- What is the proper forum for criticism – private, publicly or anonymously

- How does one prevent critiques from getting personal?

I was impressed by what Cong had done. I thought his summary was thoughtful and showed understanding of the concepts. I went through bugs and reviewed the answers to each of the 4 questions. I learned a few things about the details of our product that I didn’t know before.

I picked up the phone and called him to talk to him about his findings. I gave him credit for doing what he did. When discussing the bugs I asked him if he could do this process for every bug or if it would be too time consuming. His answer was something to this effect “Well even for bugs that I thought were trivial I still learned something from talking with the developer so I’ll have to use judgement in the future”

In discussion of one particular bug I asked him to think about whether it was something we should write down as a test idea for future release and also to help future testers learn about the product. I didn’t give explicit directions/instructions on whether this is an expctation/requirement for him. He will have to figure that out for himself and decide where the info would go.

One point we discussed was that there was no response to my initial request until it was done. Throughout the week I wasn’t sure if he was putting any effort into it. I’m also at fault for not setting expectations for an initial response. We both learned a lesson from this and can now communicate more effectively in the future.

I feel like now when I respond to his daily updates we can speak a common language and have a common understanding.

Here is my summary of this interaction

What went well

• Attempting to make Cong feel safe and that I wasn’t threatening or unhappy with his work. I am his boss’ boss afterall and I had never given him a direct request before
• Clearly set completion date
• Set out expectations on how the end result should be delivered
• Gave some local options for getting help
• Follow up via phone to discuss findings and do a little bit more coaching

What could be improved

• I could have picked up the phone in the beginning and explained the exercise, following up with the email writeup
• In my initial request I could have asked Cong to respond and either accept, reject or negotiate my request and give him a date to do this by
• I could have followed up with him mid-week to see how things were going with the task
• I could have set up a skype call with him

Things to remember when coaching remotely

1. Look for things that will teach you about something of interest to you – not only the person who you are coaching
2. It’s important to set two deadlines – one for the initial response/commitment and one for the actual deadline
3. If you’ve not met the person or had very little interaction with them make sure they don’t feel threatened
4. Give the person options for getting local help
5. Follow up via phone/skype, give feedback watching for anything from #1

I’d love to hear any comments on this situation or thoughts/experiences you have with remote coaching/training

Cong’s response  (only edited formatting)

<begin response>

Summary

One of the most important things in testing, is to have a sufficient number of perspectives. We can either look at a product by its elements: Structure, Function, Data, Platform and Operations-SFDPO, or by quality criteria from customer’s point of view, CRUSPIC-Capability, Reliability, Usability, Scalability, Performance, Installability, and Compatiblity. Note that scalability and performance are closely related, and usability, what might annoy or frustrate the person or program that uses the product, and installability should not be forgotten.

In test design, risk-based testing will try to anticipate a problem and test for it. Rapid Testers model the risk in four parts. There is a risk when a victim may be affected by some problems caused by a vulnerability in the product that is triggered by some threat. The model itself is a good application to the exploratory testing we do here at Platespin. When deciding how and whether to test the risks on the risk list, where risks are assessed via a combination of the four parts, tasks are prioritized and used to guide exploratory testing.

As a tester, especially a tester in exploratory testing, we examine the criteria (CRUSPIC) from a user’s point of view. Surprised, annoyed, puzzled, frustrated, or maybe satisfied, emotions are generated when user experiencing our product. Emotions, “arousals”, should guide us in testing and work as oracles that trigger heuristics and wake us up to the discovery of a new problem, a problem that might be function-wise correct but interfere customer’s goals.

In addition, there are several key points that can lead to successful testing: A diversity of workflows tends to expose sequence-related bugs; A scriptable or programmable API to a product makes it more testible (which is the case that we use vmware ESX and vCenter) ; shortening the feedback loop between developer and end-user is a powerful force for beneficial change; real (realistic) circumstances tends to reveal problems and to help identify the kinds of problems that are important to real users.

5 Bugs discussed with developers

Edit – removed bugs that were discussed

Please point out anything you like.

Thanks,
Cong
<end response>

I thought I would share something I did recently regarding remote coaching/training. My inspiration for sharing comes from here and here

I have always enjoyed training new software testers. Lately it’s been more difficult for me to do since I’m in a different office from the bulk of the testers on my team. See my linked in profile for an explanation as to why I’m away.

Given that I’m away I started looking for ways to continue engaging with my team. One opportunity came from reading Cong Zhao’s daily updates.

About Cong: Cong joined us from Waterloo for his second summer work term. His first work term was with ACRP. They provide testing fixtures to Rim to test blackberry. You can find out more about him on linked in

About daily updates: I have everyone on my team send a little summary of the interesting things they learned that day. It’s meant to resemble newspaper headlines that catch your eye, make you want to read further and hopefully learn something of value to you.

Note: I’ve asked Cong for permission to share the process we went through.

Here is the initial correspondence I sent to Cong based on something I read in his daily update.

Cong,

Mike and Herb are out so I’m going to do some remote mentoring with you. I reviewed your list of bugs. You did a good job at re-running the original steps and documenting that in the bug. I want to challenge you to figure out what other areas of the product might have been affected by the changes you are testing.

I want you to ask yourself (and others) “What else could have been broken as a result of this fix?” “What areas of the code where changed and what other areas depend on/use this area” This will cause you to become familiar with the code and will probably require talking with the developer who fixed it.

Questions you should be able to answer for every bug

• What area of the code was changed?
• Why was it changed (explain why the change fixes the bug).
• What needs to be done to verify that the fix resolves the original problem.
• What needs to be tested to prove that the fix does not cause the product to break in some related but different area.

For some guidance on different approaches to thinking about the software check out these articles

• More Than One Answer More Than One Question
• Is There A Problem Here
• Test Design With Risk In Mind
• The Proof Of The Pudding

Action items I expect you to complete by next Friday.

For Articles listed above

• Read these articles over the next week
• Write a summary of your learning from them
• Send summary to Mike, Herb and myself

Look for and document your exploration of bugs beyond the original description.

• Pick 5 issues that had code changes that you are assigned to close.
• Speak with the developer and get the questions from above answered. Document the answers in the bug when closing.
• Show evidence of the above to Mike, Herb and myself.

Given that Herb and Mike are out and I’m remote – you could also rely on Joseph for som
e guidance as well. Let me know if you have any questions or if you cannot complete this assignment by next Friday.

Thanks,

Adam

What I think I did well with this communication

• Explained to Cong what I was intending. I attempted to make Cong feel safe and that I wasn’t threatening or unhappy with his work. I am his boss’ boss afterall and I emailed him fairly out of the blue.
• Clearly set completion date
• Set out my expectations on how the end result of the work should be delivered
• Gave him some local options for getting help

What could have made this communication stronger

• I could have asked Cong to respond and either accept, reject or negotiate my request and give him a date to do this by

I’d love to hear any comments on this situation or thoughts/experiences you have with remote coaching/training

I’m going to jump back to Rypple for a bit. From my last Rypple post I’ve got multiple jumping off points for a deeper discussion with Tiho. I’m sure there are things that I’ve missed that would be important to them. I’d love to sit down with him and brainstorm. I would go more into the capabilities with him and see what is important.

For reliability/robustness I would have to clarify with Tiho what this means to him. If I were to follow the heuristic test strategy model it guide me to ask if the product will work well and resist failure in all required situations. It goes deeper to talk about error messages and data integrity. I would explore the error messages in the program and ways for invoking them. If I had an architecture diagram I might be able to find ways to invoke these error messages individually and make my reliability testing more thorough. I could even make a mock up diagram based on what I know and ask him to fill it in. Then I could start asking what would happen if certain things failed or didn’t work properly.

I find it interesting that reliability and robustness got combined together. I take reliability to be the way a system performs over time. Robustness is resilience to failure and when failure does happen how the product handles it. A quick idea might explore doing some kind of denial of service attack on Rypple or corrupt the database in some way to see how it handles failure. Another idea would be to use perlclip to insert some really long strings and see if I can past their validation.

Usability is also important. For this I would have to focus on my opinion and present it. I can also poll the people I send message to through Rypple to see what they think of the usability of the program.

On the security side I’ve discovered something I didn’t know – they have an SSL site. There is also mention of an enterprise version. They take steps to prevent sql injection, cross site scripting, and use a state of the art facility to store the data. This is a gold mine for more test ideas. I also have to do more research into security exploits and ways to test SaaS apps for this.

Installability is not perceived to be a problem since they are SaaS. I found this intriguing as I’m not familiar with installation processes for SaaS. They will have to have some method/process of getting the code from dev to production.

I need more info on Testability. Tiho mentions that there are testing harness in place. This is encouraging. What would help me more effectively is to know what kind of logging they have for error messages. Are the logs primarily on the web server? Are there any other types of hooks/harnesses? What special tools do the developers use when debugging?

Maintainability in the test strategy model talks about how economical it is to build, fix and enhance the product. For me this ties closely into the support process and patching process. How easily can the support team, or at Rypple’s stage in the game probably the developers directly, diagnose and fix problems. What’s the turn around time? How readable is the code? If Rypple experiences a growth spurt will the developers coming in say “Wow – this is great code and it’s so easy to understand” or will they say “Wow – this is the worst code I’ve ever seen”.

For portability Tiho mentions that they picked a stack that would allow them to easily move to another hosting provider. I wonder what easy means to them and how “easy” it really is. I wonder how important this might be in the grand scheme of things. If relationships with the provider go sour then it will probably be a pretty big deal.

Localization. There isn’t much to explore since it’s not a focus right now. I think this was perfectly acceptable especially given the phase they are in. If I was a tester at Rypple it would be something I would be heavily encouraging the development team to integrate in a way that allows all the static text strings to be in separate files – especially error messages. This has the potential to save them lots of time and pain in terms of making the product support different languages. We went through this ordeal at PlateSpin and it required a significant re-work and re-test of the product.

At this point to help keep me focused I’m starting a spreadsheet of possible charters. Stay tuned for that.