Adam White

The purpose of this entire exercise is to show how I would apply the heuristic test strategy model developed by James Bach

What follows in the post is the notes from my first testing session intended to show show my thought process during my exploration.

*Warning – I’m stepping out on a limb here. I’m putting my work out there for review. Keep in mind they are my notes and may not make sense to you. They aren’t edited for spelling mistakes, grammar or anything else. The notes were taking as a stream of consciousness while I was testing.

If you want to know more about session based testing check out the this write up

To accompany my notes I have a 60 minute video taken with BB Test assistant. I also have a few screenshots that I took. I will try to find a way to post these so that i’m mindful of bandwidth constraints.

# Charter: Explore Rypple functionality

# Tester: Adam K White

# Length: Normal

# Charter vs opportunity
100% charter

# Set-up
0

# Notes
Clicked link in email for account activation. Typed in a 52 character password. After putting in my password again to verify I was asked to invite contacts I was prompted with a page that asked me to watch a video to get started. It showed import contacts on this page as well. I would think import contacts should have been on the invite page for those who are familar with the functionality.

The main functionality seeems to be
- Get feedback
- Review feedback
- Give Feedback
- My Network

In the top frame we have
- Welcome, Adam
- Help
- Give Us feedback
- Logout

I will explore the functionalities within the get feedback area.

Get feedback has 3 sections

Section 1
What is your question?

Max Characters appear to be 140. The training video said 100. This is inconsistent. I would imagine they found that 100 characters is not enough. The text box that appears to allow 2 lines of text to accomodate 140 characters. I would assume 70 characters per line. Will test later.

There is an icon of lightbulb below the “What is your question text box” with a + sign at the bottom right of it. A mouse over shows a tool top of suggested questions. Click the button expands a list of canned questions.

Section 2
What is your question about?

Text box has the following text
<Select up to 3 attibutes, seperated by a comma. E.g. leadership, negotiating, communication>

There is a similar light bulb icon which upon clicking reveals key words such as
leadership, integrity, presentation, attitude, communication, more attributes

Clicking more attibutes reveals
delegation, vision, strategy, listening, reliability less attributes

Noticed the the L in Less attibutes in capitilized. Problem? Don’t remember if the m in more attributes is capitalized. Clicking less attributes confirms that it is. Should it be? Something says to me that is should be. It’s consistent within itself but it violates something else. Will think about it.

No character max displayed
Section 3

How do you want to ask?

Two radio buttons
- Ask advisers through email
- Get a unique URL

There are two blue icons that seem to indicate information.

Just noticed them in the other sections as well.

Clicking gives a tooltip/fly out of information.

Text box is bigger than previous two. Appears to be 3 lines. No character max displayed

Can createa group from these advisors as well. Interesting functionality. I wonder how many people can be in a group of advisers.

—–

At the bottom of the 3 main sections there is a personalize your request, preview and create feedback request.
Personalize your request
- Shows a 3 line text box. Max appears to be 440 characters

Below the text box there is an arrow that reminds me of the undo arro in word or something similar to a back button in a browser. Mouse over shows a fly out saying “Reset personal message”. Seems to fit my expectations.

Preview
- Click preview says that need to enter at least one attribute. Thought this was strange. I expected it to ask me to enter a question first. Scrolling up I found that I had a question in there.

Removed question and clicked preview. Program prompted me to type a question. That meets my expecations.
Going to sumbit a question now.

Typed question “How would you test this program?”

Attributes “testing, listening, delegation”

I wonder if there is character max on the attributes – test for later

Unchecked rate these attributes

Started typing the emails of people i added earlier during invite session. They show up in a drop down while typing. Meets my expecations. If they didn’t show up I’d be disappointed.

Typed a personal message “I’d like your help in exploring this program. I’m testing it out for fun – not for profit. ”

Click Preview Shows a dialog. First line is a red circle with a white exclamation point in it. Text to the right of it says “Heads up. You’re one of two people being asked”

That’s odd – since I didn’t typ in my own email. Investigate later

Sumbitted my question. Dialog shows submitting and then success I think. Don’t rememeber and wasn’t taking a video.

Final page shows the date, my question, advisers asked, attributes, labels.

Not sure what this label business is about – investigate later.

Drop down box with “select action” is visible. text in box says add advisors, close request, send reminder.

There is a button called share feedback – will investigate later

There is a page divider and below it shows

No responses so far with links Download CSV, Expand All, Collapse all. Then button with Share feedback and back to listing.

These buttons seem to be duplicates of the ones from the top of the frame. Might make more sense once I’ve asked more questions and gotten feedback.

Going to stop this session now, clean up my notes and organize my findings

While typing up notes I noticed that I’m now in the review my feedback section.
# Issues/Things to follow up on
- Not being given the option to import contact right away after activating. I might have missed it. I have no way to reset my account to “new”. As a tester this would a useful tool to have. Often times I want to set my account back to a known state. Unfortunetly this doesn’t always work properly either.

Click Preview shows red circle with a white exclamation point in it. Text to the right of it says “Heads up. You’re one of two people being asked”. I wasn’t one of the two people being asked


# Test ideas
- Test for max characters in text boxes
- Research share feedback functionality
- Research labels under review my feedback.
- Explore download CSV functionality
- Explore Expand all and collapse all
- Create a workflow diagram of all the dialogs and their functionality.
- Find a way to do some automatic data input and exercise the workflow.

From my test ideas I have a set of new charters to explore. It took me roughly 60 minutes to get this basic list of ideas. From there there are a few different directions I can take. I can discuss the issues I found and I could prioritize my test ideas with them. Before I run off and operate the product more I can also jump off on a different thread. Stay tuned….

I did some exploring on their site and liked their premise “Get fast, honest, feedback from people you trust” 3 easy steps 1) Ask a question of people you trust  2) Get honest, private feedback 3) Use their feedback to improve. Sounds easy enough. I noticed that they are in beta so I thought I would sign up and try their product out.

I got a note saying it would be a few days before my account was set up. I sent them a note stating that I’d like to test their product for two reasons. I’m a Director of Software Testing who would like to get back in touch with hands on testing during vacation and secondly I really like the idea behind the product since I’m a Director of Testing and enjoy feedback. I got my activation within about an hour Sometimes it pays to ask.

I’ve been watching James Bach and Michael Bolton’s blog threads on their testing of IMVU. They have uncovered some things I find very interesting. Most of the items pointed out in IMVUs software turned out to be not important by IMVU’s standards and their impression of what end users want (and I got the impression that it wasn’t important to IMVU as a business either).

One thing I would like to see James and Michael blog about is how they could provide information to IMVU that was useful and important to them. What is that IMVU actually cares about in regards to its software? How does one go about discovering this information?

I decided I could demonstrate my process for discovering this type of information. I’ve been heavily influenced by James and Michael throughout my career and this seems like a great way to showcase those learnings applied to products most people can access.

What I will be posting over the next few days is my interpretation of the heuristic test strategy model and the principles of Rapid Software Testing applied to two applications I have been exploring. The first one is called TinEye from Idee and the second one is a product called Rypple 

I’m still forming notes – so stay tuned for my findings.

P.S – Thanks to those who wrote to give me a kick in the butt to remind me to write

Warning – 3/4 finished thought.

In the event of an unknown or unexpected system behavior in the software world – the test plan documents mean nothing.

When we develop our testing strategy we do two things – outline what we WILL  test AND what we are NOT going to test. The latter is always one of the hardest to figure out, explain and do effectively.  A close relationship with development is required. Trust from both sides is crucial. Expert knowledge of the system and how changes affect it is required. 95% of the time this process works but other times it backfires. It is usually the unknowns that screw us.

For project X a lead tester wrote a test plan document to satisfy project stakeholders. The document brought up lots of good questions and discussion. The one section that got a fair amount of discussion was the “Not going to test” section. We explicitly stated that PERFFORMANCE would NOT be TESTED. We would eyeball the performance (web based UI connected to our 4 year old core server technology) and not provide a formal analysis of request/response times. Why take this approach? We had done load testing with a shipping release build 1 month prior. We got benchmarks, provided this info to development and discussed the risks. Everything looked good. There was no reason for any of us to suspect performance problems.  Everyone agreed “There should be no reason we would have performance problems”. All project stakeholders “signed off” on the test plan. 

What do you think the single biggest problem found while testing?

(more…)

What do I do with this information as a user? I don’t know how to troubleshoot this – nor do I have time to. Why can’t it tell me what RSS feeds it’s having a problem with.Of course I searched for the error message using google. I searched for “outlook RSS feeds error” and “RSS feeds error 8004010F” neither of which yeilded a link to a solution.

I like to do testing exercises with my team. Below is a list of links to various activities I have done with the team at some point in time. I usually turn these things into an interactive session. I thought I would share my list with you in case you want to do some of them with your team (or by yourself)

Observation skills
Which way is she spinning?

A similar picture

Cartoon
Powers of Observation

Activity – watch videos and answer questions at the end
New TV Show

Observation – General Link
http://www.smart-kit.com/scategory/brain-art/>Brain Art

Are the two blocks different colors?

How many differences can you find?
Spot the Differences

Activity for Team
How many F’s?

Activity
Optical Illusion

Cartoon
Perception

Video
The Amazing Color Changing Card trick

Where  possible with these exercises I get the team write out their observations before we go into discussions about what they see. This is to help prevent certain biases from showing up – mainly confirmation and hindsight biases

Confirmation bias, according to wikipedia, is a tendency to search for or interpret new information in a way that confirms one’s preconceptions and avoid information and interpretations which contradict prior beliefs.

Hindsight bias, sometimes called the I-knew-it-all-along effect, is the inclination to see events that have occurred as more predictable than they in fact were before they took place. Hindsight bias has been demonstrated experimentally in a variety of settings, including politics, games and medicine. In psychological experiments of hindsight bias, subjects also tend to remember their predictions of future events as having been stronger than they actually were, in those cases where those predictions turn out correct.”

When we did these exercises it made for some really interesting discussions – especiallly the spinning dancer and the different color blocks.  Doing these exercises can help you become aware of (and possibly manage) confirmation and hindsight bias.

Places you might find more information and exercises
Inattentional Blindness
Selective Attention

Michael Hunter, of Braidy Tester fame, posted a challenge on his blog. He showed a parking sign photo and asked how many different ways it could be interpreted. I decided to take 45 minutes during one of our weekly test team meetings and try it out. What the hell does this have to do with testing you might ask? Well - thanks for asking. Here is how it relates for me

• Idea generation – we always need to generate new ideas for testing
• Clarification – requirements are mostly ambiguous and this was practice at clarifying them.
• Sharing and discussion of ideas – “It could be interpreted this way” which lead to someone else saying “Yes and this way too” - Team work, supportive attitude with a focus on sharing.
• Time boxing so we don’t get carried away - Important in testing and just about anything else
• Coming up with alternate solutions/problem solving – It’s good to be seen as a problem solver instead of “Bearer of Bad News”

Needless to say we had a great time doing this activity. I time boxed the idea generation to 15 minutes and then had about 15 minutes of discussion.  We used index cards to write down our ideas. We went around the room and each person read an idea from their card. We ran into some duplicate ideas but that was ok – we weren’t focused on having every idea be unique. The puprose was to put on our “thinking caps”

Now it’s easy to take an idea and tear it to shreds.  To quote Edward De Bono in The Thinking Course “..critical destuction of one hypothesis has never produced a better one. It is creativity that produce the better hypothesis.” As a follow up challenge I asked the team to come with a sign that wasn’t ambigous. We explored some great ideas and had a great discussion. A few of the ideas and designs came out to be really clear and (mostly) unambiguous (I’ll have to get them and post it with this article.) The discussion was going so well that it actually went over time and into lunch.

Here are the results which were summarized by Aqiqul Hoda and Michael Hetmanczuk. The participants were Adam White, Alan Walker, Ali Khan, Aqiqul Hoda, Christy Gnanapragasam, Herb Bal, Joseph Kubik, Michael Hetmanczuk, Mortaza Abhari, Thomas Yook and Zhe Chen

1. Can park for 2 hrs from Mon to Sat between 7AM to 6PM.
2. If you have Zone 4 Permit you could park as long as you want.
3. No Limits of parking on Sundays and Holidays.
4. No Parking for Zone 4 permit vehicles.
5. No parking on Sun and holidays.
6. No parking in the Night.
7. Zone 4 permit vehicles parking only between 6 PM to 7 AM and Sun and
holidays.
8. 2 hr parking this side of street.
9. 2 hr parking both side of street.
10. 2 hr parking from this sign onward.
11. Parking at 6PM allowed can go past 6 PM for 2 hrs.
12. Parking once a day only.
13. Zone permits vehicles parking only after hrs but not on sun and
holidays.
14. Zone permit sign maynot be related to parking.
15. 2 hours limit parking between 7am and 6pm on days except Sunday and
Holiday.  Above has exception by Zone 4 permit means Zone 4 permit could
park at anytime
16. Board number 2: hour parking
17. Is the #2 the sign ID or does it indicate 2 hours?
18. Can we park during other ours or only 7 AM – 6 PM
19. Where is the sign?
20. Does it mean I can park from 6 AM – 7 PM?
21. Hol?  What qualifies as a holiday?
22. Does this apply to bikes?
23. Except sun/holiday means on Sunday and holidays – no parking at all
24. On sunday/holiday you can park all day
25. On sunday/holiday the 2 hour limit is lifted, but parking is only
allowed from 7 AM to 6 PM.
26. Zone 4 permit means you can park all the time.
27. Zone 4 permit means you can park 7 AM – 6 PM.
28. Except Zone 4 permit means you can’t park at all if you have a zone
4 permit.
29. Except Sun/Holiday: means only those with a zone 4 permit can park
on sun/holiday.
30. Could be interpreted as 1 hour parking.  The #2 could be something
else, i.e. street number, parking spot.
31. Except sun-holiday: starts on sunday, ends on a holiday.
32. What is a zone 4 permit?  Do I automatically get one?
33. Does that mean I can park long on sunday-holiday?
34.  Maximum of 2 hour parking allowed between 7 AM and 6 PM except
sunday or holiday.
35. If you have zone 4 permit pass you can park the car anytime.
36. Within this zone, a maximum of 4 vehicles are permitted to park.
37. The #2 indicates it is a second sign that indicates “hour parking”
zone between 7 AM and 6 PM except on sundays, holidays.

It’s been a while since my last post. I’ve got lots to write about and not enough time to do it.

 The one thing I do want to mention is the upcoming CAST conference which will be held in Toronto this year. Some of the best minds in software testing will be at this conference. Speakers like Michael Bolton, Robert Sabourin, Jerry Weinberg, Cem Kaner. Not to mention I’ll be doing a presentation as well

The full program is available on the website – you can find it here http://www.associationforsoftwaretesting.org/drupal/conference

Make your reservations soon – Jerry’s tutorial is selling out quickly (if it’s not already sold out)