Posted by Ben Simo
- My fuel gauge is on empty.
- I don't want to stop, but pull into the gas station.
- I tell the children to stay in the car.
- I get out of the car.
- The children are asking me questions from inside the car that I can't hear well enough to understand.
- I swipe my credit card in the pump's card reader.
- The pump responds by prompting me to "SELECT WINDOW OR OUTSIDE".
- The children are still talking to me. I still don't understand.
- I pause and stare at the keypad.
Is there a problem here?
- I pause to think for a moment.
- I hear the children asking me questions that I still don't understand through the closed car windows.
- I scan the keypad again.
- I cant find the "OUTSIDE" button.
Recognizing Bugs
At CAST last week, Pradeep Soundararajan gave a Lighting Talk about the importance of testers being able to recognize a bug. Tests may be of little use if the tester doesn't recognize the bugs triggered by the test.
Sometimes bugs are obvious. Sometimes bugs are not clearly violations of requirements documents. This is especially true when it comes to human computer interaction problems.
Requirements are not always clear and objective.
So, do you recognize why I may have paused when I read the prompt on the gasoline pump? It wasn't the price of the $4 per gallon fuel. It wasn't because I had to think about how I wanted to pay.
I paused because I didn't see a button labeled "OUTSIDE". Plus, I already swiped my credit card indicating that I wanted to pay at the pump. And even if I were to pay at the cashier window, I would still be outside.
I wonder how many minutes are wasted each month prompting customers to select where they want to pay after they have swiped their credit card. I wonder how many other people pause and read twice in search of the button to indicate that they want to pay outside.
The developers and testers of the software in this pump may have not recognized this problem. Maybe the makers executed test scripts -- either automated or manual -- and were blind to the problem. Or maybe they didn't deem it important enough to change.
Sometimes familiarity with the technical details of a system can hide problems that are obvious to those that don't know the technology, the requirements documents, and the test scripts. As testers it is important that we be careful not to let our familiarity with a system make us blind to to bugs -- things that bug our users.
Will you recognize a problem if you see it?
6 Comments:
July 30, 2008-
Pradeep Soundararajan wrote:
-
-
July 31, 2008
-
Joseph Kubik wrote:
-
-
July 31, 2008
-
Ben Simo wrote:
-
-
July 31, 2008
-
Joseph Kubik wrote:
-
-
August 07, 2008
-
Anonymous wrote:
-
-
August 20, 2008
-
Jaanvi wrote:
-
-
Is there a problem here?
There are problems everywhere. If I don't know to recognize them then there aren't.
Users/Customers of the product I test, recognize bugs that I was unable to.
An important part of education for a tester like me is to keep learning different ways that different people use to recognize problems.
In the quest of learning ways to recognize problems, I learned that Specification/Requirement document is one such way to recognize problems.
When a customer finds a bug and reports it, do the developers or the management say, "Hey, but that's not in spec"?
If they don't, then some developers and some management people say that for testers who go out of specification. They are bound to fail more than those who don't say that to testers.
As an exploratory tester, I use the help of oracles to recognize a problem. Oracles ( not the database ) are principles or mechanism by which we humans identify problems.
An oracle might help me to find some problems in the product I test. How do I as a tester find more problems with the product I test?
By learning as many oracles as possible. In the process of learning, I learned that "emotion" is an oracle, too. That sweet lesson came from Michael Bolton who might acknowledge it to Jerry Weinberg.
Ok, now I have "emotion" as an oracle. What problems did I find with "emotion" as an oracle?
I initiate some action in the product as a part of my test, it takes more than a minute to allow me to do something else as it is processing that long.
As a tester, I am frustrated about the fact that, if each action takes that long, then I would run fewer tests as compared to the product responding in a jiffy.
You dont even need to be too smart to recognize that my "frustration" helped me shoot a performance issue with a few operations and functions within that product.
Rapid Software Testing ( www.satisfice.com/rst.pdf ) has a list of oracles that has helped me.
Sometimes other testers are surprised or wonder about how I found that bug. I wonder if they had the oracle - would they have wanted to wonder about it?
We ask a question, "How did you find that?"
People usually respond what actions they did as a reply to that question without recognizing that anyone could have done those actions but without recognizing the bug they wouldn't have found it.
James helped me learn - finding includes recognizing. I argued that not many thousands of testers who find bugs recognize recognition to be a part of finding bugs.
Will you recognize a problem if you see it?
No, I wont unless I learn how to recognize it or more ways to recognize more problems with what I see.
I wouldn't want to learn ways of recognizing problems if:
a) I am happy with the bugs I find.
b) If I think specification will help me to find all bugs.
c) If I am not aware that "recognition" plays such a vital role.
d) If I don't want to test better.
e) If I don't know how to learn to recognize problems.
If we don't recognize this message as important for our testing career, we won't recognize many problems that are right in front of us.
While it took me a couple of minutes to write this comment, I see that Ben has enabled word verification to accept a comment to prevent spammers. In one occasion after typing a long comment, I type d the word that is displayed as an image, it asked me to enter again displaying the new word because there appeared to be a time out for that.
The frustration that I had when all the things I typed disappeared when the new word was shown helped me gain an oracle to find a problem these word verification might have.
I really like your example.
This is also a good example of what a bug report does not need in it.
We as "developers" don't need to know about the kids. All of the steps with the kids can be removed and the problem still exists.
-Joseph-
Joseph,
Developers may or may not need to know about the kids. The steps with the children help explain the possible state of mind of the user -- which may be an important factor in deciding if something is a bug and if it should be fixed.
I suspect that being distracted by children is a common situation for people fueling their cars. Thinking of this user scenario can help a development team step into a user role and think about their software from the user's point of view.
Had I not been distracted by the children, might this have not been as big a problem? Maybe, or maybe not.
Considering various user states of mind and emotions can help us create better software.
Ben
I certainly agree, and depending on your developers / project owner you may need to provide LOTS of context.
The fine line is determining what information helps and what information distracts. I commented, because I read your post, and kept thinking "OK, what do the kids have to do with this?" I was expecting one of them to get out of the car, or have had some insightful help for the confusion had you only rolled the window down. I'm sure someone else will read the same story and think that the extra context is critical information.
-Joseph-
There is certainly a problem in the example that you have given. The pump interface has a usability bug - the user interface and the instruction are incongruent. You are right. Over a period of time, many customers would waste their valuable time after being stumped with the instruction. Moreover, they might not feel good after the completion of their transaction.
Even though it looks like a simple bug, it can have severe implications. Any transaction is based on some level of trust. I wonder if a potential customer would suspect other problems in the system after being exposed to this bug. Your fuel tank was on empty so you probably had no other choice. Another customer of the gas station could take her business elsewhere.
I really liked the post and you have explained it wonderfully. I agree, that many a times a problem goes unsuspected and this is where real worth of a tester is tested.
I go by the rules of suspecting anything and everything in the software and this helps me.
Post a Comment