Usability Testing

The goal of usability testing is to get feedback from real users primarily by having users complete a set of tasks within your prototype using a script that you will provide.  This feedback is to ensure that they can understand and use your application and feature as intended as well as to gather suggestions for improvements to the product or features being tested.  If they can’t then it is during this time that you’re keying in to where they got stuck or had questions.

For your script you’re going to want to start with a quick introduction, i.e., “Thank you for coming in today and for giving some feedback.  Today we’d like you to go through this prototype doing a few set tasks to test the product.  We’re not testing you, but the product itself.  If you would please think aloud as you’re completing these tasks so we can gather as much info from you as possible.  You’re not offending me with any commentary you provide.”  We do this so that our participants feel very comfortable and they give us as much honesty as possible so that we aren’t led down the wrong path in developing our products and features. 

After the initial intro you’ll want to get some context around the user’s existing behaviors and workflows.  You’ll want to ask them open-ended questions like who, what, where, when, and why.  These questions will tell you how your user behaves and will structure and adapt the tasks you’re about to give them so that it fits whatever they were doing before.  A few examples of questions you’d want to ask would be:

• When was the last time you did X?
• Can you give me a specific example of what you were doing during that time?

After your intro and introductory questions you’ll then show them the rough prototype.  You’ll explain that the screens you’ve done are just mock ups and they aren’t all perfectly functional and that I’ll point out the times I know something isn’t functional.  From here you’ll make sure to take copious notes, check to make sure they actually completed the task, and you’ll want to understand contextual points.  This could be things like seeing the user get stuck at a certain point, seeing them click somewhere expecting to be taken somewhere, but didn’t.  

Be sure to ask follow up questions.  Asking, “What did you think was going to happen when you X?”  or “Is there any other way you might have tried to do that?” or “What did you expect would happen when you clicked on that button?”.  Commentary from them after these would allow you to take the feedback to improve your product. 

When you’re wrapping up you’re going to want to ask debrief questions.  Questions you could ask would be, “What did you like or dislike about this current prototype?” or “What are some features or functionality you would’ve liked in this prototype?” or “What parts of this page were very important for you?”

An example of a usability test script I’ve created follows here:

Usability Test Script Template

1. Introduction

Take some time to make your participant feel comfortable and use this opportunity to set

some expectations for the usability test.

Items to mention include:

“Thanks for coming in today; we really appreciate your time and your frank feedback. This is an

informal session where we’ll be going through a prototype and we’re not testing you, we’re

testing the product.”

“Please think aloud as you complete these tasks.”

“I didn’t design this product so please be as honest as possible and don’t worry about hurting

my feelings or offending me.”

2. User Context Exploration

Ask open ended questions to get to know your participant and better understand their current

workflows & behaviors in relation to the prototype that you’d like to test. A few examples of

questions include:

“When was the last time you did X?”

“Can you give me a specific example of what you were doing at that time?”

3. Tasks

Here is where you have a series of tasks laid out for your participant to complete. Again, you

want to set expectations that your prototype is not going to be 100% functional. Remember to

take detailed notes when your participant is going through these tasks!

Task 1:  Login as a production manager and create a new project.

Task 2:  Login as a crew member to see how far along John S., a customer, and his project is.  Clock in and then complete the project as a crew member.

Task 3:  Login as a customer.  Check the existing comments and then also add one of your own.

Also remember to follow-up with relevant questions as your participant is going through tasks,

I.e.:

“I noticed you tried to do [x], what did you think was going to happen there?”

“Is there any other way you would have tried to do that?”

“What did you expect would happen when you clicked on that button?”

3. De-brief / Wrap-up

Use the last few minutes to ask high level debriefing questions to get further insights. You

might want to ask questions like:

“What did you like/dislike about this prototype?”

“What are some features or functionality you wish you had here?”

“Which parts of this page were very important to you?”

How to Create a User Persona or Avatar

Your company is rockin and rollin.  You’ve got good people in place, your branding is on point, and your core product is sky high.  You want to expand and compliment your team and product with another dynamic product in a new segment that you see opportunity in.  Where do you start?  For starters you need to truly understand your customer and creating a persona or avatar is one way of doing that.  What follows is a structure I use to discover what unique problems your business solves for your customer.  We’ll have a better idea of the kind of story could you tell to help you connect with your customer and what things your products or service do to solve these fears and frustrations.

Now we’re going to build your customer avatar or persona…..

Who is your customer?

Age?  

Gender?

Socioeconomic status = how much money they make?

Spending Habits?   

Personality?  

Name?  

Photo?  

Where does your customer hang out?

Newspapers, magazines, websites?  

Physical locations, conferences, retreats?  

Other businesses they frequent?  

Pains & Problems?

Customer fears?

Pains?

Frustrations?

Hidden costs?  

Financial?

Physical?

Emotional?

Opportunities?

Financial?

Physical?

Emotional?

Small tweaks?

How to ask the Right Questions as a Product Owner

An interview guide is what you’ll use in the main body questions portion of your interview.  Be sure to think about how to phrase non-leading questions that won’t influence the answer.  Try to ask about specific moments in the past.  We’ll use email to try and illustrate good and bad questions.  A bad example of a question to ask would be as follows:

• “How anxious do you feel when you have a large number of emails that you haven’t responded to yet?”

Vs

• “Can you tell me about the last time you felt anxious about your email inbox?  What happened and how did you feel then?”

Version one of the question is not good for a few reasons.  One you are specifically pointing to them feeling anxious when there are a large number of emails in their inbox.  News flash.  Not every single user feels anxious when they have a lot of emails in their inbox.  Some have 10,000 plus read and unread emails and live happy and full lives.  Some people may feel anxious because they have a bunch of labels that do or don’t get used.  Some may feel anxious that they have archived 10,000 emails, but just haven’t ever read them.

Version two of the question is a lot more open ended.  It isn’t specifically pointing to the large number of emails, but more about the last time they did feel anxious, which will elicit a truer picture of what their problem is/was.  This gives them more freedom to respond a more valuable way.

Another example of good vs bad questions to ask.

• “What was the last productivity app you used?”

            Vs

• “Tell me about the last time you used a productivity app.”

Lean towards asking “who, what, when, where, and why” with why being the number one question to ask.  Do not give up questioning early.  Be that annoying five year old that keeps asking why after someone has explained something to them.  Be sure to keep digging to get to the core reason why someone does or thinks something. 

Bad questions: “Would you…?  Did you…?  Is it…?”

Better way to ask this question would be:

• Why?  When?  How?  
• What’s an example of that?
• Why does this matter to you?
• Talk me through the last time that happened.
• What else have you tried?

Bad question: “Do you think this is a good idea?”

Better way to ask this question would be:

• Walk me through how you currently solve this problem.
• What parts of your current process do you love or hate?
• What other products/processes have you tried before settling on this one?
• Are you actively looking to replace this product/process?  If so, why are you still using this one?  If not, why not?

Bad question: “Would you buy a product which did X?”

Better way to ask this question would be:

• How are you currently solving X?
• How much does it cost you to do this?
• How much time does it take to do this?
• Talk through what happened the last time X came up.
• Have you tried searching for a solution to X?

Bad question: “How much would you pay for this product?”

Better way to ask this question would be:

• How much money does it cost you to solve this?
• How much time does it take to solve this?
• What is the budget you’ve allocated to solve this?

Keep asking “Why?”!  If you’re building something like a notification app that lets a teacher know if a parent’s child is going to be late or absent and you’ve got built out all these cool buttons and it is all automated, but the confirmation messages are cold and robotic the app may not get used.  To solve that problem say you add a custom message option that the parents can then use, but it still feels robotic and cold the app may not get used again.  Then you also add the teacher’s picture to each page of the app.  Asking why parents aren’t using the app or only weird parts of it would bring to the surface that you can use automation, but instead of, “Your message has been received.” you could change it to, “Thank you Jane.  We’ve let Mrs. Periwinkle know that your child be will running late.”  This way it doesn’t feel like their message goes into a black box or the ether of the internet never to be seen.

The following is a list of potential questions you could ask during your user interview.  Post the interview you’ll want to list each of these in the column headings of a spreadsheet and then each row is a user’s name and then their response would go to the right and underneath each question.

1. What are your big goals and areas of focus right now?
2. What are your big problems right now?
3. What are the implications of that problem?
4. Can you walk me through the last time this problem happened?
5. What makes it so awful?
6. How are you currently dealing with this problem?
7. How are you currently coping without a solution?
8. Why haven’t you been able to find a solution to this already?
9. What alternative solutions have you tried?
10. How might you fit a solution into your workflow?
11. How much time does it take to solve this?
12. What is the budget you’ve allocated to solve this?
13. Can you think of anything else that I should ask you about this problem?
14. Is there anyone else you would recommend I speak to about this problem?

Asking the right questions that don’t lead your respondent to biased responses that don’t reflect reality will help you develop and solve real problems that will generate revenue and increase adoption and use.  Worst case scenario is you’ve validated some problem and you go off and build a solution, but then after you realize they never had a budget to solve it in the first place.

User Interviews

User interviews are where you will actually sit down with or collect information from people to see if your solution to a problem is viable or not.  Expect to block out 30 minutes to an hour of time where you’ll be actively listening. When finished you will need time to analyze what information was gathered both quantitatively and qualitatively.  When trying to identify who you want to interview you need to know who to include, but also who to exclude. Using a screening questionnaire will help guide you in this endeavor. Google forms is a great way to create your screening questionnaire.  Below is a brief example of what that could look like.

Having a monetary incentive is a great way to get qualified participants.  Your screening questionnaire will also have filtering questions in it so that you know the submitted participants would be useful in your research.  Once it is created you’ll want to post it online to find your candidates. Some places you can go to place them are:

• Craigslist
• Social Networks
• Community Groups (gmail forum, productivity forum, etc)
• Student Groups
• Personal Networks

User Interview Structure

Typically your format will be as follows:

• Introduction
• Warm up/context questions
• Main body questions
• Follow-up questions
• Debrief/Wrap-up

In your introduction be SURE to state something along the lines of, “Hey just so know I am not the creator of this product and so please be as honest as possible because you won’t offend me on any feedback given.”  This will make them feel more comfortable and honest about what they have to say. Again we’re legitimately trying to solve a problem, not get confirmation bias from friends and family and participants who think getting paid would be in some way tied to painting the products, features, solutions in a positive light.

When you’re conducting the interview it is ideal to pair up.  This allows one person to ask the questions and guide the interviewee through the interview while the second person is taking notes.  If possible try to record the session whether that be audio or video with the interviewee’s permission of course. This helps to see if you had any leading questions and discount the responses if so.  It also helps to see if you missed anything outside of the notes you took during the interview.

Interview Analysis

After the interview is completed you’re going to want to clean up your notes.  Be sure to pull out key insights, key quotes, and top learning moments. It is helpful to create a spreadsheet to organize your thoughts and user responses where the column headings have key questions you wanted to ask in your interview and the rows are the responses of each user with each cell containing the user’s response to the column heading question.  If at all possible you’re going to want to distill commonalities and themes and see if there are any common relationships via affinity mapping or open/closed sorting.