How to ask the Right Questions as a Product Owner

An interview guide is what you’ll use in the main body questions portion of your interview.  Be sure to think about how to phrase non-leading questions that won’t influence the answer.  Try to ask about specific moments in the past.  We’ll use email to try and illustrate good and bad questions.  A bad example of a question to ask would be as follows:

• “How anxious do you feel when you have a large number of emails that you haven’t responded to yet?”

Vs

• “Can you tell me about the last time you felt anxious about your email inbox?  What happened and how did you feel then?”

Version one of the question is not good for a few reasons.  One you are specifically pointing to them feeling anxious when there are a large number of emails in their inbox.  News flash.  Not every single user feels anxious when they have a lot of emails in their inbox.  Some have 10,000 plus read and unread emails and live happy and full lives.  Some people may feel anxious because they have a bunch of labels that do or don’t get used.  Some may feel anxious that they have archived 10,000 emails, but just haven’t ever read them.

Version two of the question is a lot more open ended.  It isn’t specifically pointing to the large number of emails, but more about the last time they did feel anxious, which will elicit a truer picture of what their problem is/was.  This gives them more freedom to respond a more valuable way.

Another example of good vs bad questions to ask.

• “What was the last productivity app you used?”

            Vs

• “Tell me about the last time you used a productivity app.”

Lean towards asking “who, what, when, where, and why” with why being the number one question to ask.  Do not give up questioning early.  Be that annoying five year old that keeps asking why after someone has explained something to them.  Be sure to keep digging to get to the core reason why someone does or thinks something. 

Bad questions: “Would you…?  Did you…?  Is it…?”

Better way to ask this question would be:

• Why?  When?  How?  
• What’s an example of that?
• Why does this matter to you?
• Talk me through the last time that happened.
• What else have you tried?

Bad question: “Do you think this is a good idea?”

Better way to ask this question would be:

• Walk me through how you currently solve this problem.
• What parts of your current process do you love or hate?
• What other products/processes have you tried before settling on this one?
• Are you actively looking to replace this product/process?  If so, why are you still using this one?  If not, why not?

Bad question: “Would you buy a product which did X?”

Better way to ask this question would be:

• How are you currently solving X?
• How much does it cost you to do this?
• How much time does it take to do this?
• Talk through what happened the last time X came up.
• Have you tried searching for a solution to X?

Bad question: “How much would you pay for this product?”

Better way to ask this question would be:

• How much money does it cost you to solve this?
• How much time does it take to solve this?
• What is the budget you’ve allocated to solve this?

Keep asking “Why?”!  If you’re building something like a notification app that lets a teacher know if a parent’s child is going to be late or absent and you’ve got built out all these cool buttons and it is all automated, but the confirmation messages are cold and robotic the app may not get used.  To solve that problem say you add a custom message option that the parents can then use, but it still feels robotic and cold the app may not get used again.  Then you also add the teacher’s picture to each page of the app.  Asking why parents aren’t using the app or only weird parts of it would bring to the surface that you can use automation, but instead of, “Your message has been received.” you could change it to, “Thank you Jane.  We’ve let Mrs. Periwinkle know that your child be will running late.”  This way it doesn’t feel like their message goes into a black box or the ether of the internet never to be seen.

The following is a list of potential questions you could ask during your user interview.  Post the interview you’ll want to list each of these in the column headings of a spreadsheet and then each row is a user’s name and then their response would go to the right and underneath each question.

1. What are your big goals and areas of focus right now?
2. What are your big problems right now?
3. What are the implications of that problem?
4. Can you walk me through the last time this problem happened?
5. What makes it so awful?
6. How are you currently dealing with this problem?
7. How are you currently coping without a solution?
8. Why haven’t you been able to find a solution to this already?
9. What alternative solutions have you tried?
10. How might you fit a solution into your workflow?
11. How much time does it take to solve this?
12. What is the budget you’ve allocated to solve this?
13. Can you think of anything else that I should ask you about this problem?
14. Is there anyone else you would recommend I speak to about this problem?

Asking the right questions that don’t lead your respondent to biased responses that don’t reflect reality will help you develop and solve real problems that will generate revenue and increase adoption and use.  Worst case scenario is you’ve validated some problem and you go off and build a solution, but then after you realize they never had a budget to solve it in the first place.

User Interviews

User interviews are where you will actually sit down with or collect information from people to see if your solution to a problem is viable or not.  Expect to block out 30 minutes to an hour of time where you’ll be actively listening. When finished you will need time to analyze what information was gathered both quantitatively and qualitatively.  When trying to identify who you want to interview you need to know who to include, but also who to exclude. Using a screening questionnaire will help guide you in this endeavor. Google forms is a great way to create your screening questionnaire.  Below is a brief example of what that could look like.

Having a monetary incentive is a great way to get qualified participants.  Your screening questionnaire will also have filtering questions in it so that you know the submitted participants would be useful in your research.  Once it is created you’ll want to post it online to find your candidates. Some places you can go to place them are:

• Craigslist
• Social Networks
• Community Groups (gmail forum, productivity forum, etc)
• Student Groups
• Personal Networks

User Interview Structure

Typically your format will be as follows:

• Introduction
• Warm up/context questions
• Main body questions
• Follow-up questions
• Debrief/Wrap-up

In your introduction be SURE to state something along the lines of, “Hey just so know I am not the creator of this product and so please be as honest as possible because you won’t offend me on any feedback given.”  This will make them feel more comfortable and honest about what they have to say. Again we’re legitimately trying to solve a problem, not get confirmation bias from friends and family and participants who think getting paid would be in some way tied to painting the products, features, solutions in a positive light.

When you’re conducting the interview it is ideal to pair up.  This allows one person to ask the questions and guide the interviewee through the interview while the second person is taking notes.  If possible try to record the session whether that be audio or video with the interviewee’s permission of course. This helps to see if you had any leading questions and discount the responses if so.  It also helps to see if you missed anything outside of the notes you took during the interview.

Interview Analysis

After the interview is completed you’re going to want to clean up your notes.  Be sure to pull out key insights, key quotes, and top learning moments. It is helpful to create a spreadsheet to organize your thoughts and user responses where the column headings have key questions you wanted to ask in your interview and the rows are the responses of each user with each cell containing the user’s response to the column heading question.  If at all possible you’re going to want to distill commonalities and themes and see if there are any common relationships via affinity mapping or open/closed sorting.

User Research

You are not your user.  You are not your user. No that is not a typo.  I typed that twice on purpose. We do user research because we are trying to identify and address our own biases and misconceptions, which could otherwise be detrimental to the success of the product in terms of sales and or adoption and use.  We want to make sure that we are actually solving a real problem in people’s lives.

“We tend to project our own rationalizations and beliefs onto the
actions and beliefs of others.”

-The Psychology of Everyday Things

As a product owner we want to be sure we’re solving actual problems and not what
we think the market might need.  So why user research? It helps us understand
some of the following things:

• User insights: What problems, needs, and motivations do users have?
• Prioritization: How important are these problems?
• Usability: Do people understand my product?  Can people use my product?

The Process

User research generally involves the following steps:

1. Objectives: What are we trying to learn about our users?
2. Hypotheses: What are our assumptions?
3. Methods: How are we tactfully going to learn about users?
1. a. I.e. user interviews, focus groups, surveys
4. Research: Gather information from our users.
5. Synthesis: Understand and generate insights from the information gathered.

You don’t want to do the “mom test”, i.e. ask someone if it is a good idea you have that solves a problem if they are close to you, biased, or have a stake in you.  Asking leading questions won’t actually give you the result you’re looking for and you won’t be able to tell if you are actually solving a problem or not that has value in the marketplace in terms of solving a real problem that gets used and of course brings in revenue for the business owner(s).

A bad example would be as follows.

User research is important to creating worthwhile products and features for products that will actually solve the needs of your current and future customers.  Neglecting these steps will result in low to no sales to begin with, blown budgets, frustration at lack of user engagement, and repeat or upsale purchases. So neglect them at your own risk!

Working with Cross-Functional Teams: Design and Others

Designers

Designers can have strong opinion and are going to want to exercise their creativity
and so it is important to let go of your personal bias.  Prior to kicking off a meeting
and to ensure that designers don’t feel slighted about designs they make not being
implemented make sure you research your users, their needs, and what their core
problems are.  Once you’ve prioritized which needs need to be worked on during the
kick off meeting you’d show the design brief and here is the full context of the user
need/problem we’re trying to solve right now. This will give the designer a great
starting point and result in more relevant design coming to the forefront.

Sales/Customer or Tech Support/Marketing, etc

When giving an update/general report to other teams or upper management you
really want to summarize and be concise with the information you’re providing them.
 Say during a weekly update, summarize everything in bullet points for what
happened in the past week, what the team is working on, and perhaps a quick metrics
update and what is coming up in the work pipeline.  Also be sure to have regularly
scheduled meetings with the key stakeholders within the cross functional teams.
This allows a quick check in for both ends and can just be verbal too if needed.

Questions I ask to get up to speed on a new scrum team as a product owner or a scrum master Part 2

Welcome to part two of this series on questions I’d ask during hour one of meeting with the team to get up to speed on what they’re working on and how I could help or see where I know I could help.

1.      What is a typical distribution of story sizes in your sprint backlogs?  

This one tries to figure out, where the commitment sweet spot of the team is, based on the sprint backlog composition. To my observation, teams often work in a more successful way, when a sprint backlog comprises of one or two larger user stories, some medium sized stories and a few small ones.  Clearly actionable stories to start the sprint should always be ready with others that can be moved up to be worked when the first ones are finished.

2.      Are you re-estimating user stories at the end of a sprint? 

If so, under which circumstances are you doing so? That should always be done if a user stories turns out to be way off its original estimation.  The PO should update the stakeholder accordingly so they’re aware of the changes/delays in the project.

3.      What was your velocity of the last three sprints?  

The team should know its velocity, how could it otherwise possibly improve?  If you can’t measure it, you can’t improve it.

4.      How many user stories are typically not finished within a sprint and for what reasons?  

If the team is bullish and picked more user stories than it could probably handled at the beginning of the sprint, so be it—nothing to worry about. Also, there are other incidents that might negatively affect the team’s actual velocity, e.g. sick leave or a critical bug a few days into the sprint. If the team, however, is regularly leaving user stories on the board because estimations were wrong, this is a sign for concern that needs be resolved asap or at the very least the next backlog grooming session.

5.      Are you changing user stories once they become an item of a sprint backlog? 

If so, under what circumstances are you changing the user stories? Making them smaller if the team runs into a problem is certainly not great, but acceptable—if the user story in its reduced form still delivers value. Making it larger after the sprint planning is, however, not acceptable.  If they’re being made larger perhaps breaking it out into two stories would be better.

6.      What are the obstacles the team is facing today?  

If they’re systematic what ideas/tools/additional personnel brought on could be done to remove them?

7.      What are the dependencies on other teams?  

And if there are dependencies, are you waiting for other teams to complete their tasks?  

8.      Define and discuss at least two or three key team goals for the project.  

Some of the answers may seem obvious, i.e. meet our deadline within our budget, but this discussion can often bring out other goals which are not obvious to the team initially. It can help the SM or PO understand team motivations and dynamics.

9.      What are key success factors (KPI’s) to achieve our team goals?  

Defining and discussing key success factors, i.e. minimizing the impact of dependencies, can help identify project-level impediments, risks, and issues which the Scrum Master can begin to address. It also is a good benchmark to review and update as the project progresses.

10.  What do team members hope to achieve with this project?  

I like to get a sense of people’s personal goals for the project in addition to the team goals we will establish collaboratively. Having this information can help keep people motivated over the course of the project. Some people may want to learn new technologies, be part of a high-performance agile team, or have other goals.

11.  What type of work environment do we want to create on this project?  

This question can stimulate good discussion about how team members want to interact with each other to achieve the project goals. Often the discussion centers on trust, communication, collaboration, and respect, but it’s good to make sure there is some agreement (or an acceptance of differences) by the team about what is important.  If there are differences that need mending then a group setting may not be the best place to resolve said differences. 

12.  What can we do as a team to make sure that we support each other to achieve our team goals?  

This question can help the Scrum master understand how team members understand the importance of making commitments as a team rather than as individuals. It can also help the team establish informal agreements about the need for everyone to support each other, to take on roles outside their specialty, and trust their team when they need to ask for help.  Often there are senior or tenured people on a team mixed with newcomers that would love some mentorship and coaching.  Making this a regular goal helps elevate team cohesion, morale, and job satisfaction.

13.  What should we do when we are not achieving our goals or not supporting each other?  

Obviously, this can be addressed in a retrospective, but having the discussion early can be helpful to understand how team members perceive how these situations should be handled. It can help establish the need for open and honest communication built on trust.  Adding an impediment column to your workflow might also help the team feel and see that though work has occurred that just because it is stalled isn’t their fault.  Sometimes defects just need time outside of the team to be resolved and having that column could help in your workflow and planning.

14.  How should we celebrate success for achieving our goals?  

It’s important for the team to visualize and expect success. This discussion can help the team discuss rewards that are meaningful and keep them focused on realizing those rewards.  While monetary rewards can be nice, if you worked any amount of time you know other types of recognition often are more valuable intrinsically to a team and its members.

15.  Who are the domain experts outside of our team?

This is useful to establish who to talk to when a story comes to an impasse, but perhaps an official designee for resolving said impasse hasn’t been identified who could resolve the issue.

Scrum Backlog Grooming and why

In purist scrum planning you can plan a sprint, which is a time boxed event of development/work of anywhere from a week to a month, during what is called a Sprint Planning Meeting.  These sprint planning meetings can last up to 8 hours for a one month time box sprint.  That is a long time to plan all the work that is going to be done in the next month.  Not to mention that perhaps things will change during that time that will make some or much of the planning irrelevant as time goes on.  This is possible in situations where consistent work hasn’t already been done with definitions of done completed and estimates by the dev team provided.  Where there has already occurred plenty of work that can be looked back on and estimated accurately going forward you don’t necessarily run the risk of planning that is useless since you already have a good idea of what it is going to take to do the work.

All that said how can you potentially cut down on the amount of time a sprint planning meeting will take?  Answer: Backlog grooming sessions.  Ideally these sessions happen regularly so that the sprint planning meetings are both shorter and more effective.  These sessions are meant to improve the product backlog.  In it you can:

• Write/update user stories
• Decompose a story into several stories that are too big to realistically be done in a time box sprint
• Devs often love as much detail as possible to accomplish a story and so adding more definition or rewriting poorly constructed stories is useful
• With the dev team estimate the backlog items that haven’t already received estimates
• Add acceptance criteria and flush out the definition of done
• Assess deeper (lower down on your product backlog) stories/backlog items and do long term planning with domain experts or just when those items can be done.

Yes it is important to stay focused on the current workload, but don’t get too hung up on not looking beyond the current sprint if needed.  Just because you’re on schedule for the next release doesn’t mean you can’t proactively look to the future and see what additional research, domain experts, or technical hurdles may need to be addressed now before getting to the future.

So good things to do during a grooming session:

Have a stated goal.  Everybody hates meetings that drone on and don’t actually accomplish anything.  Devs especially hate this since they’re up against what they’re already working on and being in a pointless meeting just takes them away from the work they need to get done now.

At the bare minimum hold your grooming session a few days before the sprint planning so people are familiar with the backlog and can hit it hard during the actual planning session.

If you have stakeholders present, limit them to just a handful.  Don’t have 10-20 people in the meeting because they often don’t know the rules of scrum, will make things chaotic, plus people check out when there are too many people in a meeting anyways and input by default is minimized by the sheer size of the group whereas smaller groups can have more people contribute.  Go figure.

Implementing regular backlog grooming meetings helps sprint planning meetings go better and more efficient, familiarizes the team with what is on the product backlog, which then leads the team to committing easier to sprint goals/commitments.