When conducting usability studies or field studies, it’s a great idea to ask lots of open-ended questions. Typically, researchers ask questions before, during, and after research sessions. It’s easy to focus on what you want to know rather than on how you ask, but the way you ask questions matters a lot in terms of what and how much you can discover. You can learn unexpected and important things with this easy technique.
Open-ended questions are questions that allow someone to give a free-form answer.
Closed-ended questions can be answered with “Yes” or “No,” or they have a limited set of possible answers (such as: A, B, C, or All of the Above).
Closed-ended questions are often good for surveys, because you get higher response rates when users don’t have to type so much. Also, answers to closed-ended questions can easily be analyzed statistically, which is what you usually want to do with survey data.
However, in one-on-one usability testing, you want to get richer data than what’s provided from simple yes/no answers. If you test with 5 users, it’s not interesting to report that, say, 60% of users answered “yes” to a certain question. No statistical significance, whatsoever. If you can get users to talk in depth about a question, however, you can absolutely derive valid information from 5 users. Not statistical insights, but qualitative insights.
How to Ask Open-Ended Questions
Are you satisfied?
How satisfied or dissatisfied are you with this process?
Did it act as you expected?
What would (did) you expect to happen when you ... ?
Did you find it?
Before a task:
After a task:
Do you think you would use this?
How would this fit into your work?
How might this change the way you do that today?
Does that work for you?
What do you think about that?
Have you done this before?
What kinds of questions or difficulties have you had when doing this in the past?
What happened when you did this before?
Please describe your level of experience with …
Is this easy to use?
What’s most confusing or annoying about … ?
What worked well for you?
Did you know … ?
How do you know ... ?
Do you normally … ?
How do you normally ... ?
Did you see that?
What just happened?
What was that?
Do you like this?
What would you most want to change about … ?
Which things did you like the best about … ?
Did you expect this kind of information to be in there?
Before a task:
After a task:
Why Asking Open-Ended Questions Is Important
The most important benefit of open-ended questions is that they allow you to find more than you anticipate: people may share motivations that you didn’t expect and mention behaviors and concerns that you knew nothing about. When you ask people to explain things to you, they often reveal surprising mental models, problem-solving strategies, hopes, fears, and much more.
Closed-ended questions stop the conversation and eliminate surprises: What you expect is what you get. (Choose your favorite ice cream: vanilla, strawberry, or chocolate.) When you ask closed-ended questions, you may accidentally limit someone’s answers to only the things you believe to be true. Worse, closed-ended questions can bias people into giving a certain response. Answers that you suggest can reveal what you are looking for, so people may be directly or indirectly influenced by the questions. Don’t ask, “Does this make sense?” Ask, “How does this work?” and listen closely to discover how well the design communicates its function. Note users’ word choices, because it might help to use their terms in the interface.
Start open questions with “how” or with words that begin with “w,” such as “what,” “when,” “where,” “which,” and “who.”
Don’t start questions with “was” (an exception to the “w” tip) or other forms of the verbs “to be” and “to do.”
In general, avoid “why” questions, because human nature leads people to make up a rational reason even when they don’t have one. We normally ask “why” only about ratings, to tease out more open-ended feedback. Say “Please tell me more about that,” instead.
Aim to collect stories instead of one- or two-word answers.
Even when you must ask closed-ended questions, you can ask an open-ended question at the end, such as, “What else would you like to say about that?”
Adding Other __________ to a set of multiple-choice answers is also a good way to get open-ended feedback.
When to Ask Open-Ended Questions
- In a screening questionnaire, when recruiting participants for a usability study (for example, “How often do you shop online?”)
- While conducting design research, such as on
- Which problems to solve
- What kind of solution to provide
- Who to design for
- For exploratory studies, such as
- During the initial development of a closed-ended survey instrument: To derive the list of response categories for a closed-ended question, you can start by asking a corresponding open-ended question of a smaller number of people.
When To Ask Closed-Ended Questions
- In quantitative usability studies, where you are measuring time on task and error rates, and you need to compare results among users
- In surveys where you expect many (1000+) respondents
- When collecting data that must be measured carefully over time, for example with repeated (identical) research efforts
- When the set of possible answers is strictly limited for some reason
- After you have done enough qualitative research that you have excellent multiple-choice questions that cover most of the cases
Whenever possible, it’s best to ask open-ended questions so you can find out more than you can anticipate. Test your questions by trying to answer them with yes or no, and rewrite those to find out more about how and what. In some cases, you won’t be able to accommodate free-form or write-in answers, though, and then it is necessary to limit the possibilities.
Think on a deep level (get to a place where you can completely focus on something, then focus on this). Ask your self "What's an open ended question? You will find an open ended question is the opposite of a closed ended question, meaning a question someone can't answer yes or no only to. Seriously ingrain it into your mind so you never have to ever ask that again! Understanding is the key.
Start by planning your day or week (what ever is easiest for you) then, pick out each situation you might get into where you need to ask these questions. Making it personal will help you learn it on a deeper level.
Take the easiest situations and write out a short script of how a conversation would typically go (with all the positive and negative parts). All the good and the bad.
After you've scripted out a short or long conversation (again, what ever is easier), rewrite it so you can fit in some OPEN ENDED QUESTIONS! It's best to replace some response with one or two, because that will keep some balance to the equation. Too much info can overwhelm anyone so know them before you try to test your new "script". Even if it's just writing it out, when you're successful at implementing an open ended question on paper, you might internalize that success and actually do that when you have a conversation with them.
Practice using open ended questions. The more you script it out, the easier it will be to practice. If you say the wrong thing or accidentally use a closed ended question and sabotage your self out of habit, just take note of it. Then when you try again to plan it ahead of time you can remember "well Susie has the best smile so I need to remember to use bigger words to distract me from that". Or "the boss is so smart, I know if I get too close to him I need to use words that distract me from him or else I will stutter for sure". Some pit falls can be totally avoided if you pay attention, note them, keep trying and be persistent. Until you can overcome the pitfalls and or use open ended questions automatically.
Apply your trial and error on the many situations that matter just a little until you have it as a habit. Then when you have done it enough to the point you can do it easily, apply this new knowledge on paper with essays or say in the toast masters (where you can get some really good training). The more practice the easier the task.