Expert Panel Recap — User Research: Stop Guessing and Start Talking to Real People

Be honest. How often have you planned and built a digital product without once talking to the people who will actually need to interact with it? If you’re like most interactive teams out there, you probably do this all the time. The problem is, you are not your user. Anything you decide for your users is just a guess and guesses — even educated ones — have a high likelihood of being wrong. That’s fine if you’re guessing how many jellybeans are in a jar. It’s bad if you’re guessing with decisions that will determine the success or failure of an expensive, long-term website or application project.

The remedy to guessing? User research. It can seem like a costly, time-consuming endeavor up front, but the hours and money you’ll save in the long run will more than make up for it. Take it from Airbnb, whose business success hinged on unplanned interactions with actual users.

User research can take a lot of forms and there’s a learning curve to getting unbiased, objective, usable data. We asked a panel of user research experts to share their tips, tricks, and experience in gathering useful data.

The Expert Panel

Alice Rhee, Senior User Researcher at Mozilla Firefox
Jill Reed, Senior User Researcher at XPO Logistics
Steve Schang, Principal and Owner at Midwood Usability

What We Learned (in a nutshell)

1. Figure out the big picture. What is the problem that needs to be solved?
2. Make your users comfortable and allow them to fail tasks.
3. Know your biases.
4. Turn your research into actionable results.

1. Figure out the big picture. What is the problem that needs to be solved?

I’m a big fan of doing whatever makes sense for a client or team to help them understand who their users are and how they’ll use the products.
-Steve Schang

Learn the why behind whatever is being tested.

Before user research begins, it is imperative to understand what you are testing and its purpose. Ask stakeholders the basic, general questions beforehand so you can understand what interactions are most valuable to the business. If you don’t ask stakeholders big picture questions at the beginning of user research, it will be impossible to determine what successful research results should look like for them.

I’ve learned to listen, ask a lot of questions, and get to, “Why do you want this thing? What is it supposed to solve?”
-Jill Reed

Search for a larger issue if stakeholders ask for a small part of the app to be tested.

Often, stakeholders will come to user researchers only wanting one small part of their digital product to be tested. While it is important to conduct user research around that specific request in order to get buy-in from those stakeholders, always search for a deeper-seated issue behind it that will add more value to your findings. After you have meaningful user research results in hand, stakeholders will be more easily convinced to continue necessary user research beyond the small amount they initially had in mind.

When I start, I usually do what the client has asked me to do. As we start to see the deliverables and what’s happening it’s easy to say, now we need to do something else.
-Steve Schang

Get stakeholders involved in user research.

There’s no better way to get stakeholders on board than to have them participate in your research. Inviting them to observe and contribute in ways that won’t impact results will make them feel like part of the process. They’ll be able to see and hear for themselves exactly what users need and what gets in their way. It will take almost all of the effort out of selling the value of your research.

I always try to have my stakeholders to buy in and commit to watching and take part in the sessions.
-Alice Rhee

2. Make users comfortable and allow them to fail tasks.

It’s a very human situation when it is you and a participant in a room and if you’re not making them feel comfortable, they aren’t going to give you the data.
-Alice Rhee

Observe and test users in a safe, realistic setting.

Using a site or app in front of someone else, especially a stranger, can be nerve-wracking. When users are under stress, the data they create suffers, so it’s important to put them at ease and create an atmosphere as familiar to them as possible. Understanding the environment where users typically complete tasks can also help you make decisions as you move forward with planning and development. Is the area full of natural light, washing out their screen? Are they using a mobile device and often on the go? Are they using horribly outdated equipment and browsers? All of these things are important to note and incorporate in your findings. If going to the users’ home, place of work, or wherever they’re using the product is a possibility, consider that option. If not, create a space as close to that environment as possible where users will feel comfortable and can carry out their tasks as normally as possible or consider remote testing.

We try to go to those people’s particular location so they feel at ease and use their products so we can feel their life and realize you are not your user.
-Alice Rhee

Introduce users to the research process.

Don’t just throw users into the middle of a digital product. Remember, having a stranger watch you complete tasks is stressful. If users are flustered, it can take them longer to get things done, skewing your data. Explain that there is no right or wrong way to do what you’re asking them to. You’re only scrutinizing the digital product, not them. Start them out slowly with simpler tasks and then move to more complicated requests.

Sometimes people know that they’re failing and they start shutting down really quickly, so I give them a very easy task upfront that they won’t fail.
-Steve Schang

Allow users to fail, but don’t let them feel like it’s their fault.

It’s actually a good thing when users fail. That’s the whole point of user research, to see where something isn’t working. The difference, as panelist Jill Reed pointed out, is that the user doesn’t fail the product, the product fails the user. Sometimes stakeholders want to set up a test that is all but impossible to fail because they want to validate their ideas or their product’s design. This is exactly the opposite of why we do user research. While tasks shouldn’t be extremely difficult to complete, they should be possible to fail. Creating the most realistic scenario possible will result in the most realistic, and useful, data.

I always tell users they’re one of the first people to see this, it’s not like everybody knows how to use this, no one has ever used it before.
-Jill Reed

3. Turn your research into actionable results.

The key is turning your findings into insights and then using them in reference to the product.
-Jill Reed

Determine what your research results mean for the product.

You have the data and your findings, so now what? You need to interpret what those results mean through your expert lens. What do your findings mean for the digital product? Where did it fail? Why did it fail? Who did it fail? And, most importantly, how can those failures be fixed? Without this step, your research results cannot be applied to make the site or app better.

If you do research and nobody does anything with it, what’s the point?
-Alice Rhee

Present your results and take-aways in a way that’s easy to digest.

Once you’ve determined the path forward to fix the product’s shortcomings, you need to prepare to present your results. Panelist Jill Reed presents her findings and recommendations in a simple, straightforward PowerPoint, taking care to only highlight key data and ideas. Don’t get bogged down in the details of your research. Instead focus on what actionable takeaways you’ve found and how they can be carried out. Keep presentations and reports as short as possible or you will lose your stakeholders’ attention.

We used to do 30-page reports that everyone listened to and never saw again. Now people want the result quicker, they want them to be actionable.
-Steve Schang

Follow up with any implemented recommendations.

Your job is not done yet! Follow up with the changes you recommended and test them to see if they actually fix the problem. This is an iterative process — fix and test until you’re sure you’ve found the right solution.

I like doing rounds of research so you can see if your recommendations fixed the issues.
-Alice Rhee

4. Know your biases.

As a researcher you need to be able to see the biases that you’re facing. The data you’re going to collect is only as good as the moderator.
-Alice Rhee

Listen to yourself moderate to discover your biases.

Asking someone to identify their own biases is nearly impossible. Biases are slowly built into our consciousness and are difficult to recognize in ourselves without help. Do yourself a favor and listen to yourself moderating a user research session. Listen for any leading words or phrases you may have used. Pay attention to your voice inflection. Note how you responded to any actions that users took. Once you have yourself on record, it’s easier to look back as a third-party and take notes on where you need to improve.

Sit there and listen back to yourself facilitating the session and critique yourself. That will reinforce in your mind what to avoid and fix next time.
-Steve Schang

Find a mentor to teach you how to navigate your biases.

Even if you know your biases, consciously fixing them while facilitating a research session is hard work. Find someone who has more experience in the user research field and is willing to sit in on a research session to help you remember to keep your biases in check. Work with your mentor to create ways to correct any biased tendencies you may have. We’ve all been there and we can all work to be less biased in our research.

Something that’s so important is to find a mentor, find somebody who knows what to do and can help point biases out.
-Alice Rhee

Don’t answer user questions in a leading way.

Users instinctively want to get tasks “right.” They often ask for help or try to get you to tell them what the “correct answer.” Instead of telling them whether they are “right” or “wrong,” ask them questions to find out more about what they’re thinking. Ask: “What would you expect that to do?” or “If I weren’t sitting here, would you think you’ve completed this task?” This way you are learning from their responses and avoiding tainting your research.

If users are having trouble and turn around for help, turn it into a question like “What do you expect from that?” or “what do you want that to do?”
-Jill Reed

So Get Out There and Talk to Real People

Every time you moderate another user research session your skills will grow. You’ll learn by doing what works best and what methods you prefer to use when. Encourage your team and your leadership to invest in user research in all stages of the planning and building process. It will protect your project from risk of failure and maximize your investment in creating the site or app in the first place.

Related Reading
Let Your Users Show You What’s Not Working
User Testing Made Me a Better Designer

About truematter

Our team has been doing the real work of user experience since the earliest days of the commercial web. We’re out to make your digital products a whole lot better.

User research can actually be fun. And the best part? There are no wrong answers.

Authors: Isabelle Carroll
Graphic: @djosephmachado
Editor: @baileysendsword