An Introduction to Remote Usability Testing

What is “Remote Usability Testing”? When would you use it, and what are the potential pitfalls? In this article we’ll cover all these questions and more. Prepare to power-up your design decision-making by running screen-sharing sessions with real-end users.

Different Types of User Research

User research is something that successful companies do on a regular basis, enabling them to make design decisions that improve usability and satisfy user needs. Google, Twitter and Facebook all have have user research teams.

As a beginner, user research can be bewildering–there are so many different types that it’s hard to know which to do when. This table helps clarify things, as it divides user research into two dimensions:

On one dimension you have moderated and unmoderated. Moderated is where you interview and observe the user live in real-time (you can either do this in-person or remotely). Unmoderated is where you use an online tool to set tasks that the user then follows, and things get recorded, such as task completion rate and user feedback.

On the other dimension you have qualitative and quantitative. Qualitative research is about deep-diving into users’ minds and developing an understanding of how they think and feel about the job or problem you’re trying to solve for them. Quantitative is about measurement, often employing scientific methods and statistical analysis.

A Broader Perspective

It’s vital to understand that it’s a bad idea to focus on just one quadrant from the table above–doing so makes you shortsighted. User research is like shining a torch in a darkened room, only illuminating one spot at a time. You need to move the torch around to get the “bigger picture”. A common mistake made by modern start-ups is focusing purely on AB Testing or Multivariate testing; tools which focus on just one quadrant. With that approach, you end up with lots of “sciencey” data, but a relatively shallow understanding of what motivates your users and how to design things to really meet their needs and engage them deeply as paying customers.

Moderated, qualitative user research (the top left quadrant from the diagram) is probably the simplest, easiest and most flexible form of user research. Back in the 1990s, this had to be carried out in a lab, with analogue cameras and VHS recorders in the back room capturing each camera angle. Some agencies still provide labs (pictured below) but these days it’s not really necessary; you can do it all on a low-end laptop using completely free software, and the participant can be anywhere in the world. This approach is called “remote usability testing”, and is the subject of this article.

Your Remote Usability Testing Checklist

Here’s what you’ll need for a single round of remote usability testing:

• Budget to pay your participants (£10-20 each).
• A few hours to set up the recruitment questionnaire a using a tool like Typeform or Google Forms and then promoting the questionnaire via email or social media. This is best done about ten days in advance. 
• A few hours to screen the respondents and invite the good ones to sign up for an interview using a scheduling tool like. powwowapp.com or calendly.com). This is best done about five days in advance. 
• A day to run the research. As a rough guide, you can do up to 14 × 20 minute sessions or 8 × 40 minute sessions. Use Google Hangouts for screen-sharing and Quicktime for screen-recording.
• A day to do the analysis and write up the findings.

This may seem like a lot of time to invest, but the findings will give you a massive payload of design insights that you can feed into your product roadmap or AB testing backlog.

Pros and Cons of Remote Usability Testing

The Pros

One of the biggest benefits of remote usability testing is that it’s very flexible; you can can change the plan and make up questions up as you go along. For example, if a participant stumbles upon on an epic flaw in your UX design that dwarves the thing you were going to do the research on, you can ditch the rest of the interview and talk about that instead. Or if the participant starts talking about a new competitor you’ve never heard of, it’s no big deal to start talking about that in-depth for a while instead of following the script. This degree of flexibility isn’t possible in a survey or an AB test.

On the whole it’s a very easy method to set up. You set up a screen-sharing session and then record your screen while you give the user activities to do and take notes as they go (see screenshot above). 

“It’s not rocket surgery” – Steve Krug 

So bear in mind, this sort of research is not about strict measurement, statistics or databases. There’s no point in having “science envy”–it’s about building a human-to-human connection with your users and understanding how they think.

The Cons

Remote usability testing has some downsides that you should be aware of. At the time of writing, you can only do it effectively on laptops and desktop computers. Mobile devices aren’t yet powerful enough for reliable screen-sharing, VOIP and recording at the same time, let alone running an app simultaneously. There are some work-arounds but they tend to be complicated and hacky. This will probably change in the next few years but for now, handsets and tablets are out of the picture.

Another downside is that it’s a lot like face-to-face usability testing but isn’t as intimate. If you’re lucky enough to work in a city where your users are close at-hand then you’ll achieve much better rapport and a better quality interview if you meet up with them and do the session face-to-face. For this reason, some professional researchers see remote usability testing as a “back up” approach when you can’t reach users face-to-face.

Conclusion

This article should have given you a good idea of what remote usability testing is, how much investment is needed and what you can get out of it. Stay tuned for the next article: “How to Conduct Remote Usability Testing”  for a detailed guide on the nuts and bolts of getting it done.