In the previous article we talked about what “Remote Usability Testing” is, and covered the pros and cons. In this article I’ll give you some practical advice for carrying out your own remote usability sessions.

Selecting the Right Software

There are two pieces of software you need for remote usability testing:

1. a screen-recording app
2. a screen-sharing app

For screen-recording on OS X, you can use Quicktime. Just pick the Screen Recording option and you’re good to go. If you want the means to edit the recording, try Screenflow (OS X) or Camtasia (Windows).

For Screen-sharing, you can’t go far wrong with Google Hangouts. It’s free, most users trust the Google brand, it’s easy to set up, and if you need anyone else to dial in (project stakeholders), up to ten people can connect, just ask them to mute their cameras and their microphones.

There are products that combine screen-sharing and screen-recording. Validately’s “Moderated Testing” feature is probably the best, but it requires a professional plan, which currently costs $199 per month.

Writing Your Recruitment Specification

Your recruitment specification is the list of attributes you want your users to have (pictured below). For example, if you’re working on a cycling app, you’ll want to find cycling enthusiasts, and exclude any time-wasters.

It’s often a good idea to avoid getting too specific about demographic details. For example, if age range and gender are not at the core of what defines your users, then don’t set a specific quota on those characteristics.

Users are often better segmented by behaviours and attitudes anyway, as this is more likely to affect their propensity to use or buy your product. For example, how much cycling does the user do? How many bikes do they own? What do they spend on sport app subscriptions and apps in general each month?

Setting up a Screener Questionnaire

In this article we’ll assume you’re carrying out the recruitment yourself (aka “self service”) and you have a limited budget. If you have money to spend (£50-100/head finders fee) you can hire a participant recruitment agency, give them your recruitment spec and let them take care of it all. If you don’t, it’s straightforward to do the recruitment yourself. You can use a free tool like Typeform or Google Forms to set up an online recruitment questionnaire. The idea is that you then take that URL and promote it on social media, in forums, and other places your users can be found.

In your questionnaire it’s good to use a mix of structured questions (e.g. radio buttons) where you can categorise users and filter them; and unstructured questions (e.g. text fields) where you can read their responses and hand-pick individuals who seem to be good communicators with something interesting to say.

Don’t forget to ask questions about their availability to participate on the given day(s). Remember, if you offer a decent incentive (e.g. a £15 Amazon voucher for twenty minute session) people are more likely to find the time.

One of the most frustrating things about user research is encountering time wasters. People who want the payment you’re offering, but lie in the questionnaire, or people who fit the spec but turn out to be monosyllabic or uncooperative. You have to accept that about 20% of your recruits will be “duds”. Even so, it’s a good idea to take steps to filter them out.

Here are some tips for filtering out dud participants:

• Check if they already use your service: If you’re recruiting for people who are already using a live service, ask for their username and check the database for the duration they’ve been a member. Screen them out if they “coincidentally” joined on the day they filled in the questionnaire.
• Hide your intent through multiple choice questions: e.g. instead of asking if they are a cycling enthusiast, ask them which forms of exercise they most enjoy, where cycling is just one of the options. This way a determined liar will not know which answer to put.
• Ask questions to evaluate their articulateness: it’s wise to ask open-ended free text questions. If they’re able to wax lyrical about the research theme, then they’re going to be valuable to interview.
• Filter out non-spenders: if you’re working on a premium service, you need users who actually have money to spend and willingness to spend it on the problem your app is trying to help them solve. In this case, it’s reasonable to filter by income level and monthly spend on related things (e.g. app subscriptions).

Finding Users to Fill in Your Questionnaire

If you have a live product, then chances are you already have a decent marketing email list that you can send a mailshot to. Your mailshot should be brief and have a single, large call-to-action, asking users to apply. You can provide further details on the first page of the questionnaire.

If you have traffic but no email list, then another way to get users is to put a lightbox or some similar call-to-action on your site. You can do this using Ethnio (which offers a bunch of other helpful recruitment features), Qualaroo or you can ask your dev team to build one and send the email addresses to Mailchimp, Campaign Monitor, or whichever email marketing tool you use.

If you have a decent social media following, Facebook or Twitter can work well. If you’re just starting out and have none of the above, then Craigslist and Gumtree can work.

If you use more than one approach, make sure you direct them all to the same questionnaire. Some questionnaire platforms allow you to add URL parameters or filter by URL so you can work out which channel works best.

Screening Users and Booking Interviews

Once you have a decent group of respondents, you can start going through the responses to pick out the best ones. This is called “screening”.

Start with your structured questions, and filter out the blatantly irrelevant candidates; this is easy to do if you export the responses to Excel and use the Data › filter feature. From there, you can read through their free-text responses. Pick people who have something to say and sound actively interested in the problem or job your service is trying to help them with.

Once you’ve made your selection, send them an email saying their application was successful and that they’re invited to participate in the research. Use a scheduling tool like usepowwow.com, which allows you to send them a sign-up link that offers offer a list of session times they can book (pictured below). It’s also a good idea to send them some information explaining how to set-up Google Hangouts.

Facilitating the Session

Let’s cover some rules for the session itself.

Prepare Your Discussion Guide in Advance

You should prepare for usability test sessions by writing a discussion guide containing the user activities and the questions you want to ask. Don’t treat it as a script; treat it as a guide.

A good way to start a session is to begin with an “open exploration” task. For example, with our fictional cycling app, you might say “Imagine a friend recommended this app to you but you don’t know anything about it. Install it and try it out to find out whether you think it’s any good”. You can then let them explore the app via their own chosen path. This approach is more likely to get them into a self-motivated flow of interaction than a series of small micromanaged tasks like “Please fill in the registration form then stop”. When the participant finishes the initial open exploration task, you can give them a few other top-up activities to cover off anything they didn’t cover on their own accord.

Take Notes During the Session

There’s no “right way” to take notes, it depends what you’re best at. If you have terrible hand-writing and you’re a great typist then taking notes directly onto your laptop makes sense (you’ll need a dual monitor set-up so you can see the user’s session on one screen and your notes on the other). If you hate typing then you can take notes on paper. Some researchers swear by gadgets like the Livescribe pen. Whatever your chosen medium, it’s useful to write a time-stamp when something interesting happens. Later, when fast-forwarding through the video, you can just keep an eye out for the time in the corner of the screen.

Have Colleagues Observe Live

When project stakeholders observe live sessions, the research tends to be much more successful as they’ve seen the findings with their own eyes and understand the true severity of the issues observed.

Stakeholders can observe either by connecting remotely or by sitting in the room with you as it happens. Be sure to tell users that this is happening as a courtesy. If your colleagues connect via Google Hangouts, make sure they mute their microphone and camera. If you’re relaxed and friendly, the user will soon forget about them.

How to Run an Effective Interview

Try to see the interview as a lovely chat where you get to really dig into a user’s point of view and empathise with them. There are lots of little mistakes you can make, and you really only get past them by practicing your interview technique. Here are some tips:

Paying Users

After the session is successfully completed, you can send the participant online payment. Amazon vouchers are handy because it’s easy to bulk send vouchers to a group of people all at once. Your employer should pay for this, or if you’re freelance you should agree in advance to charge this to them as a “pass on cost”.

Analysis and Recommendations

Analysis is the hardest part of user research. It involves tallying up the observations and then making design recommendations. For example, if 8/10 users struggle to find the registration button, it’s sensible to recommend for it to be much more prominent, and to appear on all the logged out pages.

Most observations aren’t that clear cut, and it takes a lot of experience to work out what kind of design changes are needed, and whether or not some observations can be “taken with a pinch of salt”. One of the most common mistakes in user research is for senior management to try to cut costs by limiting the scope of the changes to minor copy tweaks or other superficial things. This is short term thinking: ignoring research findings and launching features that are not usable or useful means they will fail in the long run. To avoid this, make sure your research has buy-in from senior stakeholders right from the outset, so they can force large design changes to happen when necessary. Sometimes a UI that tests very poorly needs just to be torn up and started again.

Creating a Findings Log

A findings log is a table where you list your observations as rows, and their occurrence by user as columns. It gives you a tidy overview of what would otherwise be a few hours worth of video footage or a dozens of pages of transcripts.

Creating a findings log is straightforward but time consuming. As a rule, you should budget roughly as much time for analysis as you spent doing the research. If you test lots of different user types, or lots of different areas of the user experience, then it’ll take much longer–you’ll get a feel for this after you’ve done a few projects.

Here are the steps you need to go through to create a good findings log:

1. Go through your notes for each user. Whenever you see evidence of a usability problem, add it as a row in the findings log and enter the number “1” as a tally for the user with whom you observed it happening.
2. As the table grows, combine similar observations if you can. For example “User struggles to find the registration button on the homepage” and “User complains about not being able to find the registration button while browsing the homepage” can be combined into one.
3. You’ll end up with roughly 20-30 observations if your testing consisted of one group of users looking at a small set of features. Once you’ve compiled this together, you can move onto the recommendations column.
4. For each observation, write a recommendation. It’s reasonable to suggest more than one idea or to split recommendations up into two columns: one for long term and one for short term fixes. If you can’t think of a good recommendation, you can simply say so (“not clear what action is needed”) or you can suggest further research.

Once you’ve got the findings log finished then you’re ready to go.

Finally: Getting Design Changes Actioned

If your project stakeholders didn’t watch the sessions, you’ll need to give a presentation at this point to get buy-in. It’s pretty standard to create a slide deck in which you show screengrabs to depict each finding. If you have time, you may also want to create some highlight clips to demonstrate the key findings (Quicktime has a “split clip” feature which does the job). It’s better all-round if you get the stakeholders to watch the sessions live, or at least watch the recordings–this will make them empathise with the users much more and will make them more willing to make design changes.

The next step is to get agreement on what goes onto the agile backlog or product roadmap This can be hard: sometimes you have to fight for research findings to get actionned. If this is the case, you might want to add a couple of new columns to the findings log table: cost and impact. You can then run a group session with the project stakeholders where you work together to give each observation rough t-shirt sizes for cost and impact (S/M/L/XL). You can then focus on the things that are likely to have the lowest cost and the highest impact (aka the “low hanging fruit”). Once you’ve proven the value of remote usability testing, you’ll find that you’ll have greater influence.

Closing Thoughts

You’ve probably gathered from this article that user research involves fair amount of effort – and this fact is what puts a lot of people off. Don’t let it deter you. Remember, if you design something without user research, it’s far more likely for your product to fail in the marketplace. Even if you consider yourself to be a designer or a developer, user research is something you need to know if you want to deliver great user experiences.

Useful Usability Testing Resources

• 

  Usability Testing Tools for Quick and Early Feedback

  In this post I’m going to explain how you can run usability testing sessions remotely, or in-house, to validate your designs and deliver the best possible...
• 

  Usability Testing Methods: The Benefits of Face-to-Face Tests

  Usability testing doesn’t have to be a long-term, large project; it can be conducted fast and still deliver results that will save developers lots of...