Remote unmoderated testing comes to the rescue! Here’s a practical guide to the method as well as our best tips on dos and don’ts.
What is remote unmoderated testing?
Remote unmoderated usability testing is very similar to face-to-face moderated testing. However, there are 2 main differences:
- You’re not in the same physical location as your research participant
- You’re not moderating the test; the participant is answering questions and completing tasks based on a test script that you’ve created and shared in advance.
You’re probably wondering: “How can I analyse the findings if I’m not there to moderate and observe the test?”
Well, usually unmoderated tests are created using online tools that allow participants to record themselves while they are taking the test. The recordings are then shared with the people who created the test (that’d be you!) for analysis.
So what are the benefits?
The main one is that remote testing allows you to gather user feedback from the comfort of your home. This makes the method especially useful right now, as it allows you to stay true to a user-centred design process during the COVID-19 crisis.
But the method is also useful outside of global pandemics! Here’s why.
It’s faster
You save a lot of time on administrative tasks since the research participants themselves choose when and where to take a test. You no longer need to manage a testing schedule, book meetings, or secure space for testing. You also save a good amount of time on not having to moderate the tests.
And of course, you’ll save yourself the headache of worrying about no-shows and last-minute replacements of test participants who don’t turn up.
Many remote unmoderated testing tools also offer features that will help you further speed up your research process, such as:
- Test templates containing tried and tested questions and tasks,
- Annotation functionality that allows you to easily take notes while watching and analysing your test results, and;
- Collaboration boosting features such as sharing a link to a video with a timestamp so that you can easily share pain points with your teammates.
It’s cheaper
You can also save a lot of money by running remote unmoderated tests!
Doing your own recruitment can be very difficult, cumbersome, and expensive — especially if you need to use external recruitment agencies to access a very specific target audience.
Many remote unmoderated testing tools offer to do all the recruitment for you. Teston, for example, can help you with everything from recruiting your participants, to paying testers their incentive.
Apart from recruitment, remote unmoderated testing also removes the need for you and your company to purchase any testing equipment. Many companies are put off by the idea of usability testing because it often comes at the cost of a test lab: a dedicated meeting room, testing devices, recording hardware and software, an observation room, etc.
With remote unmoderated testing, however, all you need is a computer and access to the internet to set up and analyse a test.
Test with a wider range of people
Moderated testing typically limits the pool of available research participants for two reasons:
- Since you need to be present to moderate the test, testing typically happens during the day when most people are busy working. This means that participants who show up for moderated tests tend to be students, unemployed people, or freelancers who have the freedom to choose their own work hours.
- You’re also limited to testing with people who are able to travel to your testing venue. If your product is used by a wide range of people who live in a distributed geographical area, it can be important to test more widely. User needs often differ from location to location. Some aspects to consider are the participants’ level of tech-savviness, whether they have access to the internet and other digital resources, as well as differences in behavioural patterns.
With remote unmoderated testing tools, you can test with virtually anyone who wants to take a test, since participants can take a test whenever and wherever they like.
More realistic results
Using remote unmoderated testing tools can also yield more realistic and representative results because participants typically take tests in their own homes.
Testing in your office space, or a test lab, can feel very unfamiliar. This can cause participants to feel uncertain and uncomfortable during the test. They are, after all, being put in a new setting, asked to use unfamiliar devices, and observed at every step.
With remote unmoderated testing, participants are using their own devices, in familiar settings. You can even get a glimpse into their everyday lives and use of technology!
For example, when reviewing test results on Teston you’ll find that many people take tests while their children are playing (sometimes noisily) in the background.
We find these results to be extra rich. They give you a more realistic picture of users’ true experience of your service in the context of their real, everyday lives.
Run multiple tests simultaneously
Cheaper and faster results, combined with the fact that you don’t have to be present to moderate the test, mean that you can easily run several tests simultaneously.
This opens up for a world of opportunity:
- Test multiple prototypes against each other to evaluate which designs perform the best for the same target audience and the same use cases.
- Run competitor analyses where you use the same test script on your site as well as multiple competitor sites, to understand which performs the best and why.
- Test your site in different languages. For some products, it’s important to understand and compare cultural and local differences in user needs. Teston is especially well-suited to this, as it’s a localised user testing tool that allows you to target specific languages and locations.
Are there any disadvantages?
Like with any method, remote unmoderated testing has its advantages and disadvantages. It’s important to understand both sides so that you know when to use this method.
No follow-up questions
Digging deeper into a particular response can be really important to fully understand your participant’s experience. Unfortunately, asking follow-up questions on the fly is not possible in unmoderated testing.
This means that results from unmoderated testing are limited to the quality of the test script. If you didn’t include a particular question in your script, chances are that your participants won’t answer it. This can sometimes leave you with more questions than answers.
Pro tip: Add follow up questions to your test script. This will help make sure that you get deeper insights from your results. For example, after a participant performs a task, you could ask:
- Did [insert task/feature] meet your expectations or was there anything surprising?
- What did you think of [insert task/feature] overall?
- What did you think of the content?
- Was there anything confusing or was everything clear to you?
No way to prompt testers to think aloud
While many unmoderated testing tools have built-in ways to remind their test participants to think out loud, you’ll still come across the less chatty type, that’ll forget to speak their thoughts out loud now and then.
In moderated testing, you can easily prompt participants by asking them what they’re thinking, or what they’re looking at. In unmoderated testing, however, you’ll need to rely on the participant to remember to think aloud on their own.
Pro tip: Tools like Teston continuously remind participants to think aloud, but it never hurts to add further reminders in your questions and tasks.
This is especially important to keep in mind if you’re asking your participants to complete an important task in your product, or if the participants are about to complete a longer task (that’s where they’re more likely to forget to think out loud).
Lower flexibility
Not being able to moderate also means lower flexibility if things go wrong. For example, a participant might misunderstand a question or a task, or accidentally click the wrong button and navigate into the wrong area of your website or prototype, without even realising it.
If this happens during moderated testing, you can easily step in and politely help them get back on track. During unmoderated testing, however, this isn’t possible.
This is why it’s particularly important to run a pilot test when running a remote unmoderated test!
Pro tip: Pilot your test script and prototype with one or two participants to “test your test”. Your pilot will surface any confusing questions in your script as well as problems in your prototype. This will ensure that you get a chance to fix any issues before launching the full test.
So when should I use this method?
As you can tell from the advantages and disadvantages, there are times when this method is more suitable and other times when it’s best to rely on moderated methods.
When is it not as useful?
- Remote unmoderated testing can be tricky at the very start of a design process, when you’re trying to understand and shape the problem space, or when you’ve just started with early concepting and ideation. That’s when you need to be able to ask follow-up questions to really understand user needs, expectations, values and behaviours.
- If you’re aiming to test very complex designs or task flows with an unfinished prototype, unmoderated testing is also not ideal. Participants may have a harder time understanding and navigating such prototypes without the support of a moderator. If they get lost in the prototype during an unmoderated test you may find that you get very little value out of the test result.
When is it particularly useful?
- When you’re doing evaluative, validating research. For example, when you’re testing a fairly finished prototype of a new user flow, a new feature, or changes to visuals/content/copy. This method is also especially suited to testing live websites, as well as late-stage high-fidelity designs and prototypes. At this stage of testing, you’ve typically already established what problem you’re solving, and you’re looking to optimise your designs for your target audience.
- When you’re running recurring tests. For example, if you have an ongoing project where you’re gathering recurring feedback on design, content, and/or visuals, such as a newsletter or an email campaign.
- When you’re limited by time, budget, and/or access to your target audience. If you’re having a hard time convincing management to sign off on resources for a larger user research project, this method can ensure that you get user-feedback quickly and easily despite limited resources.
Our last parting tips
Raring to give it a go? We won’t keep you! We just want to share some final tips.
Tip 1: Start with your research goal and research questions
Unmoderated tests are typically shorter than moderated tests. This is because research participants have a shorter attention span without a moderator present.
Outlining your research goal and research questions is important in any user research, but it’s especially helpful for remote unmoderated testing, as it will help to focus your test script on the questions and tasks that really matter.
Tip 2: Use as simple language as possible
As an expert in your business and all things digital, it can be easy to forget that you might be testing with someone who’s not very tech-savvy or familiar with your industry.
The accidental use of difficult terms and phrasing may result in your research participant misunderstanding questions and tasks. To avoid this, make sure to remove any industry jargon or technical terminology from your test script. Your language should be simple enough for a 10-year-old to understand!
Tip 3: Provide clear directions
It’s always a good idea to ask open-ended questions in user research. However, in unmoderated testing, you might need to add some additional direction to such questions.
For example, if you’re asking your participant to take a look at a website and share their first impressions, it’s a good idea to specify a time length. Without this, you might get anything from a 10-second impression to a 20-minute impression. To ensure that you’re getting the sort of results you’re expecting, provide additional instructions.
Go forth and test
That’s it! You’re now ready to run remote unmoderated tests and ensure that you get continuous user feedback, despite social distancing and COVID-19.
If you’re curious about Teston, check it out here. (Psst, if you sign up for a free demo, you’ll get offered a free test on the platform!)
Happy testing!
Originally published at https://teston.io on April 29, 2020.
