How to conduct user research without even showing up

Unmoderated testing provides yet another method for gathering valuable insights

Setting up qualitative user research requires recruiting the right set of participants, and determining the best way to conduct the sessions. The options include using a lab or testing center, visiting a participant in their “natural environment” (e.g. a home, an office, a retail setting), or conducting an interview remotely — an option made even more popular by the pandemic. Sessions are typically moderated by a researcher or a member of the design team, who guides a participant through a set of questions, or in the case of UX or usability research, through hands-on tasks using some sort of prototype. 

Now, with the help of digital tools, user research can be easily set up and conducted without the need for a moderator. Unmoderated testing has become increasingly popular in the last few years, in large part due to online platforms such as UserTesting.com, Userlytics, and UserZoom, which make the process easy for both the research team and the participants. 

Does my research need a moderator?

There are no hard and fast rules here — if you can design your study so that it doesn’t involve any direct interaction between people, you can consider using unmoderated research to achieve your learning goals. However, this requires taking a user-centered approach to ensuring that your research will go smoothly, without anyone to provide guidance or answer a participant’s questions. 

What this means:

• Your learning goals should be very focused 
• The flow of your research must be tightly scripted 
• Everything you present must be clear, as there won’t be any way for the participant to ask clarifying questions
• Participants will experience fatigue quicker without a moderator there to engage them, so the duration of each session should be kept short – we recommend no more than 30 minutes, but have found the “sweet spot” to be under 20 minutes

Consider the pros and cons of unmoderated testing 

Pros: 

• Lower cost: not only do you avoid travel/facility expenses, but the entire end-to-end process (including recruitment and incentive costs) is often much less expensive
• It removes any potential moderator bias, interruption, or “leading the participant” in a specific direction (we are only human, and this can happen unintentionally) 
• Speed: because testing sessions can be done simultaneously, you can usually complete your research in a much shorter time frame
• Ease of recruitment & setup: build your screener online and upload your prototype, and boom – you’ve got a research study ready to launch
• Geographic diversity: the sky’s the limit in terms of where your participants might live
• The services allow for structured data collection (e.g. time on task, success rate, SUS scores) as well as auto-transcriptions of the sessions
• Easy access to videos of the sessions – which you can re-watch later, and can also use in readout reports or other client share-outs 

Cons:

• You can’t step in to course-correct if someone is struggling to understand something or is having technical difficulties 
• You’re at the mercy of the limitations of the testing platform, and can only use the components available (example: we recently used a service that didn’t have a “ranking” module, which would have come in handy)
• Quality of participants can vary: a good professional recruiter can usually “weed out” participants who may not seem very engaged or articulate, or may be fudging some detail about themselves for inclusion in the study
• Less control over who is recruited – when working with a recruiter, your screener can get very specific with regards to demographics (e.g. I want 2 college-educated women who work in retail between the ages of 30-35, 2 stay-at-home dads in their 40s, and 3 retirees living in the mountains of Montana). Note that you CAN work with an unmoderated service’s customer service team if you’re looking to get granular with your recruit, but this does take a bit more time, and costs extra
• Shorter sessions – which means your focus needs to be much narrower
• Flow is fixed – which means you can’t dig deeper or ask someone “why?” in the moment

Principles to ensure your unmoderated research is successful

• Keep all tasks simple and focused.
• Set the context up front: For instance, if you’re asking someone to give feedback on an app that helps them find the nearest fast food restaurant, be sure you are clearly setting up the proper context at the start.
• Remove anything from your designs that may cause confusion – such as an unfinished section, or “lorem ipsum” text. Remember that you won’t be there to explain things, and you don’t want your users to get tripped up.
• Recruit a few more participants than you would in a moderated study – the quality of the sessions will vary a lot. You should expect to have a few participants that simply aren’t all that articulate, and these sessions may be less helpful.