UX researchers have developed many user experience testing methods over the years, and more techniques are still being researched.

There are so many options that it’s easy to become overwhelmed and unsure of which method will work best for your circumstances. Let’s take a look at the top usability testing methods, and let’s see how they all compare, and identify the best use case, advantages, and disadvantages of each.

Summary of usability testing methods for SaaS

• Usability testing is a UX research method used to identify design problems and areas of improvement. This is done by testing the product or certain parts of it with a group of participants.
• While they sound similar, usability testing and user testing are different concepts: the former is used to determine whether users are able to use the product successfully, while user testing determines whether there is a market for the product and determines its viability.
• Usability testing is a subset of user research. User research is a broader concept and also involves focus groups, surveys, and interviews. These both give you insight into your users’ general behaviors, but user research focuses on understanding your users while the other focuses on understanding how they interact with your product.
• Usability tests can be moderated or unmoderated, remote or in-person, explorative or comparative, formative or summative.
• The eight usability testing methods discussed in this article are:

1. Guerilla testing: The simplest form of usability testing is guerrilla testing. Guerrilla testing entails going to a public location, such as a coffee shop, and asking for feedback on your prototype.
2. Unmoderated remote usability testing: The testing is conducted remotely without a moderator. Test results are fast, robust, and inexpensive, and can be analyzed further using the information.
3. Lab usability testing: Lab usability testing is about getting detailed feedback from a small group of people. Typically, you segment users, invite them to offer more in-depth feedback, and offer an incentive for their efforts.
4. 5-second test: With the 5-second test, you expose a portion of your product to your users for five seconds and then interview them to see what they took away from the experience.
5. First-click testing: First-click testing aims to determine whether users can easily identify the happy path they need to follow in order to accomplish a given task.
6. Card sorting: The card sorting method is perfect for prioritizing content and features in user interfaces. You place concepts on cards and allow participants to group and categorize the cards.
7. Session replays: Replays are non-audio video recordings of a user’s journey and interactions on your site.
8. Eye-tracking: A heat map is a visual representation of where a user looks most on a webpage.

• Usability testing consists of 5 major steps: choosing the method, defining tasks, recruiting participants, documenting the results, then analyzing and acting on them.
• With tools like Userpilot, you can collect quantitative and qualitative customer feedback for different usability testing methods.

What is usability testing?

Usability testing is a popular method for UX research.

A usability test measures how easy it is for users to use a design with a group of representative users where they are observed as they attempt to complete tasks. Tests are often repeated from early development to the final release of a product.

Usability testing vs. user testing

What’s the difference between usability testing and user testing? Those two terms sound similar, but there’s actually a huge difference between them.

User testing is entirely user-focused and serves as the first opportunity to establish a potential market for a product.

Usability testing is more product-focused and involves assessing users’ needs within an existing product.

Both aim to create a design solution that meets real user needs and actively seek ways of meeting those needs or addressing user pain points.

Usability testing vs. user research

You can think of usability testing as a subcategory of user research. The latter is conducted to understand customer needs and behaviors, while usability testing is used to determine how users interact with your product.

Besides usability testing, user research often involves other methods such as surveys, customer interviews, and focus groups.

Why you shouldn’t skip usability testing when designing a product?

There are many benefits to implementing a usability test as part of your product design process. Here are some:

• A usability test lets designers and developers identify problems before launching the product. The earlier usability issues are identified and fixed, the less expensive the fixes will be.
• This allows you to receive valuable user feedback on how you can improve the product.
• A usability test measures the effectiveness of personalization and localization features and allows you to optimize accordingly.
• It uncovers pain points in the user journey, which gives you valuable insights to optimize the user experience and boost customer retention.

What are the different types of usability testing methods?

There’s no one-size-fits-all when it comes to usability testing. There are a vast variety of methods out there. The method you should choose depends on your unique circumstances.

This section compares the different kinds of usability testing methods so you can properly choose the best option(s) for you.

Moderated usability testing vs. unmoderated usability testing

Moderated usability tests are primarily used in the initial phase of a product design. This is an intense method that involves direct supervision by a facilitator.

Moderated tests can be done virtually or in a person. The facilitators watch as users test the product and record their observations. Sometimes, users are asked to verbalize the thoughts running in their minds, and the moderator records that too.

Moderated usability testing examples include user interviews, guerilla testing, screen sharing, etc.

Unmoderated usability testing is the opposite of moderated testing. It’s conducted without supervision.

The advantage of unmoderated usability tests is that they cost less—you won’t have to pay for a testing lab, hire moderators, etc. It also has a reduced risk of bias.

However, this method is not effective if you want qualitative data as it’s difficult to conduct follow-ups on thousands of people. Examples of unmoderated tests include eye-tracking, 5-second tests, and first-click tests.

Remote usability testing vs. in-person usability testing

As the name suggests, remote usability tests are performed remotely using a tablet or computer, while in-person tests require the test to be completed in the physical presence of a UX researcher.

In-person tests provide additional data points over remote tests since researchers can observe and analyze body language and facial expressions. In-person testing, however, is usually costly and time-consuming: you must find a suitable location, set a specific date, and recruit (and pay) participants.

The main benefits of remote usability testing are the possibility of recruiting a large number of participants from different parts of the world, and that participants can complete the tasks in their natural environment as they use their own devices.

Explorative vs. assessment vs. comparative testing

Explorative tests aim to get the user’s thoughts, feelings, and opinions about the product. Typically, the information is gathered at an early stage in product development and helps researchers identify gaps in the market and potential new features. It’s best to use open-ended questions so that testers can express themselves freely.

Assessment testing is used to evaluate a product’s overall functionality. Your goal is to find out how satisfied the user is with your product. Customer satisfaction surveys are a good way to figure this out during the testing.

The last on this list is comparative testing. Comparative testing is the best way to find out the product the target audience will pick between you and your competitors. The test involves giving users a variety of product options and asking them to pick one. It would be ideal if the participants in this test had used the products you asked them to compare.

Formative usability testing vs. summative usability testing

Formative usability testing is a qualitative research process that seeks to understand what about the product needs to be improved. It happens in the earlier stages of the design and development process, resulting in qualitative findings and ideas that can be incorporated into prototypes and wireframes.

Summative usability testing is a research process that’s more quantitative in nature. It happens later in the product development process and seeks to understand whether the solutions you are implementing are effective. Summative usability testing results in quantitative findings can help determine broad or specific areas for product improvement.

Why is it important to use multiple usability testing methods?

It’s often best to combine the different methods to help you achieve a holistic result. Here are some reasons you shouldn’t limit your product to one testing method:

• Different usability testing methods might be better for different users
• Each method has a specific focus and may also have specific shortcomings
• Different usability methods can help you identify different things
• The data you collect will be more detailed and accurate

8 different usability testing methods and how each can help

The previous section gave you an overview of different usability testing methods—many of which have sub-categories. It’s time to get granular. This section shows you eight usability testing methods you can start implementing for your SaaS product.

Guerilla testing

The simplest form of usability testing is guerilla testing. A guerrilla test involves going to an open area, such as a coffee shop, to ask people about your prototype. People participating in the test are randomly selected.  Users are often asked to complete a quick usability test in exchange for a small gift (such as free coffee). It is low-cost and relatively simple testing that provides real user feedback. During guerilla testing, it’s essential to keep in mind that participants may not represent your target market. Due to this, guerilla testing might not be appropriate for testing niche products.

It is also a good way to collect personal opinions and emotions about ideas and concepts.

Here’s an example of guerilla testing: we created this tooltip on Asana to showcase how fake door testing works. The end goal was to track how many users clicked on the button. This will enable us to estimate the demand for the potential new feature and assess whether it is worth building it or not.

The next step was inviting interested users to test the beta version.

Unmoderated remote usability testing

An unmoderated remote usability test is performed at the user’s discretion without any moderator supervising the process and offers fast, robust, and inexpensive results. Unmoderated remote usability testing will help you test a particular question or observe user behavior patterns. This test is best conducted when you want to test a hypothesis, but not something you should do when you need in-depth insight for product development.

Lab usability testing

The purpose of lab usability testing is to gather detailed feedback from a small cohort of people. As a general rule, you segment users, invite them to offer deeper feedback and provide incentives to encourage their participation. The incentive can be in the form of discounts, credits, or anything relevant and helpful to the user segment you choose. Here is a real-life example. A modal was triggered for a specific user segment and is inviting them to take part in beta testing.

Five-second test

During the 5-second test, you expose your users to parts of your product for five seconds each and then interview them to determine what they learned regarding:

• Your product’s purpose, as well as its key features or elements
• Target audience and trustworthiness of the brand
• Usability and design of the product as perceived by the target audience

Using this method, you can quickly collect a lot of qualitative feedback about your site’s first impressions. Customers decide whether to trust your website within 0.05 seconds to 2.8 seconds, according to various studies. Does your website make a good first impression and inspire people to stay and take action?

First click testing

A first-click test evaluates whether users can easily identify where they need to navigate to complete a given task. You can determine how intuitive your site design and linking structure are by measuring how long it takes users to decide.

How to implement this? Give users a checklist, offer a reward for finishing the tasks in it, and then track how they engage with the UI to see if they get stuck.

Your goal is to track if they stay on the happy path or not. This helps you uncover friction and drop-off points—basically the unhappy paths—within the product.

As promised, reward them with something valuable.

Card sorting

Card sorting will help you to form a hypothesis on how your content and features should be arranged. The technique is relatively simple. Just place concepts on cards, then allow participants to group and categorize the cards. At the end of the exercise, a moderator asks the test participants the reasoning behind the way they sorted the cards. Card sorting will help you to form a hypothesis on how your content and features should be arranged.

There are two types of card sorting:

Open card sorting:  Participants can group topics according to the way they see fit. Participants label these groups of topics on their own, without predetermined fields.

Closed Card Sorting: Participants sort topics into predefined categories.

Session replays

Session replays are unmoderated and performed remotely. The user doesn’t know that they’re being observed because session recording tools are preinstalled on the website or app without the user’s knowledge.

This method is great for identifying user behavior and trends and has an advantage over user testing since the participants aren’t aware they are being observed and display their real on-site behavior.

Eye-Tracking

An eye-tracking study is a useful technique for gathering information about users’ visual interactions with a page. It can also be used to test layout and design elements and see which elements may distract or take someone’s attention away from the main page elements.

This test is essential as it lets you learn if users and web visitors focus on the areas you want them to see the most.

Usability testing stages

Now that we have discussed the different usability testing methods, let’s dive into the usability testing process.

Define your goals and determine which feature you want to test

Be clear on what exactly you’re trying to accomplish. Would you like more information about how your users will interact with particular parts of your design? Will you be testing different versions of a particular feature? To start with, you can test three different versions of the registration flow to see which is the least confusing. Think about the pros, cons, and areas for improvement of your product or website in order to create a solid hypothesis.

Choose the usability testing method

Decide which of the above methods you will use based on your goals. What kind of data do you need – quantitative or qualitative? Will the session be moderated or are users going to perform the tasks independently? Do you have a sufficient budget to perform the tests in person? By answering these simple questions, you will be able to choose the method which bests fits your situation.

Define the tasks within the test

Test participants’ responses and reactions are influenced by almost everything they are presented with during the test, so you need to be careful not to set up biased tasks for them to complete. Instead of giving explicit instructions about what the user is supposed to do, ask her to imagine that she is in a specific scenario and then ask her to demonstrate what she would do.

Tasks may either be open or closed, but finding a middle ground between these extremes would be beneficial to users.

Open/Exploratory Tasks tend to be more research-oriented and broad. Participants are given the bare minimum information on how to perform the task in this case. Tasks of this kind may have more than one answer since answers differ from one user to another.

The Closed/Specific Tasks are clearly defined and goal-driven. They limit the scope of interpretation and confusion for the user because they have an endpoint, which results in more accurate and appropriate results.

Recruit participants

Keep your end goal in mind when recruiting participants. For instance, if your goal is to improve a feature, then the right segment for this test would be those users who used the feature more than once and can give insights on what to improve.

Then, you can invite the user segment of your choice to test the feature by triggering a slide-out, banner, or modal like the one below.

Be clear on the number of participants you want before sending your invites. The number of participants depends again on what you want to achieve and how many resources you have. On average, companies use 8-10 participants for moderated research. Unmoderated usability testing can have up to 50-150 participants.

Moderate the testing and document results

Remove every possible distraction and other logistical issues that may affect the test. Do this in advance before the process begins.

When you start, allow participants to complete one task before moving to the other. If you’re moderating, give participants the freedom to be flexible as they want—but not to the point that it will affect your goal and the end result.

Collect the results for analysis after the process is complete. You could summarize your findings so it will be easy to analyze when the time comes.

Below are some tools to make the research process easy and quick.

• Userpilot: Good for running surveys, fake door tests, and inviting users to participate in beta tests.
• Hotjar: Heatmaps for visualizing user behavior and identifying the areas of the product that grabs their attention and the areas that users ignore.
• Eyewear: Eye-tracking software that works like heatmaps to enable you to detect eye and head movements as users interact with your tool.
• Usertesting: Platform for recruiting vetted software testers.

Analyze the insights and act on them

As you go over the collected data, draft a usability testing report that aims to identify the most recurring and serious problems mentioned by participants. Rating and tagging each problem according to its seriousness and the number of complaints will help you determine which ones to tackle first.

You could also cross-reference the problems to identify patterns for better analysis.

After you have prioritized the issues, ask follow-up questions to customers who gave you a good NPS score, work on solving the problems, then get back to users to close the feedback loop.

Conclusion

Usability testing is an effective strategy for identifying areas of improvement in your product. It’s something every SaaS company should conduct at least once during the product development phase and once in a while as new features are released.

As discussed in this piece, there are many approaches to usability testing, and none of them is better than the other. It just depends on your product, audience, and the problem you want to test for.

Ready to get started with some of the usability testing methods we discussed? Book a demo call with our team to see how we can help!