Product Experimentation Guide for Product Managers

A mindset of product experimentation is vitally important for product success and growth.

If product managers can consistently apply the scientific method, analyze the effect of multiple variables changing, and gather reliable data about the future direction of the product, they’re in a strong position to improve customer outcomes.

Ready to learn more about experimentation? Let’s make a start!

TL;DR

• Product experimentation is a systematic and data-driven approach, where you consistently design and run experiments that help you make better product decisions.
• A product experimentation framework is a structured approach to testing and evaluating product changes to isolate and determine the impact on user behavior and business metrics.

There are many different types of product experiments. Let’s explore below.

• A/B testing is a type of experiment in which two versions of a product are shown to different groups (i.e. you randomly assign users to group A or B).
• Multivariate testing (MVT) is a method of experimentation that simultaneously tests multiple variables within a website or product.
• Fake door testing is about presenting users with a call to action or a low-fidelity prototype of a product idea (e.g. a landing page or modal) and seeing how many people will try to use it to gauge interest.
• Usability testing involves observing and analyzing how real users interact with a product to identify areas for product improvements.
• Click tracking describes the process of recording and analyzing user clicks on a website, app, or within your product to gain insights into user behavior, identify areas for improvement, and optimize the user experience.
• Funnel testing involves analyzing each of the specific components in the funnel and identifying areas where users are dropping off.

In the modern SaaS world, being able to form a solid product experiment hypothesis and execute successful experiments is a key product management skill.

• You need to start by gathering data to generate insights – you can do that from various sources.
• Next, define the end goal. What exactly do you want to achieve?
• Once you’ve set that, you can formulate testable hypotheses (i.e. tentative statements that can be tested).
• Finally, you need to start planning in earnest. What type of experiment do you want to run, and how large does your sample size need to be? You’re then free to launch your experiment!
• Common best practices for launching experiments include: adopting the most effective experimentation frameworks, prioritizing the highest impact areas, and continually reflecting on user feedback to determine future direction.
• Of course, you’ll need to choose the right tool for the job. Userpilot is a powerful and multi-faceted product adoption platform with a range of experimentation capabilities. Book a demo to learn more!

What is product experimentation?

Product experimentation is the process of testing new features, designs, or other changes – based on user research – to a product to see if they improve user experience, engagement, or a range of other metrics.

It is a systematic and data-driven mindset, where you consistently design and run experiments to make better product decisions.

What is a product experimentation framework?

A product experimentation framework is a structured approach to testing and evaluating product changes to determine the impact on user behavior and business metrics. It provides a set of guidelines and processes for designing, conducting, analyzing, and communicating product experiments.

It offers product teams some key benefits:

• Reduced risk: By testing changes in a controlled environment, product organizations minimize the potential for negative impacts on user experience or business performance.
• Improved decision-making: Experimentation provides valuable data-driven insights to inform product decisions, reducing reliance on assumptions and guesswork.
• Enhanced user experience: By understanding how users interact with changes, companies can optimize product features for better usability.
• Increased innovation: Experimentation fosters a culture of continuous improvement and encourages the exploration of new ideas.

Types of product experiments

There are many different types of product experiments. You might want to run experiments differently for several reasons – but primarily, you’re looking to generate a different set of actionable insights.

Remember, a good experiment solves problems by showing you one or more solutions, regardless of individual experiment results… it’s all about what you learn along the way.

A/B testing

This is a type of experiment in which two versions of a product are shown to different segments (i.e. you randomly assign users to group A or B). The product versions should be identical except for one specific change – such as the color of a button or the wording of a call to action.

The results of the experiment are then used to determine which version performs better (for example, which results in higher conversions). You can then adopt that version and roll it out to other customer segments.

You can test a whole host of things: landing pages, onboarding flows, sign-up pages, and more.

It is highly scalable – A/B testing can be applied to various elements of a product, from individual components to entire pages or user flows. It’s also a relatively inexpensive way to gather valuable insights and improve product performance.

Multivariate testing

Multivariate testing (MVT) is a method of experimentation that simultaneously tests multiple variables within a website or product. The goal is to figure out which combination of changes produces the most favorable outcome (i.e. increased conversions or improved user engagement).

In a nutshell, it’s the same as running an A/B test – but rather than changing just one element, you’re running experiments with multiple components or multiple pages simultaneously.

Usually, this gives you more detailed insights than A/B testing.

Fake door testing

Fake door testing is a user research and experimentation technique used to gauge interest in a new product, feature, or service before it is developed. It involves suggesting the non-existent product or feature is real, and then observing how users interact with it.

How does that look in practice? You could present users with a call to action or a low-fidelity prototype of a product idea (e.g. a landing page or modal) and see how many people will try to use it.

For example, in the mockup below, users are prompted to ‘add a goal’ within Asana (guided by a handy tooltip).

Once the user clicks on it, the pop-up says it’s not ready yet – and Asana tracks the CTR to see if it’s worth building.

Usability testing

User testing is a crucial aspect of the product development process, enabling product teams to evaluate the usability, effectiveness, and overall UX of their products or services. It involves observing and analyzing how real users interact with a product to identify areas for improvement.

Typically, this will involve giving your user base a simple list of tasks to complete in order to observe how they attempt to tackle them. It’s an easy way to validate if a design is viable.

There are lots of variations to think about though.

• Card-sorting. This involves giving participants a set of cards and asking them to group the cards in a way that makes sense to them.
• Five-second test. This is a quick usability test aiming to gauge users’ first impressions of a product or design within a very short period – great for minimal investment.
• Lab usability testing. Users participate in testing sessions in a controlled laboratory environment for more precise control.
• Session replays. This is a recording of a user’s interaction with a website or product – it captures everything the user does (i.e. their mouse movements, clicks, taps, and scrolling behavior).

Click tracking

Click tracking – sometimes known as clickstream analysis – is a process of recording and analyzing user clicks on a website, app, or within your product to gain insights into user behavior, identify areas for improvement, and optimize the user experience.

Essentially, it involves tagging UI elements, auditing clicks, and performing data analysis across the customer journey.

You can use click tracking to experiment with different UX designs or experiment with new features – and then see how many users engage with the new elements.

Funnel testing

Funnel testing, also known as conversion rate optimization (CRO), is a data-driven approach to evaluating and improving the effectiveness of your in-app funnels.

It involves analyzing each step in the funnel, from the initial point of contact with a customer to the final conversion, and identifying areas where users are dropping off or abandoning the process.

Over time, your analysis will show you where users are dropping off; you can start to figure out what’s blocking conversions and start to create more favorable outcomes (i.e. boost user retention).

For example, you might want to look at a subscription flow, identify how users navigate from a home page, and decide which steps in the journey you can optimize.

How to run product experiments

Next up, we’re going to cover how a product manager can run successful experiments. In the modern SaaS world, this is a vital skill to inform the product roadmap and offer a route to continuous improvement.

Collect data to uncover valuable insights

You shouldn’t launch right into running random experiments: they need to have a clear purpose. For that, you need to start by gathering data to generate insights.

There are many data points to choose from: direct user feedback, in-app user behaviors, and other product analytics.

Define the end goal

Next, decide on the business goal and key metrics you are trying to improve. That could be increasing free to paid conversion rate, improving onboarding completion, customer acquisition, or any number of others.

“Begin with the end in mind.”

By defining your success criteria up front, and how they map onto a clear desired outcome, product teams will be motivated and on the same page. That’s why realistic experimentation goals are so important.

Create a hypothesis relevant to the end goal and future experiments

A hypothesis is a tentative statement that can be tested through experimentation or observation. They’re fundamentally important: good experimentation connects data to relevant hypotheses and helps to build a clear understanding.

Defining hypotheses helps de-risk high-risk ideas, validate assumptions, and move toward a mindset of continuous discovery.

Now your hypothesis must be relevant to the metrics set in the previous step, specific, testable, and measurable – or it’s useless. You need to generate specific and testable hypotheses that propose a causal relationship between the change being tested and the expected outcome.

For example, your hypothesis might be, “We believe page X adds friction to the onboarding flow, and including it will decrease user retention.”

Choose the experimentation type and sample size

Next, you need to pick a type of experiment based on your goal and then set the parameters (i.e. sample size and timeframe). Good experiments always factor in logistics.

The number of people you test with and the sample size is extremely important because it affects the accuracy and reliability of your results. If your sample size amongst test groups is too small, your results won’t have statistical significance meaning customer insights might not be replicated across user segments.

Pick the best tool for your experiments

There are many factors to consider when picking your tooling – evidenced by the fact not everyone uses the same tools all the time! Deciding factors usually include:

• Price
• Available features
• User experience (i.e. how intuitive is the user interface)
• Integrations with other products, and
• Data security.

Run experiments

You’ve done all the prerequisites – now it’s about executing your plan.

Deploy your experiment into the pre-planned experiment, gather and analyze data, and begin to validate ideas (i.e. if you’re running multivariate tests, you might see how multiple variants impact the customer experience).

Analyze and interpret results to inform future experiments

The final step in the process is to conduct a robust analysis.

Examine your results in detail and map them to your hypothesis. Have you been able to prove or disprove what you set out to? How might user journeys be influencing results?

Whatever your experiment results tell you, you can make data-driven decisions about the next steps and key product improvements.

Best practices for running product experiments

Running effective product experiments requires careful planning, execution, and in-depth analysis to ensure that your results are reliable and most importantly actionable.

Here are some best practices for conducting good product experiments:

Adopt the best product experimentation frameworks

There are an unbelievable number of different frameworks out there: part of product management as a discipline is about making decisions and choosing the right one for the job.

Ultimately, you need to try out different processes, identify which works for you and your product, and then stick with the most effective.

Once you find the right framework, you’ll find:

• Increased focus. Frameworks make it simple to align experiments with strategic goals and objectives.
• Improved experiment quantity. Enhances the quality and consistency of experiments, minimizing human bias.
• Drive decisions with data. Fosters a culture of data-driven decision-making, enabling you and your team to make informed choices based on empirical evidence.
• Drastically reduce development risk. Mitigate the risk of making costly and time-consuming mistakes by validating product changes and features before full-scale implementation.
• Continually improve. Promotes a culture of continuous improvement, enabling product organizations to quickly adapt and optimize products based on user feedback and data-driven insights.

Prioritize experiments that have the potential for high-impact

Product managers don’t always have the luxury of limitless time and resources. The name of the game is prioritization: which are the most pressing issues, where are your riskiest assumptions, and where your biggest opportunities for improvement are to improve the user experience?

A bonus tip is to start with small, well-defined experiments to gain quick wins and build your team’s confidence in the process.

You can then gradually increase the complexity and scope of experiments as the team matures and gains experience.

Constantly collect qualitative data with in-app surveys to identify opportunities for optimization

User interviews are often an incredibly rich source of data. But you don’t often have the chance to conduct them.

In-app surveys can be just as useful for gathering insights into the user experience, understanding what causes friction, and spotting areas that need improvement.

Trigger in-app surveys at different touchpoints- making sure to include both open and closed questions – to gather data with ease.

Conclusion

That just about wraps things up.

You should now have a solid understanding of what product experimentation is, how valuable a tool it can be for product managers, how you can use it to gather valuable insights, and best practices for running them in your product.

If you want to build product experiences code-free, book a demo call with our team and get started!