Lean Experimentation: Step-By-Step Guide for Product Teams

In this article, we’ll explain how you can generate ideas, run experiments, generate a falsifiable hypothesis, validate multiple hypotheses with analytics, and use what you learn to drive growth.

Let’s get right into it!

Summary of lean experimentation

• Lean experimentation is a systematic, scientific process for testing and validating hypotheses about products, services, and features using the least amount of time and resources possible.
• Experiments help drive product success in many ways. They help you decide based on data and metrics rather than guesswork, ensure organizational agility, and give you a better chance of achieving product market fit with your target customers.
• When should you conduct lean experimentation? An answer you’ll be familiar with in product management: it depends. Experiments can be valid throughout the product lifecycle – during the idea validation stage, during testing, or even after launch.
• So, what are the steps for conducting lean experiments in your app? The first step is deciding what exactly it is you want to test: each idea you come across can be tailored into a distinct hypothesis you want to prove or disprove.
• Next, choose what sort of experiment type makes sense for you depending on your context. There are many options to choose from: a smoke test, a full prototype, a simple ‘Wizard of Oz’ style innovation, or targeted A/B experimentation.
• After that, you’ll want to create a detailed plan for your experiment, including which metrics you’ll look to measure, the quantitative data and outcomes you want to track, how much evidence you need, what constitutes failure and success, experiment duration, sample size, and more.
• Once you’ve executed your experiment, launch straight into analyzing your results. Remember to look at both qualitative and quantitative data for a rounded view, and check that the results are statistically significant before making product decisions.
• Now, based on the experiment’s results, you can effectively iterate on your SaaS product or feature.
• If the experiment is successful (i.e., you’ve proved your hypothesis), consider iteratively implementing the changes permanently. If it’s not, identify what you’ve learned and adjust your approach accordingly.
• Of course, you’ll need the right tool for the job to help you drive the product outcomes and innovation you’re after. Userpilot is a powerful, multi-faceted product adoption tool perfect for building, launching, and analyzing the results of experiments in your product.

What is lean experimentation?

Lean experimentation is a systematic approach to testing and validating hypotheses about products, services, and features using the smallest amount of time and resources possible (i.e., it’s not always about throwing more resources into the development process).

Lean experiments are a key part of the Lean Startup methodology, which is a process or business model for developing and launching new products and services quickly and efficiently. It is also used by many established companies as a form of corporate innovation to rapidly improve their existing products and services.

Why is it important to run experiments?

Experiments can help you in a number of ways:

1. Make data-driven decisions: Don’t rely on guesswork. Lean experiments help you make decisions based on the scientific method: form a hypothesis, gather evidence, and determine your outcome. The better your decisions, the lower the risk of failure and the greater your chance for success.
2. Put the customer at the heart of your product: Acting directly on customer feedback means you’re more likely to build something truly valuable, getting the most out of your development effort. Experimenting with functionality means you can quickly learn from your findings.
3. Ensure organizational agility: Lean experiments help you be responsive to a dynamic and shifting landscape, meaning you can quickly identify and address trends or challenges as they emerge.
4. Find a product-market fit: The results of even a single experiment can help ensure your team focuses in the right direction and ‘build the right thing‘ for your customers.

When to conduct a lean experiment?

An answer you’ll be familiar with in product management: it depends.

Put simply, it’s up to you to evaluate when experiments do and don’t make sense. You can do them throughout the product lifecycle: during the business ideas and assumption validation stage, throughout feature and product development, and after the launch to ensure you’re continuously improving the product.

Ideas become more valuable once they’re validated.

How to conduct lean experiments in your SaaS?

Next up, we’re going to break down exactly how to implement experiments in your SaaS and leverage results to gain insight from your users.

Step 1: Develop new hypotheses to test

Start with defining a hypothesis that can be tested.

There are many techniques for coming up with new ideas for hypotheses – start with what changes, improvements, or tweaks you think will help you achieve your business goals. You can then go through a prioritization exercise to figure out which are likely to drive the most impact.

For example, a hypothesis might look like this: adding a visual progress bar to the onboarding flow will lead to higher completion rates (and therefore activation).

Step 2: Select an experiment type for testing your hypothesis

There are many examples to choose from.

Smoke test

You create a landing page or marketing campaign to promote a new product or feature that doesn’t yet exist, then measure interest by tracking user interactions. It’s a great way to ‘practice’ – you can get an idea of engagement from a few simple landing pages.

Let’s explain with a simple example. Raise captured potential demand from users interested in financial modeling with an email form to join a simple waitlist.

Wizard of Oz test

This sort of experimentation is about testing users’ interactions (and helping you validate each assumption). Essentially, you build the front end of a feature before you’ve built the underlying tech. Wizard of Oz prototypes are often used to test user interactions with new technologies or features before they are fully developed.

It’s a good first step to reduce risk: you can validate your MVP for a fraction of the effort. They can help you test the feasibility of a new idea before investing a lot of time and resources into development.

Customer interviews/surveys

Interviews help you understand customer needs and preferences in detail – extremely useful data – but they are expensive to conduct in high volumes (research participants are hard to come by).

Surveys are great for capturing customer insights at scale. Remember you can implement a combination of open and closed questions to generate the most actionable feedback.

Prototype testing

This involves creating a simplified, non-functional version of a product. You can then use your prototype to gather user feedback on its design, layout, and usability.

Prototyping is one of the best examples of lean experimentation in practice: depending on how advanced the prototype is, you can even learn about potential performance impacts.

A/B testing

A/B testing – sometimes known as split testing – is a method of comparing two versions of a variable, such as a web page, email, or ad, to determine which one performs better.

It’s done by randomly assigning visitors to one of the two versions and then measuring the desired outcome (i.e., click-through rate, conversion rate, or sign-up rate).

A/B testing helps you validate your decision-making and ensures your ideas are tested thoroughly before committing.

Step 3: Create the experiment design and execute it

Next up, you’ll want to create a detailed plan for your experiment, including:

• The metrics you’ll use to measure success and the data you’ll gather.
• The target audience or user segments you’ll involve in the experiment (remember to factor in how large your sample size will be).
• The duration of the experiment.

Once you have this information, you’re ready to execute. Create your experiment, invite existing customers or new ones (depending on what you’re testing) to participate, and get started!

Step 4: Analyze the results from the lean experiment

Analyze both quantitative data and qualitative data. Remember to segment and analyze user data to uncover trends among different user groups: the devil is in the details.

The experiments you run are only as valuable as the lessons you learn from them. That’s what it’s all about. Despite that, it’s important you don’t automatically assume every finding is the same. Assess statistical significance to determine whether observed differences are valid.

Step 5: Act upon insights and improve the product experience

Now, based on the experiment’s results, you can effectively iterate on your SaaS product or feature.

If the experiment is successful (i.e., you’ve proved your hypothesis), consider implementing the changes permanently. If it’s not, identify what you’ve learned and adjust your approach accordingly.

Of course, you might need to conduct further testing to come to a strong conclusion.

Let’s look at another example: let’s say your experiment proves a checklist helps boost product adoption. Next up, you might want to determine how other visual prompts impact various other metrics.

Conclusion

There we have it – lean experimentation in a nutshell. It’s not quite a three-step process, but it’s still pretty simple and actionable.

Hopefully, you can see how lean experimentation not only helps generate new ideas, but simplifies the process, reduces risk, and helps allocate resources more effectively.

Next steps? You need to decide the right tool for the job.

If you want to conduct lean experiments code-free, book a demo call with our team and get started! Check out the banner below for more details and get started with lean experimentation today.