Are you running product experiments continuously?

Great products are built on experimentation–evaluating ideas and making decisions based on product growth insights data, not just what you think is an amazing idea (even if it is).

Using your time and resources to challenge your assumptions and intuition can seem daunting, but there is a method to the madness.

That’s why we spoke with Alex Levich, Product Management Lead at Google, to find out her unique method to running product experiments to grow products. Plus, we’ve added a few of our favorite tips for SaaS tools at the end.

You can watch the full interview here, or read this article for a summary of the key points.

Overview of interview with Alex Levitch on how to run product experiments

• Product experiments allow you to evaluate the impact and prioritize product changes and roadmap.
• Alex Levich, Product Management Lead at Google, explains how “auto-generate subtitles for video” became a starting goal for a product experiment to grow video at Facebook.
• A framework for running product experiments is “Think Big, Start Small, Scale or fail fast.”
• Thinking big: big-picture goals that need to be concretely defined.
• Starting small: breaking down the big picture into smaller parts by thinking about product problems your users experience that you are trying to solve.
• Start with a Minimum Viable Product ( or go directly for a minimum lovable product)-a product introduced in the market with just enough features to attract early adopters.
• Scale or fail fast: Looking at the data, learning from it, and iterating based on that; this goes for both successes as well as failures.
• You can grow products using product experiments, but also use such experiments to boost key metrics once you’re up and running.
• Segmenting the audience helps serve only the ones that you think are relevant while running experiments.

Why do we run product experiments?

In order to reduce the risks involved with spending a lot of time and resources on something your users may not use, you can run product experiments.

Experimentations require applying the basics of the scientific method (e.g., controls, variants, randomized samples, etc.) to establish causation between changes to the product and their outcomes.

“Product experiments allow us to evaluate the impact and as a result, prioritize products,” Alex Levich.

Instead of making assumptions, they allow product managers to learn whether customer success rates would improve.

In other words:

• We don’t want to build products and include features that don’t work
• We want to measure the potential impact
• We want to optimize a couple of approaches to find the best solution

While many people recommend the scientific approach to product experimentation–Observation, Hypothesis, Testing–Alex Levich has a more tangible, real-world approach unique to her years of experience running product experiments.

Let’s take a look!

A framework for running product experiments

There are three components to this.

1. Thinking big
2. Starting small
3. Scaling or failing fast

Thinking Big

“This may sound like a cliche title, but really thinking big is about three things: 1) being crystal clear on your goals and what you’re trying to do 2) envisioning the biggest, best-case scenarios that could happen, and 3) brainstorming a bunch of different ideas,” Levich says.

First, define your goals

Why are you doing product experiments?

For example, are you launching a new product and seeking product-market fit?

Are you thinking about which features you can add to make your current product more attractive to its users? Are you optimizing a conversion funnel?

These are all the big-picture goals that need to be concretely defined.

Alongside that, you’ll want to know how you will measure such goals, aka your success metric or north star metric.

A north star metric is the key measure of success that defines the relationship between the customer problems that the product team is trying to solve. It can also include the revenue that the business aims to generate by solving this problem.

For example, is your north star metric the number of repeat users? Active users? New users? Think about how you will actually know that your goal is on the right track.

Next, envision the best-case scenarios

“To envision the future, there are a couple of questions I ask myself to help me figure that out,” Levich says.

These questions are:

• What will the future look like?
• What are your assets?
• What are the problems people have?
• What do you do really well?
• What about other players?
• What types of problems are you excited about solving?

A lot of this has to do with understanding what you do really well and how you can position your offer.

“For example, Google knows how to organize information and rank it really well and bring it back to the users,” Levich tells us.

Putting it into context

“Here’s a situation that I found myself in. When I was working in video at Facebook, we saw video consumption is on mobile but many videos don’t have sound,” she explains.

“We also saw people consume videos around the world and many times these videos are in foreign languages and sometimes they don’t understand. They also wanted to see videos that are relevant.”

Based on this story, we know that the problem is that people aren’t engaging as much with video while on mobile.

Based on that, Levich goes on to explain that one idea from this situation could be: auto-generated subtitles for video.

But if you think bigger than that, more ideas will flow.

“Our big goal was to make the video consumption better [while] our success metric was to see more video watch time. The vision we put together was ‘better video that’s understandable by anyone, anytime, anywhere.”

Based on that big idea, here are some additional ideas that can stem from it:

• Open up API for third parties
• Auto-translate into user’s language
• Auto-dub the video into the user’s language using the original voice
• Auto turn on/off subtitles based on user environment

Starting Small

Now that you have a big, expanded set of ideas, you can go start to piece everything together.

According to Levich, starting small is about breaking down the big idea into smaller steps, creating a roadmap, and then validating ideas.

“The one thing I found very helpful in breaking things down is focusing on user problems,” she says. “It’s exactly what it sounds like. It is problems your users experience that you are trying to solve.”

To get to the bottom of that, Levich recommends asking these questions:

• Why is your big vision not yet a reality?
• How do we know these problems are real pain points?
• How will we know once we’ve solved them?

Building the roadmap

“You can’t solve everything all at once. It’s going to be an iterative process,” Levich confirms.

When it comes to such baby steps, that’s when it’s good to begin with the first baby step–or, the Minimum Viable Product.

Start with users’ number one problem, then sequence the subsequent user problems to solve in order to prioritize them all, building more and more value along the way.

Editor’s note: read more about this in our Overview of Teresa’s Torres Continuous Discovery framework article (or watch the video here)

Minimum Viable Product

A minimum viable product, or MVP, is a concept in which a product is introduced in the market with basic features, that is, enough features to attract early-adopter customers.

When it comes to running product experiments to grow a product, you need to start with MVP.

This allows your team to validate product assumptions outlined on your roadmap.

For SaaS companies, low-code and no-code development tools can help build MVP (Minimum Viable Product). Then, you can get feedback from real users as fast as possible and make changes accordingly.

“This [MVP] is your first draft and depending on the learning you make throughout this process, you are going to evolve it.”

Scaling (or Failing) Fast

In this process, you need to use data. Validate your ideas through qualitative and quantitative data.

“The whole idea [of this framework] is for you to be successful. But the truth is, there’s no success without failure. Failure is just about learning and iteration. If you haven’t failed you probably haven’t aimed high enough.”

The same thought process remains for successes as well. While they are something that should be celebrated, you should invest just as much in learning when you succeed.

It allows you to elevate those successes and continue growth in the right direction.

Product Experiments That Will Boost Your Key Metrics

Now that it’s been clearly explained how to create and grow products using product experiments, we can scale things back to using experiments to boost key metrics once you’re up and running.

Let’s look at three types of product experiments you can ponder to see what solves various problems.

Boosting User Activation

See if you can make a big difference to activation rates by utilizing onboarding checklists.

Checklists help show a user all the steps in sequential order, leading them on a journey to Activation and other key-value milestones.

Boosting Feature Adoption

To boost feature adoption, test and evaluate the usefulness of promoting additional features.

There are a variety of ways you can help users see more value in your product faster.

Extend past user onboarding checklists and use tooltips and modals to highlight additional features that you think your users will love, furthering your relationship with the user.

Improving User Retention

Experiment with ways you could reinstate churned users or help prevent churn from happening.

Either through NPS surveys to identify detractors…

…or churn surveys to see what leads to canceled subscriptions.

For some quick wins in product and feature experimentation today, here are some additional things every SaaS product owner and product team could test:

• Changing navigation options to a different order
• Adding gamification elements such as progress bars to your checklists
• Changing the location of a Call To Action (CTA)
• Try CTAs in a different color
• Contextual tooltips that pop up when a user slows progress or seems to hit a crossroad

Bonus tip: Segmenting the audience

Levich already suggested that you should focus on your users’ problems. That’s because a clear understanding of the audience you’ll serve is key.

But what happens when your experiment only serves one section of your audience?

We strongly believe in segmenting your audiences–especially when it comes to product experiments.

A segment is a section of your target audience that has similar behaviors, needs, characteristics, and more.

Segmenting your audience will help you test hypotheses on relevant customers only.

Conclusion

We know your product team and management are loaded with amazing ideas!

Running product experiments is all about prioritizing those amazing ideas, envisioning an awesome future for the product, and breaking it down into a sequence of steps to make it happen. Then, it’s about letting the sequence change based on each iteration and how each step shows its success or failure.

It’s also about prioritizing your customers; making sure these are real problems that your users need solving.

We hope this inspired and gave you the roadmap to start experimenting on your own!

Remember that no matter if you succeed or fail, you always receive learning from it. So, really, there are no failures when it comes to product experiments–other than experiments with no data.

Want to get started with product user analytics and run in-app product experiments? Get a Userpilot demo and get actionable insights from analytics and improve in-app onboarding experiences for your users.