Product experiments help product managers and their users get better results.

It’s that simple. You need to be running them.

In this blog, we’ll show you WHY and HOW.

Product experiments: A/B testing

Source: apptimize.com

Experimentation and A/B testing are bread and butter in marketing, but in product…it gets complicated.

While most marketers are familiar and well-equipped with code-free tools that allow them to run such experiments, product managers often think they would need to involve development resources each and every time they want to test something.

And in companies where dev resources are scarce, product experiments are often dropped altogether. That’s a bad mistake.

Product experiments CAN and WILL improve your customer activation rates, engagement levels and ultimately user retention – so if you’re not doing them, you’re leaving money on the table.

In this blog, we’ll show how product managers can design and run product experiments – WITHOUT CODING – to enhance user experience in a data driven way.

  1. Why Product Managers Need Product Experiments
  2. Goal Setting For Product Experiments
  3. Product Experiments That Will Boost Your Key Metrics Without Coding
  4. A Note On Running Product Experiments Correctly

Why Product Managers Need Product Experiments

Product roadmaps are intelligent guesses at best, and blind gambles at worst.

Product roadmaps are best guesses at best

Source: researchgate.net

In order to offset the risks involved in shipping the wrong features and spending a lot of time and resources on something your users may not use, you can run product experiments to help you decide how to prioritize the different items on your roadmap.

But there are plenty of other reasons why you should run product experiments:

  • Show you which designs, navigational controls, content etc lead to better conversion or more user success
  • Help you to understand WHY users behave in the way they do, enabling you to design better features in future
  • Disprove false ideas about your users, their needs, what they value and how they think
  • Resolve arguments between people who “feel” too strongly that their instincts are correct…

Product experiments give you real data about how users behave in your app.

Instead of making assumptions, they allow product managers to learn whether customer success rates would improve:

  • If Calls To Action were in different locations or in different colors
  • If navigation options were in a different order
  • If a checklist showed them how far they had progressed through a task
  • If contextual tooltips popped up when it looked like they were struggling to understand how to progress
  • If a different user journey more suited to their precise needs would be more valuable

If you’re new to this blog it may surprise you to learn that there are easy-to-use, easy-to-deploy, no-code tools that enable you to run in-app experiments of this sort.

If you’re a regular reader or a Userpilot user, it won’t surprise you at all – because that is one of Userpilot’s main features.

Userpilot audience segmentation

Source: userpilot.com

Goal Setting For Product Experiments

But wait.

If you want to extract actionable insights from user behavior, you have to put some thought into the tests you run.

We’ll dig a bit deeper into good experiment design in the final section of this blog, but at this point I want to look at SETTING GOALS.

user adoption userpilot

Source: userpilot.com

The best place to start is with your user journey. Here’s our adoption journey map, for example.
P.S. Want to learn more about adoption? Join our Product Adoption School:

As a product manager, your aim is to push more users along this journey, faster.

We’ve talked about this a lot in other blogs, but moving a user from Aha! to Activated, for example, is a matter of getting them:

  • From a position where they can SEE the value they’ll get from using your product
  • To a position where they are EXPERIENCING that value for the first time
Aha to Activation

Source: justinjackson.ca

So a user at Aha! can see the Better Life on the other side of the Obstacle, but they have yet to climb over it…

For Userpilot, our Activation point is when a user installs our code snippet onto their website.

At that moment, they can start designing product experiences, contextual in-app help using our visual interface – AND see them work in practice.

So for us, the Obstacle users need to overcome is anything and everything that stops them from deploying that snippet.

For example, a lot of Userpilot’s end-users do not have the technical knowledge or access rights to install the code. Hence, they need to outsource it to their developers.

Or, we could provide comprehensive help pages explaining how to do it yourself – including how to integrate data extracted from other apps using Segment.

Userpilot FAQs

Source: userpilot.com

Which option would get better results?

Product experiments mean you don’t have to guess or assume.

You can give the help to one set of users and not to another and SEE which group Activates more.

Setting experiment goals

Souce: Alexandra Levich, Google

As Alexandra Levich of Google explains in this video, the best way to design product experiments is to Think Big:

  • Define your goals by identifying an obstacle to user journey progress
  • Envision the future by sketching out what things would have to be like for that obstacle to no longer exist (or at least to be less of a blocker)
  • Brainstorm ideas for hypotheses to test. That is, changes to the product experience you could make which you think may improve user success

To sum it up: your product experiments should be aimed at reducing users’ pain points.

Product Experiments That Will Boost Your Key Metrics Without Coding

In this section, we’ll look at four different types of experiment you can discover what moves the needle on various different problems.

  1. Boosting User Activation
  2. Boosting Feature Adoption
  3. Improving User Retention
  4. Increasing Revenue

Boosting User Activation

Onboarding checklists are widely believed to make a big difference to Activation rates.

Userpilot: checklists

Source: userpilot.com

By showing a user all of the steps they need to go through to hit a key value milestone – and how many are left to complete – checklists help users along their journey to Activation.

But what is the best way to use a checklist?

Social media scheduling tool Postfity used Userpilot to find out.

At the start, new Postfity users were presented with a checklist with none of the items marked as completed. This was giving them a 67% engagement rate but only a 6% completion rate.

Our Customer Success Team suggested trying this with some items on the list pre-ticked, so as to leverage the psychological bias most people show to completing tasks that they have already started.

So Postify added two alternate versions: one with one dummy items already ticked on the checklist; and one with two items.

Postfity tests checklist completion

Source: postfity.com

New users were shown one of the three checklists at random.

The results were striking.

Product experiment results

Source: userpilot.com

The checklist with two items pre-ticked outperformed the other options in live tests – even though all the items were the same.

This can be attributed e.g. to the Zeigarnik effect.

SalesFlare has experimented with gamification – providing incentives for completing onboarding tasks.

SalesFlare gamifies onboarding

Source: Jeroen Corthout, Salesflare

As co-founder Jeroen Corthout explains in this video, his product needs new users to go through 12 steps before they start to see value (ie before Activation).

As a CRM tool, Salesflare shows the value the more you use it, the more your colleagues use it and the more you import your own data.

So, they ran an experiment to see if offering extra days onto users’ free trials in exchange for completing tasks helped Activation rates improve.

At the time of recording the video, Jeroen had collected the following insights:

Source: Jeroen Corthout, Salesflare

That is super-valuable intelligence that Salesflare can use to keep improving the user experience with new product experiments.

Gamification can be even simpler than this though.

When users reach Stripo’s Activation point, they see a little celebration image, like this:

Stripo gamifies onboarding

Source: stripo.com

Another onboarding experiment would be to test the impact of a welcome screen.

Show your new users one of the following alternatives at random:

  • No welcome screen at all – confront them with a dashboard and leave them to get on with it
  • A welcome screen that provides some useful “getting started” information or invites them to take an interactive tour
Welcome screen in Userpilot

Source: userpilot.com

  • A welcome screen that asks the user for information about themselves, so that you can tailor their experience to their specific needs through segmentation

Our research found that only 60% of SaaS products use welcome screens. It may be the case that some tools don’t need one – their users prefer to just get on with it.

You can only tell what’s best for your users by running product experiments.

Boosting Feature Adoption

Perhaps you already know that users see more value, use your product more regularly and stay as paying customers for longer when they use more than one of the product’s features.

Promoting additional feature adoption is a challenge for many SaaS companies because they throw all their efforts into Activation – getting to first value.

But if your onboarding ends there, you are missing out on the opportunity to move users further along the journey and to deepen your relationship with them.

The question is HOW best to promote those additional features.

Well, this is another area where product experiments can provide actionable insights.

In Userpilot, we let you incorporate Driven Actions into your experiences.

Userpilot's driven actions

Source: userpilot.com

These are contextual “hints” that allow you to push users towards certain desired actions. For example:

  • (Native) Tooltips – Information about certain options on-screen can be conveyed with pop-up or slide-in boxes. These can be present on load or perhaps triggered when a user shows intent by their mouse movements (eg hovering over a spot)
Product experiments: driven actions with slide-ins

Source: userpilot.com

  • Hotspots – These are a little bit more subtle. The button you want the user to click changes color or pulsates in some way that catches the user’s eye and makes them pay attention
Product experiments: driven actions with hotspots

Source: userpilot.com

Alternatively, you could promote feature engagement in one of these other ways:

  • A secondary onboarding checklist. Once users have completed the basics, introduce them to a new set of tasks to complete.
  • Gamify further learning by including rewards and presenting it as a matter of “leveling up”. Duolingo uses this brilliantly to motivate users and promote further learning
Gamification with Duolingo

Source: duolingo.com

  • In-app marketing – Run webinars and other tailored help sessions to enable users to get more out of the product

Once again, there are any number of ideas that might encourage your users to adopt new features.

But you can only find out which are best when you experiment.

Improving User Retention

What if there was a way you could turn around churning users and reinstate them as happy customers?

Well, maybe there is…

Many SaaS companies ask users to complete a survey when they cancel their subscriptions. Here’s one from Get A Newsletter:

Product experiments: survey exiting users

Source: getanewsletter.com

There’s not much you can do if your customer is closing their business, but for almost every other option there may well be something you could do to improve matters.

You have two options here:

  • Follow the feedback up and try to reverse the churn by offering to fix the problem
  • Don’t follow it up with the outgoing user – but take their comments on board to improve the experience for all future users

Get A Newsletter opted for to follow up feedback, like this:

Product experiment: following up exiting users

Source: getanewsletter.com

Which works best?

There’s only one way to find out…

Show one option to one set of users, one to another and the third to a third.

Then analyse the relative performance of each group against one core metric – in this case, churn.

Increasing Revenue

It’s also simple to experiment with upselling and cross-selling to your existing users.

One way is to impose usage limits on certain features.

Postfity, for example, allows users to schedule up to 10 posts on its free plan.

After that, users are directed to upgrade their subscriptions.

Many SaaS products display all of their available features on new users’ dashboards, but then restrict access to some of them to premium users.

Dropgalaxy: premium users only

Source: dropgalaxy.com

Alternatively, the upsell call to action can be triggered contextually. Here is one way that Hubspot does it:

Hubspot: upselling

Source: hubspot.com

Would a modal, a slide-out or a banner get better results?

Or perhaps you could show some users case studies or testimonials from happy customers, explaining how they achieved value with your service?

Again, the key is not to just try things out without a plan.

You need to rigorous, experimental approach to uncover insights you can depend on.

Run experiments carefully

Source: userpilot.com

And that’s what we’re going to look at in our final section.

A Note On Running Product Experiments Correctly

It’s very easy to run BAD experiments, where the data you collect is worthless.

You probably know the basics of scientific method, so I won’t go over basic concepts like Controls and Variants, statistical confidence, randomized samples, etc here.

Instead, here are some lessons I’ve learned from running product experiments myself and helping other people to do so with Userpilot:

  1. Design your experiments to test hypotheses. Don’t fit hypotheses into experiments you’ve already set up!
  2. Set a time limit and stick to it. Don’t end an experiment early because the results seem to be leaning one way. They could change!
  3. Segment your audience properly. Don’t try out variants on regular users who are already familiar with your app. Restrict experiments to new users who have no preconceptions
  4. Even if you get a positive result, run the same test again to see if you get the same results. The more constant the results, the greater confidence you can have that the change you’ve made is causing the change in results
  5. Supplement your data with qualitative feedback, such as interviews, field studies etc. This will help you understand WHY the changes you made led to better results
  6. Don’t just carry out single tests. Your experiment program should include multiple tests per KPI.
  7. Avoid “local maxima” problems. If your Variant and Control are very similar, the gains you achieve by switching may not be as great as if you had tested the Control against something else.

But as Alexandra Ciobotaru of Novorésumé explains in this video, the BEST results come from an integrated cycle of testing, improvement and retesting.

Here’s a screenshot showing her Experimentation Framework, which she outlines in the video (15:08).

Product experiments: a framework

Source: Alexandra Ciobotaru, Novorésumé

Building a culture of product experiments into your company will save you money and time, because it pushes every decision to be data-driven.

If you can achieve that, you’re well on your way to great success.