Marketing Experiments in SaaS: Different Experiment Ideas To Try and How To Run Them

Marketing experiments give you an insight into how well marketing strategies will perform before implementing them. But how should you run them?

In this article, we’ll start by exploring how to conduct a marketing experiment and give you different experiment ideas to try in your SaaS business.

So let’s jump right in.

TL;DR

• A marketing experiment is a type of market research to validate the effectiveness of an existing marketing strategy or discover a new one.
• Marketing experiments help you to innovate new ideas, try new marketing strategies, identify marketing mistakes, make smarter business decisions, and optimize campaigns to maximize performance.
• Marketing testing is a technique used by businesses to evaluate the viability of their strategies before implementing them.
• There are seven steps to conducting a marketing experiment:

1. Brainstorm experiment ideas
2. Analyze existing data
3. Make a hypothesis
4. Choose the audience
5. Select metrics to measure success
6. Run the experiment
7. Analyze results

• Here are five marketing experiments types you can try in your business:

1. In-app onboarding experiments
2. Product messaging experiments
3. Email marketing experiments
4. Paid media experiments
5. Case study experiments

• Tools like Userpilot, Wynter, and Hotjar can help you run successful marketing experiments.

What is a marketing experiment?

A marketing experiment is a type of market research to validate the effectiveness of an existing marketing strategy or discover a new one.

Why are marketing experiments important?

Imagine spending days with your team trying to come up with a marketing campaign to increase sales and awareness.

Fortunately, you and your team came up with a marketing campaign idea that looks so good on paper and you’re sure that is the one.

So what’s next? Implementation, right?

You and your team decide it’s time to implement the campaign, and you spend so much money and time on turning this strategy into real-life actions only to see that your potential customers are not responding well to the campaign.

All that money and effort down the drain. Yikes!

However, imagine if you had tested your marketing campaign first. This would have saved you time, money, and effort on an ineffective campaign since you would have known what your target audience wanted.

Marketing experiments extend beyond validating existing campaigns. Marketing experiments can also be used to innovate, test brand-new ideas, and make smarter business decisions.

Marketing experiments vs. marketing tests

Now that we’ve understood the importance of marketing experiments let’s look at the difference between marketing experiments and marketing tests.

A marketing test is a technique businesses use to evaluate the viability of their products, services, or strategies before launching them on a large scale.

Marketing testings often include the following:

• A previous champion
• A challenger (the product, service, or campaign)
• A set budget and known timeframe
• An expectation that there will be a winner

On the other hand, a marketing experiment is a type of marketing research with one of two aims: to discover new strategies or validate existing ones. Marketing experiments include the following:

• Hypothesis
• Two or more challengers
• A set budget but an unknown time frame
• An expectation that it might be successful or it might fail

How to conduct a marketing experiment?

So you’re ready to start conducting marketing experiments, but how do you go about it? Here are seven steps to conducting successful marketing experiments:

Brainstorm experiment ideas

The very first step in the marketing experimentation process is brainstorming ideas. What do you want to experiment with and why?

Don’t know where to start with brainstorming ideas? Have team meetings in your department, exchange ideas, and take daily notes. Then sum up the notes and prioritize the ideas based on your company objectives.

Analyze existing data

Once you identify your goals and the KPIs you’re trying to influence; the next step is to analyze historical data. What strategies did you use in the past? What worked and what didn’t?

Use analytics tools to dig into your historical data and identify potential reasons why some of your strategies didn’t work.

Then, analyze where you are in marketing, how you got there, and what you’re trying to achieve.

Make a hypothesis

The next step is to create a hypothesis. Your hypothesis is what you’re trying to prove, and it should be testable.

When creating a hypothesis, most marketers fail to be specific. Suppose you said, “We need a new CTA copy for our website.”

But what happens next, and how do you know that you really need a new CTA copy? What’s wrong with the last one?

Here the hypothesis is subjective and not testable.

A better hypothesis would be, “Changing the CTA copy from “Submit” to “Join Our Community” will increase sign-ups by 5%.”

You can test this hypothesis by changing the CTA copy, and if you get more sign-ups, it means that your hypothesis is correct, and users need more clarity. If you get fewer sign-ups or do not record any changes in sign-ups, your hypothesis would be incorrect.

Using that data, you can follow up with another test like changing the CTA copy to something more basic like “Get Started” to see how users react.

Choose the audience

You don’t have to run the experiment on all your customers if it’s not necessary. You can create customer segments based on your specific goals and run marketing experiments with them.

Don’t know how to get started with segmentation?

Userpilot offers many practical ways to segment users; NPS score, user attribute, experience flows, demographics, in-app behavior, and custom events.

Just create segments with the tool and run experiments with only them.

Select metrics to measure success

The next step is to select the proper metrics that will indicate if your hypothesis is true or not.

Your metrics can be anything, including traffic, engagement, click-through rate (CTR), reach, conversion rate, and more.

Run the experiment

Once your hypothesis is ready, and you’ve selected your metrics, the next step is to conduct your experiments.

When conducting marketing experiments, you should use A/B testing. Why?

You can compare the original version of an item with the new version by using A/B testing. Additionally, it helps you determine whether your hypothesis is correct or incorrect by comparing your change to the original.

Userpilot’s A/B testing feature makes it easy for your team to make informed decisions about a marketing campaign and measure the impact of different strategies on your growth goals with higher accuracy.

Analyze results

The final step is to analyze your results. You don’t have to wait for the experiment to end before checking its results. Look at the different metrics you set for your campaign; they are the indicators that will determine the validity of your hypothesis.

Did the experiment drive the expected results?

If not, is there anything you can do to optimize it? If the answer is yes, you should take action to optimize the campaigns.

Different types of marketing experiments

Now that you know how to run marketing experiments, it’s time to figure out what type of marketing experiment you want to do. Here are five examples of different marketing experiments to try out in your business:

In-app onboarding experiments

In-app onboarding is crucial, but most businesses fail to do it right.

According to research by Upland, 25% of users abandon an app after one use.

This means one thing – marketers are not paying so much attention to in-app onboarding. There are several reasons why users will abandon your app after one use:

• The app isn’t relevant to their needs.
• The user experience is bad.
• They don’t know how to use your app.

A good in-app onboarding experience gives users a clear picture of what your app does and how to use it.

Your experiments and testing should be aimed at helping your users understand your app’s value and reach milestones faster. On that note, here are a few in-app onboarding experiments you can run:

Test welcome screens

In-app welcome screens can significantly impact your user activation, satisfaction level, churn, and retention rate. Here is an example of how you can conduct a marketing experiment and segment users into two groups.

Don’t use a welcome screen for the first group; welcome them with a simple dashboard and leave them to figure it out on their own. On the other hand, welcome another group with a welcome screen that provides helpful “getting started” information or nudges them to take an interactive tour.

Then watch closely to see how different each group acts.

Test onboarding flows

Onboarding flows are a way to introduce your brand-new users to your product or new features. The goal of every onboarding flow is to shorten the time to value and get users to the Aha! Moment as quickly as possible.

You want to identify which onboarding flow version leads users to an ‘Aha!’ moment the quickest and keeps them engaged.

For this, you can show slide-outs to half of your users and tooltips to the other half. You can also experiment with different CTAs, in-app experiences, and progress bars to determine which version works better.

With Userpilot, you can create different UI patterns for your onboarding flows. Choose from modals, slideouts, tooltips, banners, and checklists, or combine them together.

In the example below, we tested if adding an in-app checklist will have an impact on our feature adoption goal. Based on the preliminary results, our goal completion went to 60% from 29%.

You can also use Userpilot’s feature tagging to tag any UI pattern that will trigger when users click on them. This way, you can see how users engage with your UI patterns.

Product messaging experiments

Product messaging has a big influence on your customer’s purchase decision.

You can display different versions of your product messaging to see which resonates the most with your users. Be sure to test the design and copy length. You could also test different formats – for example, if feature adoption is higher in the group that got a pop-up announcement rather than introductory tooltips, then you’ll know which approach works best.

Segment users based on jobs they need to complete, milestones they’ve reached, and other characteristics, then show relevant messages according to each segment:

Email marketing experiments

Email marketing is one of the best digital channels to grow and nurture your leads. So it’s worth running experiments to see what your audience will respond best to.

Test the subject lines

Your subject line is the first thing users see before they decide to open your email. If your subject line doesn’t resonate with your users, they won’t open it, and you’ll miss out on potential conversations.

When you run marketing experiments, you should try different subject lines for your email to see what works. For example, you could personalize one email version by adding the recipient’s name to the subject line and see how your users respond.

You could also get creative by experimenting with emojis on your subject lines and see if it gets more open rates.

Test the content

If your users are opening your emails but aren’t doing anything after that, there might be an issue with your email content.

You should experiment with the email’s headline, CTA, content, and images and see how your audience responds. However, it’s important to focus on one aspect at a time, so don’t try to experiment with everything all at once; pick one focus and experiment with it.

Paid media experiments

Using paid media incorrectly can cause you to lose money and generate very few leads and return on investment.

You don’t want to experience this with paid ads.

You need to experiment with your ads to make the most of them. Rather than going with gut feelings and launching one marketing campaign, create two or more.

Then add different headlines, CTAs, and copy in those marketing campaigns and show different variations to different potential customers segment until you start getting data for key metrics like conversion rate, CAC, etc. Once you have this data, you can identify which marketing campaign works and which doesn’t.

Here is a paid media example from Datasine as they are A/B testing two different visuals.

Case study experiments

The case studies on your site influence your customer purchase decisions – as it helps you demonstrate credibility and authority.

Take a look at existing testimonials instead of making assumptions about what your audience might like.

Include quotes from clients in various roles rather than just those in executive management. Or it might be necessary to replace your case study with an updated one.

Best tools to run successful marketing experiments

Let’s take a look at some of the best tools to run successful experiments:

Userpilot – for in-app onboarding experiments

Userpilot is a code-free product experience platform that helps product teams deliver personalized in-app experiences to increase growth metrics at every user journey stage.

The special ingredients to Userpilot regarding marketing experiments are its ability to analyze in-app customer behavior and perform A/B testings on in-app experiences.

With Userpilot, marketers can segment users based on specific attributes or in-app events and create onboarding flows to get customers to get users to the ‘Aha!’ moment.

Userpilot has 3 pricing packages: $249/mo (Traction), $499/mo (Growth), $1000/mo for Enterprise plans.

Wynter – for messaging & copywriting experiments

Wynter is a message research tool to get customer feedback on how your messaging resonates with them.

It conducts messaging testing for ads, website, and email copy. As a result, you can also determine where your messaging falls short and fix it in order to attract more customers. Wynter helps you determine your message-market fit, get internal consensus, identify your customers’ pain points and desires, and validate your messaging.

Pricing: $899/test (Wynter Starter), $1489/mo (Wynter Monthly), $19,000/yr (Wynter Pro).

Hotjar – product experience experiments

Hotjar is a user feedback and behavior analytics software that provides insights that show how users behave on your site and why it is happening.

It crams a range of essential features, including heatmaps, conversion funnels, feedback pools, session recordings, surveys, form analysis, recruit testers, and incoming feedback.

Pricing: The tool comes in two different plans:

Observe: Pricing starts at $0 (Basic), $39/mo (Plus), and $99/mo (Business).

Ask: It starts at $0 (Basic), $59/mo (Plus), and $79/mo (Business).

Conclusion

Marketing experiments are a great way to produce the best strategy for your business. We shared five different experiment ideas you can try for your SaaS.

Are you looking to run in-app experiments code-free? Book a demo call with our team and get started!