Fake Door Testing: What Is It and How to Make An Effective Fake Door Test

What is fake door testing? And how can product managers use it to inform decisions about product development?

The article will give you an understanding of how fake door tests work, how to design them, and what to watch out for not to turn your users against you.

TL;DR

• Fake doors testing is a technique used to assess market demand for a product before investing in its development.
• It involves inviting customers to use a feature that is not ready (but that may be in development) to see how many of them will be interested
• Key benefits of running fake door tests include validating your product or new feature idea before committing to its development, refining your pricing strategy prior to the launch, and enlisting beta testers
• Careless use of fake door tests can create disappointed and disgruntled users, and damage your credibility.
• To avoid alienating your users, be transparent and honest. Show users the potential value of the new features/products, and try to engage them in the development process (e.g. by participating in beta testing).
• Limit the segment size for your tests to minimize the fallout.
• You can run fake door tests for potential customers (via your website) or existing users (via in-app messaging)
• Fake door testing steps: segment your audience, decide on which landing page your fake test should trigger, create the fake door test content, build the follow-up and track the success of the test.
• When using a landing page, test one thing at a time to enable reliable tracking.
• For in-app tests, be careful not to frustrate your users with too many fake doors.
• Tools that can help you run fake door tests include Unbounce for creating landing pages, Figma for designing the experience, and Userpilot for in-app messages.
• Want to build in-app tests and track results? Get a Userpilot demo and see how you can do it without having to code.

What is fake door testing?

Fake door testing (also known as painted-door testing) is a lean technique to quickly test the viability of new ideas before committing resources to their development.

It is used to gauge customer interest in a product, feature, or service that doesn’t actually exist, but which you might be considering.

How does a fake door test work (and why)?

In fake door tests, the customer gets an invitation to use a product or a feature. The catch is that the product doesn’t exist yet!

That’s why, when they enter the fake door (be it a CTA button, an in-app notification, or a fake advertisement), they are taken to another page that reveals the truth – the product is not there yet but could be coming in the future.

If you have a product idea but are not sure how much traction it can get, the data you collect from the test can help you make the call. By looking at metrics like CTR (Click Through Rate), you can decide which feature or product ideas you should consider.

Apart from giving you the data, a well-executed painted-door test can help you get new customers, spark their curiosity and increase the loyalty and commitment of the existing users.

When is it beneficial to use fake door testing?

There are a number of situations when you might want to consider running such a test.

Let’s look at some of the most important ones.

Use fake door testing to validate your product idea

There have been too many products that failed because the MVP turned out not to be that viable after all. Conducting painted-door tests allows you to test the value proposition of the product before spending any money on R&D.

If you would like to see how it works in practice, have a look at the example of Buffer, a social media management tool.

Buffer created a landing page with the details of the product and a ‘Plans and Pricing’ button. This was to assess the initial interest of potential users.

Clicking the button, however, took the users to another page that informed them the product was in fact still in development.

There was also an option to sign up for email updates on the page. Doing so would indicate interest in the product.

After introducing this two-step test, the product team was able to validate the product idea and initially only had to invest in those two pages.

Find your product pricing sweet spot with fake door testing

In addition, if enough users are ready to commit to your product, fake door tests can also determine what price they are willing to pay.

Buffer did exactly that.

As a second step to the original test, they offered users the option of choosing the pricing plan that was right for them.

Use fake door testing before developing a new product feature

You might consider doing fake door testing for potential new features if your current product is proven to deliver value. In this way, you will be able to test the validity of your assumptions and avoid creating something that no one will value.

One way of doing it is with an in-app experience, for example, a small tooltip just like the one below.

You would track how many users click on the button to estimate the demand for the potential new feature and assess whether it is worth building it or not.

Disclaimer: the Goals feature is already available in Asana and in fact was never tested with a tooltip (as far as we know), so this example is just to illustrate the possibilities.

Build a list of beta testers with fake door testing

You can also use fake door tests to build a list of beta testers for the new feature.

In the theoretical Asana example above, clicking on ‘Add a goal’ could trigger an in-app modal informing the users that this was a test, but inviting them to subscribe to your beta testing list for when the feature is available.

Once the users sign up, make sure to keep them up-to-date. If the test fails and you decide not to build the feature after all, always follow up and let them know.

Risks and disadvantages of running fake door tests and how to avoid them

Running painted-door tests carries some risks too. The word ‘fake’ is the clue here.

Customers could simply feel cheated and form negative associations with your product.

To avoid disappointing users and damaging your credibility, consider the following:

• always apologize for potential disappointment and distress
• reassure them that you are really working on the feature (or seriously considering it)
• thank them for their time
• explain the reasons for your actions
• give them a chance to provide feedback
• invite them to take part in the beta testing once the product or feature is ready
• stress the value the new development could add for the users
• carefully choose the size of the testing cohort to limit the potential damage (while still getting enough data)

How do you make a fake door test?

If combined with paid ads, you can use fake door tests not only to validate your product idea but also to pitch it to potential customers and expand your user base.

The process consists of 4 key steps:

1. Create a landing page to introduce the feature or product.
2. Use paid advertisements to drive traffic to the landing page
3. Create another page explaining to users that the feature is still in development.
4. Track the user activity and see how many people visited your website and how many followed the CTA.

How to run an in-app fake door test?

We’ve looked at the four steps you need to do and run fake door tests using landing pages like Buffer did.

If you’re looking to run fake door tests in-app, you need to follow these five steps:

1. Segment your target audience using product usage analytics.
2. Decide on which in-app page should your test trigger.
3. Create the test content (using a modal, a tooltip, or other UI pattern)
4. Build the follow-up message and experience.
5. Measure the success of your painted-door tests.

1. Segment your target audience using product usage analytics

If you want to test a potential new feature, make the test as relevant as possible. This means selecting the right test audience.

Let’s look at an example.

You’re a backlink-building tool meant for building partnerships and tracking link-building activities. You want to launch an advanced feature that will allow users to build new partnerships by connecting with other partners in the network, not just adding their own contacts.

In this situation, you’d want to see if the current users who are already using the partnership feature would be interested in this additional product functionality. Your test audience segment could look like this:

2. Decide on which landing page your fake test should trigger

Next, you need to figure out where in your app and when you should trigger the test.

You could use a tooltip like in the Asana example above or perhaps consider adding a slideout when the user navigates to the Partnership page.

This will look something like this.

It’s essential to be contextual and show your message where and when it’s relevant for the user.

3. Create the fake door test content

This bit could make or break the test.

First, you need to convince the user to engage with the message. To do that, you could indicate what pain points it solves for the user and how it adds value.

Adding screenshots, wireframes or a product demo video can drive more engagement.

Once the users engage and realize that the product is not yet ready, you must always explain why this is happening and how you are working on it.

This will help you avoid user frustration and disappointment, and perhaps even create a sense of anticipation.

At this stage, you can present a possible roadmap, let them register for updates, or to take part in beta testing.

4. Build the follow-up

Following up with the users that have engaged with the feature, signed up for the updates, or volunteered for beta testing is essential.

Keep them in the loop about how the work is going and most importantly – deliver on your promises.

Otherwise, your credibility will suffer badly.

5. Measure the success of your fake door testing

The final step is essential as this is what enables you to make an informed decision on whether it’s worth building the feature or not.

You define what you are looking for even before starting the experiment so that you can easily determine if the test was a success or not.

Things to track include how many users clicked on your tooltip or modal, and how many of those signed up for a beta test.

Also, look at any feedback you may receive, both positive and negative (but prioritize the data on actual user behavior).

A tool like Userpilot will help you track how many users see, engage with, and complete the in-app flows (these include tooltips and modals).

How many fake doors should you build?

If you’re building a painted-door test using landing pages, test one thing at a time. Otherwise, you won’t be able to track results properly.

If you are doing the tests in-app, make sure not to annoy users with too many pop-ups, banners, or modals, especially if they all lead to fake doors.

Run the tests only when absolutely necessary and don’t test multiple things using the same user segment.

Tools for building fake door tests

Are you feeling tempted to try fake door testing? If so, there are a few great tools that will help you design your tests:

• Userpilot: for building and analyzing results of your in-app tests
• Unbounce: for building landing pages without having to code
• Figma (this might already be in your product toolkit): for designing your tests and user experience

Conclusion

Fake door tests, if done well, will let you validate your product ideas without wasting precious resources.

If you would like to see how Userpilot can help you carry out your fake door tests with in-app experiences, get a demo here!