Dark Launch 101: How to Release Successful Features?
What’s a dark launch and does it benefit product managers?
These are exactly the questions the article deals with, so let’s get right to it, shall we?
TL;DR
- A dark launch involves releasing the feature to a small percentage of users before everybody else.
- It’s done to validate feature ideas, test their real-world performance and reduce the risk involved in launching a potentially subpar product to all users at once.
- Dark launching is easy thanks to feature flags or toggles. You can use them to switch the feature on and off without modifying the code.
- In canary testing, your users know that they’re taking part in a test, while in dark launches, they don’t know – you keep them in the dark.
- Despite their benefits, dark launches have downsides too. They are resource intensive, complex to implement, and can lead to technical debt.
- Apart from testing, you can use feature toggles to enable localized product versions.
- Before you dark launch your product, validate the feature ideas and test their usability with fake door tests, prototype testing, and beta testing.
- Once you dark launch the feature, use product analytics to track its performance. Conduct funnel, heatmap, and session recording analysis to identify friction and usability issues.
- Triangulate product analytics data with customer feedback and track support tickets for a more complete picture.
- To see how Userpilot can help you monitor the dark launch success, book the demo!
What is a dark launch?
A dark launch is a release of production-ready software features to a small user segment. The practice is commonly used in modern software development to test new software stability and performance before launching it to all users.
How does the dark launch process work?
Basically, you enable the feature for a small percentage of users and monitor its performance or impact on user experience. As you get insights from feedback and analytics, you iterate, release it to some more users, and so on.
To be able to roll out the feature in such a way, you wrap it in a feature flag or feature toggle. They allow you to switch the feature on and off quickly without modifying the code.
What is the difference between dark launch and canary testing?
Both dark launch and canary testing allow teams to test how new functionality affects system performance or product metrics with real users, or in a production environment as a software engineer might say.
However, there is a difference between the two.
In dark launches, nobody tells the users that they’re taking part in the test or that they are using a new feature. That’s why they’re called ‘dark’.
In canary tests, on the other hand, the users are aware of that. A beta test is an example of canary testing.
What are the pros and cons of dark launch?
There are a number of obvious benefits to dark launches.
✅ Real-world testing: thanks to a dark launch, you can validate new features and evaluate their performance with real users.
✅ Reduced risk and damage control: releasing features only to a small number first means that any negative impact on user experience is limited to the small test group, and it doesn’t cause disruption for the whole user base. Also, there’s limited risk for your product reputation if things go south.
Having said that, there are some downsides as well.
❌ Resource intensive: managing different versions of the software requires considerable resources.
❌ Complexity: traditionally, dark launches required multiple production environments or complex routing configurations. It’s less of an issue now thanks to feature flags/feature toggles but it still can result in fairly complex code.
❌ Technical debt: unused feature flags can make it more difficult for the development team to maintain the code and release new software features, especially if you’re using homegrown feature flag systems.
What is an example of a dark launch?
As a product manager of a popular workspace tool for creating, organizing, and sharing various types of content, you’ve been working on an AI writing assistant.
As a part of your dark launch, you first release it to 1% of your user base and monitor its performance.
Whenever issues arise, you fix them. Gradually, as the product gets better, you start releasing it to new user cohorts and continue testing.
When to use dark launches?
SaaS organizations implement dark launching in a number of situations.
- Technical performance testing – when you’re launching a new feature or moving to new infrastructure, you can ensure they’re bug-free and stable.
- Feature idea validation – you can launch the feature and test whether it satisfies users’ needs or drives the metrics you’re trying to improve.
- Product localization and personalization – with feature toggles you can easily enable different feature versions for different user segments.
What to use before you dark launch?
When you carry out a dark launch, the risk is limited but still significant. To reduce it, there are a few more tests you can conduct to ensure you’re developing the right features and they have a solid technical foundation.
Fake door testing
Fake door testing is the least labor or resource-intensive way of testing your feature ideas. The feature doesn’t need to exist as long as you convince your users it does.
Let’s imagine you’re an Asana product manager and you want to build the Goals feature It’s a theoretical example because, to our knowledge, Asana never did such a test.
Before you do so, however, you simply add it to your menu and trigger a tooltip to prompt user engagement. And then you sit back and watch.
If there’s enough demand, you carry on to build it. If not, pivot.
Naturally, nobody likes being taken for a ride so when your users actually click on the feature, you need to explain what you’ve just done and why. Like in the modal below.
Note that it’s a great opportunity to recruit future beta testers as well!
Prototype testing
Fake door testing is a kind of prototype testing. In this case, the prototype is very low-fidelity as you have nothing under the hood.
However, if low-fidelity testing gives you promising results, you can invest more effort into the idea and make your prototypes progressively more realistic.
For example, the next step could be pretending that the feature exists but doing all the work manually behind the scenes. This kind of experiment is called Wizard of Oz.
Or you can use other existing solutions and botch them together to achieve the desired functionality before you develop your own code.
Beta testing
Beta-testing is often seen as an alternative to dark launching, but there’s no reason why you can’t combine both of these approaches.
So, before you dark launch your feature, beta test it first with an even smaller group of users. This could be a few of your most dedicated power users who will be happy to take the feature for a spin and give you actionable feedback.
In this way, you will ensure that the feature is already robust when you release it.
How to monitor the impact of dark launches?
Once you toggle the feature on for the test group, you need to evaluate its performance.
Gather user feedback regarding their experiences with the new features
As soon as a user interacts with the feature, trigger an in-app survey to ask them for their feedback. Such contextual surveys are a great source of valid insights because the experience is still fresh in users’ minds.
If the feedback is positive, that’s great. However, when users submit negative feedback, follow up with them to know in detail about their issues with the new feature.
Monitor product analytics to identify friction points
With product analytics, you can objectively test how well the feature performs and identify ways to improve it.
Here are a few methods and techniques you can use.
- Funnel analysis helps you assess conversions from one stage of the user journey to the other. If they slow down or drop off at a particular stage, that’s where the friction is and you need to have a closer look.
- Heatmaps enable you to analyze user engagement with features on your page. The warmer the color, the higher the engagement, so they’re super easy to analyze.
- Session recordings are even more detailed because they reveal every single user action – the clicks, text infills, scrolls, hovers, you name it. This makes them a perfect tool for assessing feature usability.
Track the number of customer support tickets and bugs
Apart from tracking user in-app behavior, pay attention to support tickets and bug reports. That’s how you will be able to identify technical issues.
Monitoring customer support calls and chats is a good way to get qualitative data from customers, and it’s dead easy to automate it with AI tools these days.
Conclusion
A dark launch is an easy way to reduce the risk involved in feature releases. By enabling the feature to a small number of users, you can still test it under real-world conditions. If things go wrong, though, the damage is limited.
If you want to see how you can monitor the effects of your dark launches with Userpilot, book the demo!
