{"id":195370,"date":"2026-06-10T06:35:05","date_gmt":"2026-06-10T06:35:05","guid":{"rendered":"https:\/\/userpilot.com\/blog\/amplitude-a-b-testing\/"},"modified":"2026-06-15T11:56:33","modified_gmt":"2026-06-15T11:56:33","slug":"amplitude-a-b-testing","status":"publish","type":"post","link":"https:\/\/userpilot.com\/blog\/amplitude-a-b-testing\/","title":{"rendered":"Amplitude A\/B Testing in 2026: What It’s Built For, and Where It Falls Short"},"content":{"rendered":"

<\/p>\n

A few months ago, someone on our team suggested cutting all in-app A\/B testing<\/a> because one customer said pop-ups were annoying. Before making that call, I ran a quick, controlled test in which half of our users saw a tooltip pointing them to a new feature, while the other half saw nothing.<\/p>\n

The result was hard to argue with. Users who saw the tooltip adopted the feature at a 200% higher rate, and the data ended the debate faster than another meeting would have.<\/p>\n

What made the test especially useful was that I ran it without filing an engineering ticket or writing a single line of code. That\u2019s the part many teams miss when they talk about A\/B testing in SaaS. Not every experiment needs to be a server-side rollout, a feature flag, or an engineering-led release test.<\/p>\n

SaaS experimentation now falls into two broad lanes. One is server-side feature experimentation, built for engineering teams managing code releases, feature flags, and product infrastructure at scale. The other is in-app experience testing, built for product, growth, and lifecycle teams that want to test what users see, how they\u2019re guided, and which messages drive action.<\/p>\n

Amplitude has historically been stronger in the first lane, and it continues to lean further into engineering-led experimentation. That matters because the Atlassian State of Product 2026 report<\/a> found a strong correlation between high-performing product organizations and proactive investment in experimentation. The pressure to build a testing practice is real, but the tool only helps if it fits how your team actually works.<\/p>\n

In this article, I\u2019ll help you figure out whether Amplitude\u2019s A\/B testing capabilities are right for your team.\"demo<\/a><\/p>\n

The two Amplitude experimentation products<\/h2>\n

Amplitude A\/B testing has two separate products that serve different teams, run on different infrastructure, and unlock at different plan tiers.<\/p>\n

Web Experimentation<\/h3>\n

Amplitude Web Experimentation is a browser-based testing product. It’s available on every plan, including the free Starter tier (up to 10K MTUs), gives you a no-code visual editor for front-end variant creation, and lets you run experiments on your website or app without writing much code.<\/p>\n

Free and Plus plans ($49\/mo, up to 300K MTUs) each allow 1 active experiment. The Growth plan (custom pricing) unlocks the code editor and more concurrent tests. It’s reasonable for landing pages, website UI changes, and browser-based conversion testing.<\/p>\n

Feature Experimentation<\/h3>\n

Feature Experimentation is a server-side product built around feature flags, and the full version requires the Growth plan. Advanced statistical controls like mutual exclusion groups (which prevent concurrent experiments from contaminating each other’s results) and multi-armed bandits (which continuously shift traffic toward the best-performing variant) are Enterprise-only add-ons.<\/p>\n

While feature flags and A\/B tests are related, they aren’t the same thing. A feature flag controls the delivery of a variant (which version of a feature users see) based on user properties, plan type, or demographics. An experiment is the statistical layer that measures whether the variant actually moves a metric. In Amplitude, a flag can double as an experiment, but not every flag is one.<\/p>\n

\"Amplitude's<\/p>\n

Amplitude acquired Statsig to strengthen its position in this space, though Statsig’s original engineering and data science team did not migrate as part of the deal. That means Amplitude is building out Statsig’s statistical framework without the people who designed it. If you’re a product marketer or growth PM evaluating Amplitude for in-app testing, that context matters because the platform’s roadmap points toward engineering-heavy experimentation.<\/p>\n

How does Amplitude A\/B testing work?<\/h2>\n

A\/B testing, also known as split testing, is a method for comparing two versions of a feature or content to determine which performs better, typically by changing only one parameter at a time. Here’s how that process runs in Amplitude from setup to results.<\/p>\n

Define your goals and target audience<\/h3>\n

Before creating a test, pick a single, specific success metric: “7-day activation” or “channel connected” rather than generic engagement. Amplitude allows you to track multiple metrics in an experiment, but only one primary goal can be set for evaluating results, so choose your most critical metric upfront.<\/p>\n

Event-based tracking is how Amplitude analyzes specific actions users take, such as completing a sign-up or clicking a button to add an item to the cart. For audience targeting, Amplitude supports cohort-based segmentation<\/a>, behavioral targeting, user property filters such as demographics or plan type, and both random and targeted allocation by user or device ID.<\/p>\n

\"Amplitude's<\/p>\n

Design your variants<\/h3>\n

Amplitude automatically creates the first variant, which it calls “treatment.” There are three ways to define what changes between variants:<\/p>\n