A/B testing: how to test changes on real users

An A/B test is an experiment in which users are randomly split into groups: one sees the current version, the other a changed one, and a target metric is compared. It is the only reliable way to know whether a change improved results or they rose for some other reason. A test makes sense when traffic is sufficient, the hypothesis is written down in advance and the sample size is calculated before launch.

Here is how to prepare a test, how many users you need, which mistakes produce false results and how to implement tests technically.

Why do A/B tests matter?

Without an experiment you cannot separate a change's effect from seasonality, advertising, news and random noise. Conversion rose after a redesign — but perhaps a promotion started the same week. An A/B test shows both versions at the same time under identical conditions.

• testing hypotheses — decisions rest on data, not taste;
• avoiding harm — a change that hurts metrics never reaches everyone;
• measuring effect — you see exactly how much the metric moved;
• learning — the team discovers what really influences users.

How do you prepare an A/B test?

Preparation matters more than launch. Before the test, write down the hypothesis, the primary metric, guardrail metrics and the required sample size — otherwise it is easy to find a "suitable" result among dozens of indicators afterwards.

Element	Example
Hypothesis	Showing delivery times on the product page will make more users add items to the cart
Primary metric	Share of sessions with an add-to-cart
Guardrail metrics	Average order value, return rate, page load speed
Minimum effect	+3% on the primary metric
Audience	All new catalogue visitors on the website
Duration	Two full weeks to cover different days

On choosing metrics, see product metrics after launch.

Photo: Unsplash

How many users and how much time do you need?

Sample size depends on the metric's current value and how small an effect you want to detect. The smaller the effect, the more users you need: detecting a rise in conversion from 3% to 3.3% takes tens of thousands of visitors per group.

Rules:

1. Calculate sample size with a calculator before launch, not after.
2. Run tests for whole weeks — behaviour differs between weekdays and weekends.
3. Do not stop a test early as soon as the result looks "significant".
4. If traffic is short, test bolder changes or use qualitative research.

Which mistakes produce false results?

The most common mistake is peeking: the team checks results daily and stops the test the moment it sees a significant difference. Thanks to random fluctuation, this can "find" an effect where none exists.

• stopping at the first significant result;
• choosing the metric after the test from many indicators;
• inconsistent assignment — a user sees one version, then the other;
• event-counting bugs in one of the groups;
• simultaneous overlapping tests on the same part of the interface;
• the novelty effect — returning users click on anything new out of curiosity.

Before analysing, check that the groups came out roughly equal: a noticeable size imbalance usually signals a technical bug.

How do you implement A/B tests technically?

Technically a test is a feature flag with users assigned to groups, plus analytics recording which group a user was in. Assignment must be stable: the same user always sees the same variant.

• assignment by a hash of the user ID — see feature flags;
• an "entered experiment" event with the test name and variant;
• target events recorded to a single schema — see web analytics with events;
• server-side assignment to avoid interface flicker and caching issues;
• once finished, the losing variant and the flag are removed from code.

In short

• An A/B test compares versions simultaneously and separates a change's effect from outside factors.
• Before launch, write down the hypothesis, primary and guardrail metrics and sample size.
• Run tests for full weeks and do not stop them early.
• The main mistakes are peeking, picking metrics afterwards and technical group imbalances.
• Technically a test is a flag with stable assignment plus event analytics.