Grow Ecommerce Revenue With Better A/B Tests

A/B test optimisation: how we get the most out of every test

A/B testing is the most powerful tool in ecommerce. But most shops use only a fraction of its potential. They test a headline, wait a week, look at the conversion rate and decide.

That is A/B testing 1.0. We work with A/B testing 3.0: systematic, data-driven, with a framework that extracts the maximum value from every test, won or lost.

This article shows how we optimise tests before they start, while they run and after they end. No theory. Only methods that still have to prove themselves in a test.

Before the test: the quality of the hypothesis decides everything

The hypothesis formula

A bad hypothesis: “We’re testing a green button colour.” A good hypothesis: “We believe a green CTA button will raise the conversion rate for desktop users by 8%, because green is the only action colour on our site and the current blue button blends into the footer.”

The good hypothesis has five elements:

• Change: What exactly is being changed?
• Audience: Who is it tested on?
• Outcome: What do we expect?
• Size: By how much?
• Reason: Why should it work?

Without a reason it is not a hypothesis, it is a wish. Without a size you cannot judge the quality of the test. If the expected uplift is 2%, you need a huge sample. If it is 20%, a smaller one will do.

A fashion shop wrote 12 hypotheses per quarter. The average expected uplift was 15%. The actual uplift of the winning tests was 8%.

After the hypothesis framework was introduced, the expected uplift fell to 10%: more realistic, but still ambitious. The hit rate (the share of tests with a significant result) rose from 35% to 58%. More realistic expectations lead to better tests.

The prioritisation matrix

Not every hypothesis is equally important. We use a simple matrix: Impact × Confidence × Ease = Priority.

Impact: How much revenue is at stake? A checkout optimisation affects 100% of buyers. A home page optimisation affects 40%.

Confidence: How sure are we that the hypothesis is right? Based on data (recordings, heatmaps, surveys) = high. Based on gut feeling = low.

Ease: How easy is it to implement? A copy change = high. A checkout redesign = low.

An electronics shop had 20 hypotheses in its backlog. The top 3 according to the matrix: (1) shipping cost transparency in the basket (impact: high, confidence: high, ease: high), (2) mobile CTA size (impact: high, confidence: medium, ease: high), (3) fewer checkout fields (impact: high, confidence: high, ease: medium). These three tests delivered 70% of the quarter’s revenue growth.

During the test: monitoring matters half as much as the test itself

The 48-hour check

The first 48 hours of a test are critical. Not for the final result, but for technical validation.

After 24 hours we check:

• Is the test running correctly? (Is the traffic split 50/50?)
• Is there an SRM (sample ratio mismatch)?
• Is the tracking data plausible?
• Are any unexpected errors showing up?

After 48 hours:

• Is the traffic split stable?
• Are there significant differences in the baseline?
• Are user metrics (load time, error rate) identical?

A home & living shop started a test. After 24 hours the analysis showed that variant B was getting 52% of traffic instead of 50%. An SRM. The cause: a caching problem that favoured desktop users. The test was stopped, the problem fixed and the test restarted. Without the 48-hour check, the result would have been skewed.

Ongoing monitoring

After the initial check, we monitor tests 3 times a week. Not to make interim decisions, but as an early warning system.

We look at:

• Traffic drops (>20% = alarm)
• Conversion drops (>15% = alarm)
• Technical errors (404s, load time spikes)
• SRM coming back

A beauty shop ran a test for 2 weeks. After 5 days, monitoring showed that variant B’s conversion rate had fallen by 25% on mobile. Desktop was unchanged. The cause: a CSS bug that only appeared on iPhones.

The test was paused for mobile while desktop kept running. Once the CSS was fixed, mobile was restarted. Without monitoring, the test would have damaged mobile revenue for a week.

After the test: the evaluation that really counts

Primary metric: RPU, not conversion rate

Most shops judge tests by conversion rate. That is a mistake. A test can raise the conversion rate and still lower revenue.

Example: a furniture shop tested a 10% discount code on the home page. The conversion rate rose from 1.2% to 1.5%. But the average order value fell from €450 to €380 (because customers bought cheaper products to make the most of the discount). RPU fell by 8%. The test was a winner on conversion but a loser on revenue.

The rule: the primary metric is always RPU. Secondary metrics: conversion rate, AOV, return rate, customer lifetime value (where measurable).

Segment analysis

A test that shows “no difference” overall can be a winner in one segment. A test that wins overall can lose in one segment.

A fashion shop tested a new product page. Overall result: +5% RPU, statistically significant but only just. The segment analysis showed desktop +12%, mobile -8%. The win came entirely from desktop. The consequence: variant B was rolled out on desktop only. Mobile kept variant A until a separate mobile test was developed.

The standard segments we analyse:

• Device (desktop vs. mobile vs. tablet)
• Traffic source (organic, paid, direct, social)
• New vs. returning
• Geography (where relevant)
• Time of day (where relevant)

Documentation

A test without documentation is forgotten. A test with documentation is a building block for the future.

Our test documentation contains:

• Hypothesis (with reasoning)
• Test setup (traffic split, duration, audience)
• Result (RPU, conversion, AOV, segmented)
• Decision (roll out, iterate, reject)
• Insight (what did we learn?)
• Next steps (follow-up tests)

An electronics shop documented 50 tests over 2 years. The analysis showed that checkout tests had a success rate of 65%. Home page tests had a success rate of 30%.

Tests with qualitative preparation (heatmaps, recordings) had a success rate of 60%. Tests without preparation had a success rate of 25%. These findings changed the testing strategy: more checkout tests, more qualitative preparation, fewer home page experiments.

The A/B test optimisation workflow

In short, our workflow in 6 steps:

1. Collect data: analytics, heatmaps, recordings, surveys. Identify patterns.

2. Form a hypothesis: change + audience + outcome + size + reason.

3. Prioritise: Impact × Confidence × Ease. Test first what brings a lot of revenue, is likely to work and is easy to build.

4. Test and monitor: a 48-hour check, then monitoring 3 times a week. No interim decisions.

5. Evaluate: RPU as the primary metric. Segment analysis for every relevant dimension.

6. Document and iterate: every insight feeds into the next test. A lost test with good documentation is worth more than a won test without it.

Frequently asked questions about A/B test optimisation

How long should a test run?

At least one full business-cycle week (Monday to Sunday). With low traffic: 2–4 weeks. Never less than 7 days, even if the calculator shows significance after 3 days.

How many tests should I run in parallel?

As many as possible, but not on the same page. If test A changes the home page and test B the checkout, that is fine. If both change the checkout, the effects overlap.

What is the minimum sample size?

With a 2% conversion rate and a 10% expected lift: about 7,700 visitors per variant. Use a sample size calculator, not gut feeling.

Should I run tests at all if my traffic is low?

Only if the page you want to test gets at least 1,000 visitors a week. Below that, the results are not statistically meaningful. Focus instead on qualitative methods (recordings, surveys) and direct implementation.

How much budget should I plan for A/B testing?

A testing tool costs €500–5,000 a month. A testing manager costs €4,000–8,000 a month. For a shop with €500,000 in annual revenue, the budget typically pays for itself in 3–6 months. A/B testing has one of the highest ROIs in all of ecommerce.

You will find more on this in the Academy track A/B-Testing Mastery ↗.