A/B Split Testing: The Scientific Method for Data-Driven Decisions

A split test, or A/B test, is a controlled experiment. Two versions of a web page, an email or an ad run at the same time, so you can measure which one performs better.

Instead of relying on gut feeling, split tests give you solid data for marketing and eCommerce decisions.

What is an A/B split test?

An A/B split test compares two versions of a digital asset. Version A is the status quo: the page as it is today. Version B contains one deliberate change. Both versions are shown to randomly selected visitors at the same time. The goal is to find out which version moves a defined target metric further.

The method comes from clinical research, where drugs are tested against placebos. In marketing you test headline against headline, button against button, price against price. The core stays the same: isolated variables, controlled conditions, statistically significant results.

You can split test almost any digital element:

• Landing pages and homepage designs
• Email subject lines and newsletter layouts
• Call-to-action buttons and forms
• Product pages and checkout flows
• Paid ad creatives and ad copy

How does an A/B split test work?

Every split test follows a fixed protocol. Stick to it with discipline and you cut out sources of error and get results that actually mean something.

Step 1: Write a hypothesis

Every test starts with a clear assumption. For example: “If we change the CTA button from green to orange, the click-through rate will rise by at least 15 percent.” A good hypothesis names the variable you are testing, the effect you expect and the user behaviour behind it.

This is exactly the craft taught in the Academy track A/B Testing Mastery ↗.

Step 2: Define the target metric

One metric leads the test. Common primary metrics are conversion rate, revenue per visitor (RPU), click-through rate or average order value (AOV). Secondary metrics, such as bounce rate or time on page, add context.

Step 3: Build the variants

Version A stays as it is. Version B introduces exactly one change. If you change several elements at once, you can no longer tell which change caused the result.

Step 4: Split the traffic and run the test

Visitors are assigned to the variants at random. A 50/50 split is standard. The test runs until it reaches statistical significance, usually at a 95 percent confidence level.

Step 5: Analyse and implement

Once the test ends, the winning variant becomes the new default. Documenting the results lays the groundwork for future tests and sharpens your team’s understanding of your own audience.

Split test vs. A/B test: is there a difference?

In practice the two terms are often used interchangeably. Technically, there is a distinction:

For most marketing teams the distinction is academic. What matters is not the terminology but the discipline: one variable, controlled conditions, a large enough sample.

When should you run a split test?

Split tests make sense whenever you want quantitative data to settle a decision. Typical occasions:

• New landing page: Which layout generates more leads?
• Price change: How does the conversion rate react to a higher or lower price?
• Checkout optimisation: Which flow reduces drop-offs?
• Email campaign: Which subject line gets opened more often?
• Redesign: Does the new design beat the old one?

Split tests are less suited to decisions that need qualitative insight, such as brand perception or the emotional pull of a campaign. There, user interviews and market research complement the quantitative method.

The 5 key elements of a successful split test

1. One test variable

Change the headline, button colour and image at once and you will never know which element made the difference. Isolation is the ground rule of every valid experiment.

2. A large enough sample

A test with 50 visitors per variant gives you random noise, not insight. Sample size calculators tell you upfront how many visitors you need, based on baseline conversion, expected effect and the statistical power you want.

3. Statistical significance

A 95 percent confidence level means the probability that the result is down to chance is at most 5 percent. Stop early and you risk wrong decisions. Wait too long and you waste traffic on a losing variant.

4. Representative traffic

Tests should run under normal conditions: not in the run-up to Christmas, not during a server outage, not right after a PR crisis. Outliers distort the data.

5. Clear documentation

Every test is an investment in what your organisation learns. Document the hypothesis, setup, result and learnings, and you build an internal knowledge base that makes future tests faster and sharper.

Common split testing mistakes and how to avoid them

Mistake 1: Stopping too early

The first 100 visitors often show what looks like a trend. Stop there and you are following chance. You only have a valid result once you reach the calculated minimum sample and significance.

Mistake 2: Testing several variables at once

That is not an A/B test but a multivariate test, an advanced method that needs far more traffic and more complex analysis. Beginners should stick to single-variable tests.

Mistake 3: Selling non-significant results as wins

“Version B got 12 percent more conversions.” At 72 percent significance that is not a finding, it is a guess. Only significant results justify implementation.

Mistake 4: Starting tests without a clear hypothesis

“Let’s see what happens” is not a strategy. Without a hypothesis you have no frame of reference for reading the results.

Mistake 5: Not separating mobile and desktop

People behave differently on smartphones than on desktops. A test that lumps both device types together hides important differences. Separate test strategies for mobile and desktop give you more precise recommendations.

Tools for A/B split tests

The right tool depends on your budget, technical setup and test complexity. Here is an overview of common options:

One thing matters: a tool alone does not create a testing culture. Without clear processes, trained people and a commitment to data-driven decisions, the software sits unused or gets used wrongly.

FAQ: Common questions about split tests

What is the difference between a split test and an A/B test?

Historically, a split test compared completely different pages on separate URLs. An A/B test compares two versions of the same page with one isolated change. Today both terms are mostly used interchangeably.

How long should a split test run?

At least until it reaches statistical significance and covers at least one full business cycle, typically 1 to 4 weeks. Tests that are too short give random results, tests that run too long waste traffic.

What sample size does a split test need?

That depends on your current conversion rate, the improvement you expect and the confidence level you want. With a baseline conversion of 2 percent and an expected uplift of 20 percent, you need roughly 8,000 visitors per variant for a 95 percent confidence level.

Can you test several elements at once?

Yes, that is called a multivariate test. It needs far more traffic, though, and only allows limited conclusions about the effect of individual elements. For most teams, sequential A/B testing is the better choice.

What is a multivariate split test?

A multivariate test combines several variables at once and analyses how they interact. Instead of two versions, several combinations are tested. The method suits high-traffic pages with enough visitors.

Are split tests GDPR-compliant?

Yes, if they are implemented correctly. The GDPR requires a lawful basis for processing, transparent information for users and compliance with retention periods. Tools like Convert are built explicitly for GDPR compliance. Tracking cookies must be part of your consent management process.

What mistakes do beginners often make with split tests?

The most common ones: stopping too early, changing several variables at once, overrating non-significant results, not writing a clear hypothesis and not looking at mobile and desktop separately.

Which tool is best for split tests?

There is no universally best tool. Kameleoon suits beginners, Optimizely enterprise needs, VWO teams that want a visual editor without developers. The choice depends on budget, traffic volume and internal resources.

For more on raising your conversion rate systematically, read our A/B testing guide for beginners.

On the podcast

Der Conversion Hacker Podcast, episode #004, Test Thursday: the hypothesis before the test, or how to set up A/B tests that actually tell you something (in German)