A/B-Testing: Fundamentals, Statistics and Tools for eCommerce

A/B-testing is the only way to know with certainty what works. Not opinions. Not gut feeling. Not “that looks better.” Only data. We have conducted thousands of A/B-tests. The most important learning: 7 of 10 “improvements” we would have implemented without a test would have lowered conversion. Not raised it. Lowered it.

Why A/B-testing is the only way that counts

You can dream up a new landing page. A new order process. A new CTA. You can say: “That looks better.” But “looking better” does not convert. Data converts.

The reality is sobering. We all have biases. We love what we designed ourselves. We overestimate design and underestimate friction. We believe we know our customers. We do not.

That is why we test. Every change. Every assumption. Every idea. Not because we do not dare to decide. But because we do not dare to decide wrongly.

Our Testing Approach

We use our own testing stack. Developed over thousands of tests. Optimised for e-commerce. Not for landing pages. Not for lead generation. For shops.

Hypothesis instead of assumption. “I believe a red button colour converts better” is not a hypothesis. A hypothesis reads: “We observe that 68 % of mobile users drop off the order process. We suspect that a sticky CTA button raises mobile conversion by 15 %.”

Statistical significance. A test with 200 visitors and 5 conversions says nothing. Nothing. At low conversion rates we need thousands of visitors per variant. At least 100 conversions per variant. Or at least 2 weeks runtime. Whichever comes first.

Segmentation. A test can be neutral on average and bring 30 % gain in one segment. Mobile vs. desktop. New vs. returning. Organic vs. paid. The average deceives. The segments reveal.

What we test

Area	Typical Questions
Order process	One-page vs. multi-page? Guest order process? Payment options?
Product pages	Images, price presentation, CTA text, social-proof position
Cart	Upsells, shipping-cost transparency, trust signals
Navigation	Menu structure, search, filters, mobile menu
Hero/Landing	Headline, image, CTA, social proof

What science says

A/B-testing is not a new idea. The fundamentals were developed in the 1920s by Ronald Fisher in agricultural statistics. Today it is the standard instrument of data-driven decision-making, not only in e-commerce, but in medicine, politics and technology.

Harvard Business Review has repeatedly proven that companies conducting systematic A/B-tests show higher growth rates than their competition. Not because they test more. But because they test better.

The mathematical reality: A test with 200 visitors and 5 conversions says statistically nothing. The rule of thumb is: at least 100 conversions per variant. Or at least 2 weeks runtime. Whichever comes first. Everything below that is chance.

The segmentation trap: A test can be neutral on average and bring 30 % gain in one segment. Mobile vs. desktop. New vs. returning. Organic vs. paid. The average deceives. The segments reveal. That is not a feeling. That is mathematics.

Introducing A/B Testing: The Step-by-Step Plan

Many shops fail at testing before the first test even runs — because they start with the tool instead of the question. Here is how to introduce A/B testing properly:

1. Measure the baseline

Before you change anything, know where you stand: conversion rate, revenue per visitor, checkout completion — by device and by channel. Without a baseline you cannot prove success later.

2. Formulate a hypothesis

A good hypothesis links observation, assumption and metric: “We see that 68 % of mobile visitors abandon checkout. We suspect a sticky CTA raises mobile completion by 15 %.”

3. Set duration and sample size

At least 100 conversions per variant or at least two full weeks — whichever comes later. Our A/B test duration calculator works out the required runtime for your traffic.

4. Evaluate, segment, roll out

Averages deceive. Only segmentation (mobile/desktop, new/returning) shows where the winner really wins. And: a result you do not roll out is not a result.

A/B Testing Statistics: What Significance Really Means

Statistical significance is not a switch that flips at 95 %. It is a statement about probability: how likely is it that the observed difference arose by chance? Tests with few conversions reach 95 % quickly — and lose it again when the runtime is wrong. That is why our rule is: runtime before confidence, full weeks before weekend snapshots, conversions before clicks. Our duration calculator turns this into concrete numbers for your shop.

A/B Testing Tools: Our Approach

Most testing tools are built for landing pages, not shops. We therefore work with our own testing stack that runs in your infrastructure: your data stays with you, tests measure real orders instead of clicks, and evaluation happens on revenue, AOV and cart value. The JDK Model shows how the system is built. If you do not want to set up and evaluate tests yourself, we do it for you: A/B testing agency for online shops — 0 € setup, 10 % of demonstrable uplift.

Common Testing Mistakes

Stopping too early. After 3 days and 50 conversions the test is judged “significant”. That is chance. We see tests that had 95 % confidence after one week and landed at 60 % after 4 weeks.

Too many variants. A/B/C/D/E tests need 5x as much traffic. For most shops, A/B is sufficient.

Not segmenting. The average lies. Every segmentation is a potential treasure.

Not rolling out tests. A test shows +15 %. The shop owner is happy. And never implements. The gain stays theoretical.

Why JDKRUEGER&CO for A/B-Testing

No tool vendor. No consultant. A testing system. We have no product we want to sell you. We have a system that optimises your shop. The difference: we win when you win. And we only win when the data proves it.

Thousands of tests. 20 years of experience. Since 2006 we have conducted systematic A/B-tests, first for our own shops, then for customers. We have learned that 7 of 10 “good ideas” without a test would lower conversion. We have learned that the average deceives and the segments reveal. We have learned that patience is the biggest lever.

Own testing stack. We use no SaaS tool that costs monthly and puts your data on foreign servers. Our stack runs in your infrastructure. You keep control. You keep the data. You keep the knowledge.

FAQ: A/B-Testing

How long does a test take?

At least 2 weeks. At least 100 conversions per variant. For shops with >10,000 visitors/month: typically 2–4 weeks. For less traffic: 4–8 weeks.

Can I run multiple tests at the same time?

Yes, but not on the same page. A test in the order process and a test on the product page are unproblematic. Two tests in the order process influence each other.

What does A/B-testing cost?

Included in the model. No separate costs. You pay 10 % of demonstrable uplift.

Does testing work with low traffic?

Yes. We then use longer runtimes or alternative statistical methods. Even small shops can test, the question is only: how quickly do we reach significant results?

Related Topics

• Conversion Rate Optimisation
• Web Analytics
• eCommerce UX
• A/B Testing Agency
• A/B Test Duration Calculator
• CRO ROI Calculator

You are guessing what works?
We test it. Discuss test strategy