A/B-Testing: Fundamentals, Statistics and Tools for eCommerce

A/B-testing is the only way to know with certainty what works. Not opinions. Not gut feeling. Not “that looks better.” Only data. We have conducted thousands of A/B-tests. The most important learning: 7 of 10 “improvements” we would have implemented without a test would have lowered conversion. Not raised it. Lowered it.

Why A/B-testing is the only way that counts

You can dream up a new landing page. A new order process. A new CTA. You can say: “That looks better.” But “looking better” does not convert. Data converts.

The reality is sobering. We all have biases. We love what we designed ourselves. We overestimate design and underestimate friction. We believe we know our customers. We do not.

That is why we test. Every change. Every assumption. Every idea. Not because we do not dare to decide. But because we do not dare to decide wrongly.

Our Testing Approach

We use our own testing stack. Developed over thousands of tests. Optimised for e-commerce. Not for landing pages. Not for lead generation. For shops.

Hypothesis instead of assumption. “I believe a red button colour converts better” is not a hypothesis. A hypothesis reads: “We observe that 68 % of mobile users drop off the order process. We suspect that a sticky CTA button raises mobile conversion by 15 %.”

Statistical significance. A test with 200 visitors and 5 conversions says nothing. Nothing. At low conversion rates we need thousands of visitors per variant. At least 100 conversions per variant. Or at least 2 weeks runtime. Whichever comes first.

Segmentation. A test can be neutral on average and bring 30 % gain in one segment. Mobile vs. desktop. New vs. returning. Organic vs. paid. The average deceives. The segments reveal.

What we test

What science says

A/B-testing is not a new idea. The fundamentals were developed in the 1920s by Ronald Fisher in agricultural statistics. Today it is the standard instrument of data-driven decision-making, not only in e-commerce, but in medicine, politics and technology.

Harvard Business Review has repeatedly proven that companies conducting systematic A/B-tests show higher growth rates than their competition. Not because they test more. But because they test better.

The mathematical reality: A test with 200 visitors and 5 conversions says statistically nothing. The rule of thumb is: at least 100 conversions per variant. Or at least 2 weeks runtime. Whichever comes first. Everything below that is chance.

The segmentation trap: A test can be neutral on average and bring 30 % gain in one segment. Mobile vs. desktop. New vs. returning. Organic vs. paid. The average deceives. The segments reveal. That is not a feeling. That is mathematics.

Common Testing Errors

Stopping too early. After 3 days and 50 conversions the test is judged “significant”. That is chance. We see tests that had 95 % confidence after one week and landed at 60 % after 4 weeks.

Too many variants. A/B/C/D/E tests need 5x as much traffic. For most shops, A/B is sufficient.

Not segmenting. The average lies. Every segmentation is a potential treasure.

Not rolling out tests. A test shows +15 %. The shop owner is happy. And never implements. The gain stays theoretical.

Why JDKRUEGER&CO for A/B-Testing

No tool vendor. No consultant. A testing system. We have no product we want to sell you. We have a system that optimises your shop. The difference: we win when you win. And we only win when the data proves it.

Thousands of tests. 20 years of experience. Since 2006 we have conducted systematic A/B-tests, first for our own shops, then for customers. We have learned that 7 of 10 “good ideas” without a test would lower conversion. We have learned that the average deceives and the segments reveal. We have learned that patience is the biggest lever.

Own testing stack. We use no SaaS tool that costs monthly and puts your data on foreign servers. Our stack runs in your infrastructure. You keep control. You keep the data. You keep the knowledge.

FAQ: A/B-Testing

How long does a test take?

At least 2 weeks. At least 100 conversions per variant. For shops with >10,000 visitors/month: typically 2–4 weeks. For less traffic: 4–8 weeks.

Can I run multiple tests at the same time?

Yes, but not on the same page. A test in the order process and a test on the product page are unproblematic. Two tests in the order process influence each other.

What does A/B-testing cost?

Included in the model. No separate costs. You pay 10 % of demonstrable uplift.

Does testing work with low traffic?

Yes. We then use longer runtimes or alternative statistical methods. Even small shops can test, the question is only: how quickly do we reach significant results?

Related Topics

• Conversion Rate Optimisation
• Web Analytics
• eCommerce UX

You are guessing what works?
We test it. Discuss test strategy