A/B testing

A/B Testing Basics: Raise RPU in Ecommerce Instead of Guessing. The Method

A/B testing basics cover the systematic, Bayesian comparison of two versions of a web page, an element or a user flow to find out objectively which one earns a higher revenue per user.

Jörg Dennis Krüger · 9 April 2026 · 6 min read

A/B testing basics cover the systematic, Bayesian comparison of two versions of a web page, an element or a user flow to find out objectively which one earns a higher revenue per user (RPU). For shop owners, this is the only way to make ecommerce decisions based on data, grow revenue systematically and stop relying on opinions.

The RPU paradox: why most A/B tests fail

Three out of four A/B tests run in online shops end without a clear winner or with a “false positive”. In other words, a supposed improvement has no financial effect in reality, or even a negative one. These failures cost companies billions.

The main cause is poor method: focusing on the wrong metric, missing hypotheses, uncontrolled tests and misreading the results. Even experienced CRO specialists are wrong on 40-60% of their hypotheses. Testing removes that risk.

RPU as your compass: revenue per user beats conversion rate

The primary metric for every A/B test is revenue per user (RPU). It measures the revenue each visitor generates directly. It is the only reliable sign of whether a test variant actually moves your business forward.

An example: a test lifts the conversion rate (CR) by 10% but cuts the average order value (AOV) by 15% at the same time. Judged on CR alone, that is a “winner”. In fact the test lost revenue. Only RPU shows the real financial impact.

We judge every test result on RPU first. If an apparent CR “loser” shows a rise in RPU, it is the real winner for your business. This metric protects long-term growth.

The thinkCONVERSION method: a system for real optimisation

Professional A/B testing is a structured process that replaces opinions with evidence. The thinkCONVERSION method takes you from the problem to a tested solution.

1. Analysis and hypothesis: understand the “why”

Every test starts with a problem you can prove in the data. Tools such as Google Analytics 4, Hotjar or CRM systems show exactly where users drop off or hesitate in the buying process.

We write the hypothesis precisely and in a way that can be proven wrong, following this pattern: “If we replace [element X] with [element Y], [RPU] will rise because [psychological or data-based reason].” Vague statements like “the page will get better” are useless.

2. Test setup: one variable, full control

For the A/B test we split traffic randomly and evenly (50/50) into a control group (variant A: the original) and a test group (variant B: our change).

Important: change only one variable per test. If you change the headline, the image and the button colour at once, you cannot attribute the result to any single element. That makes the result invalid.

Common testing mistake Correct approach What it means for RPU
Several elements changed at once Only one element changed per test Clear cause and effect: you know *what* moves RPU.
Test without a precise hypothesis (“I think…”) Test with a specific hypothesis that can be proven wrong Targeted learning: you understand *why* RPU went up.
Test stopped before enough data was collected (“peeking”) Test runs until the defined probability is reached (Bayesian) Valid results: you avoid the risk of “false positives” and lost RPU.

3. Bayesian evaluation: probabilities, not a gamble

We deliberately drop the outdated idea of “statistical significance” (p-value thinking from frequentist statistics). Instead we use Bayesian statistics, which at any point gives you a direct, intuitive probability that variant B performs better than A.

The result is clear: “Variant B has a 92% probability of generating a higher RPU.” We run the test until this probability to win reaches 90-95%. That way every decision rests on solid data.

This transparent approach removes the problem of “peeking”, stopping a test too early, which produces false “winners”. Peeking is a main reason for apparent optimisation wins that actually cost revenue, and the Bayesian approach rules it out.

Untapped potential: test imagery without a photo shoot

Many shop owners shy away from testing imagery because new product photos look like a huge effort. That is exactly where surprising potential lies: with AI-generated image variants you can test complete shop imagery efficiently. This is A/B testing for visual communication.

Think of different backgrounds, moods or lifestyle settings. An A/B test can prove which visual language raises RPU. Only when the test wins is a real photo shoot worth the money. Test first, invest second.

This method works especially well on product detail pages or landing pages, where the visuals feed straight into the buying decision. Here A/B testing separates the risk from the upfront cost and gets the most out of each change.

The A/B testing principle: data over certainties

We do not give absolute recommendations about what “always” works. Claims like “red CTAs convert better” are a risky generalisation. What matters is the contrast with the rest of the page and with the brand, not the colour itself. Whether red works better in your specific context is for a test to show.

Our position: tests regularly show certain tendencies. But the final proof for *your* shop only comes from *your* own data, through an A/B test. What brings +20% RPU in shop A can do nothing in shop B. That is why we test. Instead of guessing: an A/B test. Instead of believing: data.

This principle separates systematic conversion rate optimisation from vague marketing wisdom and makes sure every change has a real, measurable financial effect. It is the route to long-term revenue growth and a real rise in RPU that shows up directly in your figures.

Frequently asked questions about A/B testing basics

FAQ: Common questions about A/B testing

What exactly are A/B testing basics?

A/B testing basics cover the methodical comparison of two versions of a website (A and B) while collecting user data. The goal is to use Bayesian evaluation to find the variant with the highest revenue per user (RPU), so you can make decisions based on data.

Why is RPU the key metric in A/B testing?

RPU (revenue per user) is the most important metric because it measures revenue per visitor directly and takes both conversion rate and average order value into account. Focusing only on the conversion rate can produce misleading “winners” that end up lowering total revenue.

How long should an A/B test run?

An A/B test should run for at least two full business weeks to even out daily and weekly swings in user behaviour. What matters is collecting enough data for a valid Bayesian evaluation that shows a high probability to win.

What does Bayesian statistics mean in A/B testing?

Bayesian statistics in A/B testing gives you, at any point, the probability that a test variant (for example B) performs better than the original. This direct statement of probability (“Variant B has a 90% probability of winning”) is intuitive and avoids the misreadings of the traditional p-value, such as “peeking”.

Can you test complex changes such as a new navigation?

Yes. Complex changes such as a reworked navigation or an alternative product detail layout can be checked effectively with template tests. The changes are made to an underlying template instead of single pages, so the effect reaches every page of the same type. That gives you a big lever for little effort and fast insights.

Which common A/B testing mistakes should you avoid?

Avoid testing without a clear hypothesis, ignoring RPU as the primary metric, changing several elements at once (contamination) and stopping tests early (“peeking”). Copying “best practices” blindly without validating them yourself is another typical mistake.

Is A/B testing worth it for smaller online shops?

Yes, A/B testing is valuable for smaller online shops too, as long as there is enough traffic for statistically valid results. Focus on pages with high revenue potential (such as the checkout or product detail pages) and test bigger changes, so you see measurable effects faster and use your resources well.

Keep reading

Related articles

Let’s start with your shop.

In the Dropoff Analysis you see where your shop loses buyers. Afterwards you know where the first step pays off.