A/B test duration calculator

Calculating A/B test duration: how long does a valid split test take?

The run time of an A/B test decides whether your results are statistically valid or pure noise. Tests that are too short lead to wrong decisions. Tests that run too long cost time and revenue. Our calculator shows you the optimal test duration based on your traffic, your conversion rate and the improvement you expect.

How does the calculation work?

The calculation is based on the statistical sample-size formula for A/B tests with two proportions. We use a standard of 80 percent power and 95 percent significance, the industry standard for eCommerce CRO.

• Statistical power (80%): The probability of detecting a real effect. 80 percent means: in 4 out of 5 cases we find a genuine improvement.
• Significance level (95%): The probability of not making a type 1 error. 95 percent means: in only 1 out of 20 cases do we see an effect that does not exist.
• Minimal Detectable Effect (MDE): The smallest improvement you want to prove. The smaller the MDE, the longer the test.

The formula accounts for the number of variants and scales the required sample size accordingly. Three variants need more traffic than two. Five variants need more still.

Rules of thumb for A/B tests in eCommerce

• At least 1 week of run time to cover weekday effects. Nobody buys on Monday morning. Saturday evening they do.
• At least 100 conversions per variant. Fewer is not statistically meaningful.
• For seasonal products, at least 1 complete sales cycle. A Christmas test in July is worthless.
• Never stop tests early. The peeking problem distorts results badly.
• Split traffic evenly. 50/50 is standard. 90/10 lengthens the test unnecessarily.

The most common mistakes with test duration

Many shop owners stop tests after 3 days because one variant "clearly looks better". That is the most expensive mistake in CRO. A test with 500 visitors a day and a 2 percent conversion rate needs at least 2 weeks to statistically validate a 15 percent lift. At 10,000 visitors a day, 3 days is enough. The calculator shows you the exact number for your shop.

When is an A/B test statistically significant?

A test is significant when the p-value is below 0.05 AND the achieved power is above 80 percent. Both must hold. A p-value of 0.03 with only 40 percent power is worthless. The calculator computes both values and shows you the exact point at which you can stop the test.