A/B testing
Ecommerce Conversion Optimisation: RPU as Your Profit Driver. System, Not Gut Feeling
It means replacing “gut feeling” with measurable facts so you grow faster, step by step.
A/B testing
A/B Testing Tools: The Independent Comparison Guide for CRO Professionals
A/B testing tools are the software that turns guesses into proof. This independent guide is for CRO professionals who know that the right tool is the difference between data-driven decisions and expensive hunches.
A/B Testing Tools: The Independent Comparison Guide for CRO Professionals
A/B testing tools are the software that turns guesses into proof. This independent guide is for CRO professionals who know that the right tool is the difference between data-driven decisions and expensive hunches.
These tools split traffic, measure conversion differences and tell you which variant actually brings in more revenue, not which one you like better. Without the right tool you test too slowly, too imprecisely or not at all.
Why choosing the right A/B testing tool matters
Most shops don’t fail at testing itself. They fail because of the wrong tool.
A tool that doesn’t understand your tech stack holds up development. A tool without a proper statistics engine produces false winners. And a tool your team can’t use gathers dust, not data.
Tool limitations that slow testing down
Worked example: a fashion shop tests for 14 months with an outdated tool and ends up with 23 tests and 0 significant winners. The ideas are not the problem. The tool cannot run server-side tests, cannot test the checkout and cannot segment new customers from returning ones, so the tests only run on the homepage, where the impact is lowest.
Another example: a B2B supplier with an average deal value of €80,000. The tool couldn’t handle long test durations and ended every test automatically after 3 weeks, at a conversion rate of 0.8%. Statistically meaningless. The tool produced data that meant nothing.
Integration with your existing tech stack
An A/B testing tool is not a standalone product. It has to talk to your CMS, your shop system, your analytics stack and your CRM. If the integration is 12 lines of custom code and a data layer held together by workarounds, something breaks with every shop update.
The best setups we see connect the tool and the shop system through a clean API. Data flows automatically in both directions. The reporting doesn’t just say “Variant B wins”, it says “Variant B wins with new customers on mobile coming from organic search”.
The key criteria for A/B testing software
Before you buy a tool, check these six criteria. Everything else is marketing.
Visual editor vs. code implementation
Visual editors are fast. You drag elements, change colours and write new copy without a developer. That works for headlines, button colours and hero images. For anything more complex, visual editors become a trap.
Server-side tests, checkout changes, price tests and algorithm tests need code. A tool that only handles visual tests covers maybe 30% of your potential. The rest goes untested.
Our recommendation: the tool must do both. The visual editor for quick iterations. The code editor for everything that affects revenue.
Statistics engine: Bayesian vs. frequentist
This is where opinions split, and so does the quality of results.
Frequentist statistics is the standard. It says: “At 95% confidence the difference is significant.” That is conservative and needs more traffic, but the results hold up. Bayesian statistics says: “The probability that variant B is better is 87%.” That is more intuitive and needs less traffic, but it is easier to misread.
In practice, Bayesian works better for shops with low traffic because tests reach a result faster. Frequentist is safer with high traffic and big financial decisions. The best tools offer both and let you choose.
More important than the method: how does the tool calculate test duration? A tool that doesn’t suggest a minimum sample size isn’t serious. A tool that ends tests after a fixed 2 weeks is dangerous.
Segmentation and personalisation
A test that shows an average 5% gain across all visitors can mean an 18% gain with new customers and a 3% loss with returning customers. Averages are the best way to hide real insights.
Good tools let you segment by traffic source, device, customer status, region and behaviour. The best tools allow personalisation: different variants for different segments, automatically and at the same time.
Reporting and export options
You don’t want to click around in the tool’s interface. You want the data in your BI tool, your data warehouse, your spreadsheet. Good tools offer APIs, webhooks, automatic exports and integration with Google Analytics 4, Adobe Analytics, Tableau and Looker.
Check before you buy: can I export the raw data? Can I define my own metrics? Can I track revenue per visitor rather than just conversion rate? If not, keep looking.
Tool comparison: the top platforms in 2026
The market has consolidated. There are no longer 50 equally good tools. There are three categories: enterprise players, all-in-one solutions and specialised lightweight tools. Here is the reality, not the marketing copy.
Optimizely: the enterprise standard
Optimizely is the tool every enterprise client evaluates first. It is also the most expensive. The platform covers A/B testing, feature flags, content management and personalisation. For large companies with multi-brand setups it is often the right choice. For a shop with €500,000 in revenue it is overkill.
VWO: the all-in-one CRO tool
VWO combines A/B testing with heatmaps, session recordings, form analytics and on-page surveys. That’s handy because you don’t have to buy 4 separate tools. The individual modules are good but not best in class. If testing is your main goal, pure testing tools give you more depth.
AB Tasty: a European provider with a clear focus
AB Tasty comes from France, is GDPR-compliant by design and has a strong e-commerce focus. The platform is easier to use than Optimizely and cheaper than VWO on the enterprise plans. For European shops that take data protection seriously, AB Tasty is a serious alternative.
Options after Optimize and Google-adjacent tools
Google’s Optimize was shut down in 2023 and has no like-for-like successor. The gap has been filled by tools such as VWO, AB Tasty, Kameleoon and Varify.
Splitbee / PostHog: modern lightweight alternatives
Splitbee and PostHog are the new players. They are fast, modern and developer-friendly. PostHog offers A/B testing, feature flags, product analytics and session recordings in one. Splitbee is lighter still and aimed at small to mid-sized teams. Both are cheaper than the enterprise tools, but less powerful.
Unbounce / Instapage: landing page focus
These are not full A/B testing tools in the classic sense. They are landing page builders with built-in testing. If you mainly test landing pages, that’s enough. For checkout tests, product page tests and server-side tests they are the wrong choice.
Optimizely in detail: features, prices, use cases
Optimizely is the SAP of the testing world. Powerful, expensive, complex, and essential for some companies.
Its strength is scale. Optimizely can test in 12 markets, 8 languages and 3 brands at once, with central control and local execution. The feature flag integration lets you connect tests with product rollouts. That is valuable for tech companies and irrelevant for a shop running a single Magento instance.
Its weakness is price. Licences start at €50,000 a year and are often much higher. Add implementation, training and ongoing support. A shop with €2 million in annual revenue spends 2-3% of its revenue on a tool before the first test runs.
Use cases that make sense: multi-brand setups, international rollouts, companies with their own dev teams that want to combine feature flags and testing. Use cases that don’t: single shops, small teams without a dedicated CRO resource, companies that mainly want to test quickly.
VWO in detail: features, prices, use cases
VWO is the Swiss Army knife of CRO. It does a bit of everything and nothing perfectly, which is exactly right for many teams.
The testing engine is solid. Visual editor, code editor, server-side tests, multivariate tests: it’s all there. The statistics engine is frequentist, with decent sample size calculators. Segmentation is good, not outstanding.
What really sets VWO apart is its built-in CRO ecosystem. Heatmaps, scroll maps, session recordings, form analytics and on-page surveys all live in one interface. That saves money and time. A test setup that takes 2 days across 3 tools is done in 4 hours in VWO.
Prices start at around €200 a month for small teams and rise with traffic and features. For a shop with €100,000 to €1 million in revenue, VWO is often the sweet spot between functionality and price.
Use cases that make sense: teams that want to combine testing with qualitative analysis. Companies looking for one tool for everything. Shops with medium traffic that don’t need an enterprise licence.
Open-source alternatives
Not every company has €20,000 for a testing tool. And not every company needs one.
GrowthBook: the open-source favourite
GrowthBook is open source, self-hostable and surprisingly powerful. It offers A/B testing, feature flags, statistical analysis and SDKs for all common frameworks. The community is active and the documentation is good.
The catch: you need a developer. GrowthBook is not a drag-and-drop tool. It is a framework you build into your code. For teams with developers that’s an advantage: full control, no vendor lock-in. For teams without developers it’s a barrier.
Other options: Firebase A/B Testing (for mobile apps), Cloudflare Workers for simple server-side tests, or your own setup with Google Analytics 4 and BigQuery. These are more limited, but often good enough for your first tests.
Choosing a tool by company size
There is no “best” A/B testing tool. There is the best tool for your situation.
Startups and small shops (<100k revenue)
Priorities: very low cost, quick to implement, visual editor.
Recommendation: GrowthBook (if you have a developer) or an entry plan from VWO or AB Tasty. Focus on quick, simple tests. No enterprise tool. The time you’d spend evaluating tools is better spent on your first tests.
Growing e-commerce businesses (100k-1M)
Priorities: a balance of functionality and price, good integration with your shop system, solid reporting.
Recommendation: VWO or AB Tasty. Both offer enough depth for serious testing without enterprise prices. Integration with Shopify, WooCommerce and Magento is documented and proven. Invest in training: a €300/month tool with a trained team beats a €5,000/month tool with an untrained one.
Enterprise and multi-brand setups (>1M)
Priorities: scalability, security, multi-user management, feature flags, enterprise support.
Recommendation: Optimizely or AB Tasty Enterprise. The choice depends on whether you need feature flags (Optimizely) or put GDPR compliance and European support first (AB Tasty). Budget is less of a problem here than implementation speed and internal adoption.
Integration and implementation: best practices
A tool is only as good as its implementation. Here are the most common mistakes and how to avoid them.
Putting the tool before the architecture.
We see shops rebuild their whole frontend just so a particular tool works. That’s backwards. The tool has to fit your stack, not the other way round.
No measurement plan.
Before the first test: what are you measuring? Which metrics are primary, which secondary? How do you define “success”? A tool that tracks everything only produces noise. A clear measurement setup produces insights.
Forgetting about flicker.
Client-side tools change the DOM after the page loads. That causes “flicker”: the user briefly sees the original, then the test variant. It skews results and annoys users. The fix: server-side implementation or asynchronous loading strategies.
Underestimating QA.
Every test has to be checked on all devices, browsers and breakpoints. A test that works on desktop but covers the checkout button on mobile can cost thousands of euros. At scale, automated QA is a must.
No sunset plan.
What happens to a winning test? Does it get implemented? Who is responsible? When? A test that sits in the tool as “running” forever eats resources and skews data. Implement the winners, delete the losers, keep the tool tidy.
Common mistakes when choosing a tool
The biggest mistakes happen before you buy.
Buying on features instead of outcomes.
“The tool has 47 features!” Yes, and you need 5. List the 5 tests you want to run over the next 6 months. Which tool makes those 5 tests easiest? That’s your tool.
Getting the price wrong.
The list price is not the total price. Add implementation, training, ongoing support, integration with your existing tools and any frontend changes. A €500/month tool can cost €15,000 in the first year.
Buying without a trial.
Every serious vendor offers a proof of concept or a trial period. Use it. Test your most important use case. Measure the implementation time. Check the reporting. If even the sales process runs smoothly, that’s a good sign.
Ignoring the future.
Where will your shop be in 2 years? If you move from Shopify to your own platform, will the tool still work? If your traffic grows by 300%, does the licence scale? Vendor lock-in is expensive.
Forgetting the team.
A tool only the CRO manager can use is a bottleneck. A tool that the content lead, the designer and the frontend developer can all use grows with your team. Check usability for every role, not just for the person championing the tool.
FAQ: A/B testing tools and choosing software
What is the best A/B testing tool for e-commerce?
For most e-commerce shops with €100,000 to €5 million in revenue, VWO or AB Tasty is the best choice. Both offer visual editors, server-side tests, good Shopify and Magento integration and fair prices. Enterprise setups with a multi-brand structure get more from Optimizely. Startups without a budget should look at GrowthBook, but that needs developer know-how.
What do A/B testing tools cost on average?
Open-source tools like GrowthBook have no licence fees but take implementation effort. Entry-level plans start at €100-300 per month (VWO, AB Tasty).
Mid-market packages run at €500-2,000 per month. Enterprise licences such as Optimizely start at €50,000 a year and rise with traffic and features. Including implementation and training, the total cost in year one is often 2-3× the list price.
What is the difference between frequentist and Bayesian statistics in A/B testing tools?
Frequentist statistics says: “At 95% confidence the difference is significant.” That’s conservative and needs more traffic, but the results hold up. Bayesian statistics says: “The probability that B is better is 87%.” That’s more intuitive and faster, but easier to misread.
For shops with low traffic, Bayesian is more practical. For high stakes and big financial decisions, frequentist is safer. The best tools offer both.
Do I need a separate A/B testing tool, or is Google Analytics enough?
Google Analytics alone is not enough. GA4 shows you what happens, but it can’t split traffic, create variants or calculate statistical significance. You can build and compare audiences in GA4, but that isn’t a controlled experiment.
For real A/B testing you need a dedicated tool that randomises traffic, serves variants and validates results statistically. GA4 is the reporting layer, not the testing layer.
What is the difference between A/B testing and feature flags?
A/B testing measures which variant drives more conversions. Feature flags switch features on or off for certain users, without measuring anything. The difference is the focus: testing is experimental and data-driven, feature flags are operational and reduce risk.
Modern tools like Optimizely and PostHog combine the two: you roll out features with flags and measure the impact with an A/B test at the same time. That’s especially valuable for software companies and often overkill for pure e-commerce shops.
How long does it take to implement an A/B testing tool?
A simple visual test on a landing page takes 30 minutes to set up. The initial implementation in your shop takes 2-5 days, depending on the shop system and how complex the integration is.
Server-side tests, checkout integration and custom event tracking take 1-3 weeks. Enterprise setups with multi-domain configuration, SSO and custom data layers can take 4-8 weeks. The longest part isn’t the technology. It’s testing the integration and running QA on every device.
Can I use several A/B testing tools at once?
Technically yes, in practice no. Two tools that split traffic and change the DOM at the same time create conflicts, flicker and skewed data. You no longer know which tool caused which change. The exception: one tool for client-side tests and another for server-side tests, covering strictly separate areas.
Even then, reporting is a lot of work. Our recommendation: one tool that does both. By the way, switching tools is easier than running two in parallel: migrate your tests, switch off the old tool, switch on the new one.
Which tool works best for Shopify shops?
For Shopify shops, VWO, AB Tasty and PostHog are the best options. All three offer native Shopify integration, visual editors for theme changes and server-side tests for checkout changes. VWO has the advantage of its built-in CRO stack (heatmaps, surveys).
AB Tasty scores on GDPR compliance and European support. PostHog is the cheapest option for technically skilled teams. Avoid tools without a documented Shopify integration: the implementation turns into a custom project that breaks with every Shopify update.
Key points at a glance
| Criterion | What matters |
|---|---|
| Statistics | Sample size calculator, frequentist + Bayesian |
| Implementation | Visual editor + code editor, supports your shop system |
| Segmentation | At minimum: traffic source, device, customer status |
| Integration | API, webhooks, GA4 export, BI tool connection |
| Value for money | Total cost in year one, not the list price |
| Team fit | Can your whole team use it, in every role? |
Choosing an A/B testing tool is not a technical decision. It’s a strategic one. The wrong tool blocks your testing programme. The right one speeds it up. Spend time on the evaluation, but not too much.
In the end, what counts is not the tool but what you test with it. The fundamentals of A/B testing matter more than the platform. You can go deeper on them in the Academy track A/B-Testing Mastery ↗.
Start with a clear use case. Trial 2-3 tools. Measure implementation time and the quality of results. Then decide, not before.
Keep reading
Related articles
A/B testing
Ecommerce Personalisation: What Works, and What Is Expensive Snake Oil
Ecommerce personalisation means tailoring content, product suggestions and offers to an individual visitor’s behaviour. Not to assumptions, not to demographic pigeonholes.
A/B testing
Grow your ecommerce results with heatmap analysis and conversion optimisation
Google Analytics shows you what happens. Heatmaps show you how it happens.
Let’s start with your shop.
In the Dropoff Analysis you see where your shop loses buyers. Afterwards you know where the first step pays off.