Page speed in A/B testing: how to test without slowing your shop down

“Won’t a testing tool like that slow the shop down?” The question came up again recently in a conversation with a Shopify merchant. The short answer: with a modern setup, page speed is no reason to avoid A/B tests.

In a modern setup the test script loads asynchronously alongside the page, weighs around 40 KB and never holds the page up. Only what a variant changes in the visible area gets hidden, and then for 300 milliseconds at most. And every variant has to prove before launch that it does not load noticeably slower than the control.

What a modern setup does differently

Asynchronous loading

The script does not sit in the path the browser needs to first display the page. The page renders while the testing tool loads in parallel.

Small and cached

The script comes from a CDN and stays in the browser cache. From the second page view on, the visitor does not download it again.

Hide only what changes

Content is hidden only where a variant actually changes something in the visible area, and then for a fraction of a second. Everything else shows immediately.

Testing tools got their bad reputation in the days when the whole page was hidden until the test had loaded. Google’s Optimize, shut down in 2023, waited up to four seconds for this. The next section shows how it works today.

What browsers can do today

In its guidance on flicker-free testing in the browser, the Chrome team describes an approach without the old anti-flicker snippets: load the test script in the page head with the attribute blocking=“render”, switch the variant via CSS and cap execution time with a budget, for example under 100 milliseconds.

Server-side, when it has to be

Where every millisecond counts, the server or CDN delivers the variant directly. Then there is no test script in the browser at all. Shopify takes this route with Rollouts, with its own limits, which we described in our article on Shopify Rollouts.

How we do it

The test script comes from a CDN, loads asynchronously and weighs around 40 KB compressed. The program code stays in the browser cache for a week; only the test configuration is refreshed every few minutes. It does not block page rendering.

Every variant is measured against the control before launch: when does the largest element become visible, and does anything shift in the layout? If the variant is noticeably worse, it does not launch. We check page speed before the test, so nobody has to argue about it afterwards.

How to spot a modern setup

1. The test script is included with async, or comes through the tag manager, which loads asynchronously anyway.
2. The source code has no snippet that sets the whole body to opacity: 0. If it does, check the timeout. A few hundred milliseconds is fine; several seconds is a leftover.
3. The configuration contains only running tests. Finished tests and drafts have no place in the package every visitor downloads.
4. If you want to know for sure, load the same page 20 times with and 20 times without the test script. Chrome DevTools lets you block the tool’s domain for this. The yardstick is Largest Contentful Paint, the moment the largest element becomes visible. Google sets 2.5 seconds as the threshold for a good score.

What counts in the end

Page speed is one metric, revenue per visit is the other. With a clean setup you do not have to choose between them. The tests run, and the shop stays fast.

Sources

• Chrome team, Modern Web Guidance: Flicker-free client-side A/B testing
• Andy Davies: The Case Against Anti-Flicker Snippets, 16 Nov 2020, on Google’s Optimize four-second default
• web.dev: Web Vitals, threshold for a good Largest Contentful Paint