Shopify Rollouts as an A/B Test: Where the Release Tool Hits Its Limits

Last week a Shopify merchant asked me: “Can’t we just run the A/B tests directly in Shopify? Markets, Rollouts, create new. Then we don’t need an extra script and the shop stays fast.”

It is a fair question. The feature is new and costs nothing extra. I looked at what Shopify built Rollouts for, what the report shows and what happens when you try to test with it seriously. The short version: Rollouts is a tool for scheduled theme releases. A/B testing is an add-on there, and you notice that in several places.

What Rollouts is built for

Shopify describes a rollout as a scheduled set of changes to the main theme. The use cases in the Help Center come in this order: preparing seasonal updates, sales and campaigns and launching them at a fixed time.

Comparing changes. Preparing edits to the main theme without creating a duplicate. Fittingly, the feature sits under Markets: it puts theme changes live in a controlled way, for specific markets only if you want.

There are two types. A launch puts a change live at a set time. An experiment shows the change to only part of your visitors, while the rest see the control. Experiments need at least the Grow plan.

You create the variant by editing an automatically created copy of the theme in the theme editor, or by replacing the theme entirely with another one. Code changes only work the second way, more on that below.

As a release tool, it is well made. For a testing programme with several hypotheses a month, a lot is missing.

Parallel tests share the same visitors

In a testing programme several tests run at once, for example one on the homepage and one on the product page. Each gets the full traffic of its page.

Rollouts works differently. Each visitor ends up in exactly one rollout per theme, and Shopify splits the traffic between overlapping rollouts proportionally. The Help Center example: a rollout with 60 percent reach and one with 100 percent leave the first with an effective 37.5 percent, calculated as 60 divided by 160.

1 test

100 %

2 tests

50 %

3 tests

33 %

Rollouts splits traffic proportionally between overlapping experiments. The more tests run in parallel, the fewer visitors each test gets and the longer it takes to reach a reliable result.

For your tests, that means two experiments at 100 percent each get half the visitors, three get a third each. A test that would have needed two weeks on its own needs roughly six with three equally sized parallel tests.

And because every experiment is tied to the whole theme, a test on the product page competes for the same visitors as one on the homepage.

The same help page also hides a trap: if a rollout that replaces the whole theme starts while a rollout with theme changes is running, the effective traffic of the second drops to 0 percent. Nobody sees its changes any more. If you are preparing a sale launch at the same time, you want to know that in advance.

Adopting a winner

At the end of an experiment you decide: roll back or adopt permanently. Once adopted, it stays adopted. According to Shopify, it cannot be undone.

The experiment works with copies. According to Shopify, each experiment has its own independent copy of the control, and the variant is a copy too. The copies are made when you create the experiment, so the live theme can still be edited in the meantime.

Shopify does not say what happens on adoption to changes made to the live theme in the meantime. A banner swap, a new trust badge, a bug fix during three weeks of testing: whether they survive adoption or get overwritten is something to check in a test shop before you find out in your real one.

With two parallel experiments that both win, the question comes up twice, because each sits on its own copy.

In a classic testing programme, a winner is built into the theme as a single change. There is one theme, and nobody has to reconcile copies against each other.

What the report leaves out: revenue

For a theme experiment, Shopify shows four metrics per variant: conversion rate, bounce rate, the share of sessions that reach checkout, and the add-to-cart rate, meaning how many visitors put something in the basket. On top of that, sessions and confidence intervals: bars that show the range the true value probably lies in.

According to the Help Center, revenue, order value and number of orders only appear in the report for a launch, not for an experiment. The metrics cannot be customised. Shopify documents no export or API for the test data. The report also gives no breakdown by device or traffic source.

A worked example, a worked example, no client figures. A cosmetics shop tests its free shipping threshold: 49 euros instead of 79 euros. Each variant gets 30,000 sessions during the test.

Rollouts shows: conversion rate up 15 percent. At the same time, revenue per visit has fallen from 2.10 euros to 2.00 euros, almost 5 percent. At 60,000 sessions a month, the shop loses around 6,000 euros in revenue a month after adopting the variant, plus the shipping costs it now pays itself more often.

More buyers with smaller baskets is exactly what a lower shipping threshold or a discount can trigger. That is why we decide tests on revenue per visit. We look at conversion rate to understand why a test turned out the way it did.

“More data is needed”: the statistics

As long as a result is not reliable, Shopify’s confidence intervals carry the note “More data is needed for statistical confidence”. Which method sits behind it, which confidence level Shopify uses and when the note disappears is not in the documentation.

In one Rollouts report I looked at, Shopify’s intervals were noticeably wider than a standard test on the same numbers would give. A simple two-proportion test would already have called the result significant. Shopify did not yet. That is conservative maths. When testing, I prefer that to the opposite.

Four things help when reading a report like this:

1. Ask for the number of orders per variant before you talk about percentages. Below 300 orders per variant, decimal places are decoration.
2. Overlapping bars do not automatically mean “no difference”. Whether two variants differ is answered by a test on the difference, not by looking at two separate bars.
3. Set the runtime in advance, at least two full weeks. If you check daily and stop at the first green arrow, you will find a winner far more often than there really is one.
4. Check how the metrics are defined. If the conversion rate is higher than the add-to-cart rate, visitors may be buying without going through the basket, for example via buy-now buttons or express payment methods. Then you cannot simply read the funnel steps one below the other.

What you can actually test

Shopify puts it in one sentence: “You can’t change Liquid templates as part of a rollout.” Liquid is the language a Shopify theme is built in. It defines how a page is structured and what is shown when.

In a rollout, you edit the variant in the theme editor. That lets you change anything your theme offers as a setting: rearranging sections, showing and hiding them, text, images, colours, spacing. For a new hero on the homepage or a different order on the product page, that is enough.

Whatever the theme does not offer as a setting cannot be done this way. A delivery date on the product page calculated from the time of day and shipping rules. A unit price that recalculates live with volume discounts. A basket that behaves differently than before. Changes like these need code, and code means Liquid or JavaScript in the theme.

The workaround: build a complete second theme with the code changes and use it as a replacement in the rollout. Then a full copy of the theme runs alongside the live theme until the test is over, and you are back at the 0 percent trap from above as soon as theme changes run in parallel.

Shopify does not document whether a custom Liquid block, which many themes offer in the editor, works in a rollout. We have not tested it.

For a testing programme this means: the simple tests work with built-in tools. Tests that add new functionality to the page run on whole replacement themes.

While an experiment runs, nobody should touch the copies either. According to Shopify, any change to the control or the variant affects the result.

And page speed?

Here Rollouts has a real advantage. The variant comes straight from the server, with no extra script and no flicker. A classic testing tool loads code in the browser afterwards.

In a modern setup that is around 40 KB compressed, loaded asynchronously, and it does not block page rendering. For changes above the fold the page is held for 300 milliseconds at most so nobody briefly sees the original. That is the price, and I would rather name it than play it down.

But Rollouts cannot do without code either once you want revenue per variant. My advice on the page speed question: measure, don’t guess. Load the same page 20 times with and 20 times without the test script, spread across the day, and look at which element actually determines the load time. It is not always the one you suspect.

If you test with Rollouts anyway

Then build revenue tracking behind it. The effort is manageable:

1. Set a fixed identifier in both theme copies: “control” in the control and, for example, “shipping-49” in the variant.
2. Write the identifier into a cart attribute (Shopify Cart API, cart attributes). Then every order shows which variant the customer saw.
3. Also send the identifier to GA4 as a parameter. With the BigQuery export you get revenue and sessions per variant and can break them down by device or channel.
4. Before you start, calculate which effect your traffic can actually detect in the planned runtime, using the share that really remains after parallel rollouts.
5. Decide on revenue per visit, and check adoption in a test shop first.

One limitation remains. Shopify does not document how long it keeps a visitor assigned to the same variant. The identifier in the basket therefore shows the variant at the time of the basket, not necessarily that of every earlier visit.

What we take from this

Sources

• Shopify Help Center: Rollouts, Rollout types, Create a rollout, Managing rollouts, Rollout analytics, Requirements and considerations, accessed 18 September 2026
• Shopify Changelog: Schedule, publish, and A/B test new themes, 5 June 2026