How to A/B Test Your Shopify Checkout: Lessons from 89 Tests

Guest post by Federico Reyes Arce, CRO and Growth Strategist at DTC Pages.

As a CRO agency for Shopify brands, we lost one of our main testing grounds when Shopify moved to Checkout Extensibility: for a while, the checkout stopped being testable for us. It was the only page in the funnel where we were making changes without being able to measure them. That has changed: tools like Intelligems let you test it again. At DTC Pages we’ve run 89 checkout experiments since 2022, across 26 brands, and 46% of them won. Since Checkout Extensibility arrived, we’ve run 37, most of them in Intelligems: 17 won, 4 lost, and 16 were inconclusive. The same 46% as before.

In this post, we cover what you can test in the checkout today, how we set up each type of test, how we read the results, and what you still can’t test.

What you can change in the Shopify checkout today

Checkout Extensibility still limits what you can test in the Shopify checkout, but you can test more than you might think. The most common tests are content blocks: trust badges, USPs, reviews, timers, progress bars, FAQs, or upsells. For these, the store needs to be on Shopify Plus. You can also turn Shopify’s native blocks (Checkout Blocks) on or off, or compare versions of them.

Not everything is content, though. You can also test shipping rates and free shipping thresholds, product prices, and express payment buttons like Shop Pay or Apple Pay, and none of these require Plus. If a third-party app is integrated with Intelligems, you can test things like shipping protection or gift wrapping too.

Most of the tests we’ve run since Checkout Extensibility were content tests: where to place trust signals, how to show reviews, and how much urgency to add.

What to test first (and why “best practice” isn’t enough)

If you’re just starting to test your checkout, our advice is to move what you already have before adding anything new. The tests that won most consistently were the ones that took existing trust signals (shipping, returns, guarantee, secure payment) and moved them above the fold or next to the express payment buttons. Across different brands, that lifted conversion by 3% to 7.7%. Next come reviews in the checkout, which won with +10.3% and +14.4% at two brands, and moderate urgency (a “cart reserved for 15 minutes” timer added 4.3%).

But a best practice won’t necessarily work for your store. Adding a new USP block cut conversion by 3% at one brand, and adding press mentions at the bottom cut it by 6.3% at another. Generic upsells in the checkout have often lost for us, and removing them won. But even winners can hide trade-offs: at one supplement brand, removing an upsell lifted conversion by 3% while lowering AOV by 1.7%. Winners also tend to be small: the median lift is +3.7% in conversion. That’s why it pays to test every change before you ship it, even when it looks obvious, and to look at every KPI, not just conversion rate.

Shipping protection is a good example of how much the result depends on the store. We’ve tested both adding and removing it across different brands. Broadly, adding it lifted conversion for brands with a higher AOV and hurt it for brands with a lower AOV. That’s not a rule, just what we’ve seen, which is exactly why it’s worth verifying in your own store.

The full list of what to test in the checkout, with results from all 89 tests, is in our post on Shopify checkout optimization.

How we set up each type of checkout test in Intelligems

Most of our checkout tests are content block tests. They’re simple: the variant shows a new or repositioned block, and the control stays as it is. That’s how we run different kinds of tests: USPs above the fold, trust banners, timers. Placement matters, because the same block can perform differently depending on where you put it. As we said earlier, a lot of the time we just moved things that already existed above the fold or somewhere else, and that improved both conversion and revenue.

Third-party apps work differently. In our shipping protection tests, for example, the app was integrated with Intelligems: Intelligems tags each visitor’s cart with their test group, and the app reads that tag to decide whether to show itself. That’s why you can only test apps that have this integration. Before planning a test with an app, the first thing we check is whether it has one.

Shipping tests are the easiest place to start if your store isn’t on Plus. With Intelligems, you can easily test shipping rates and free shipping thresholds, and the change shows up directly in the checkout.

Whatever the test, QA matters more here than anywhere else. The checkout is the last step of the funnel, where the shopper decides to buy. Check on every device that everything looks right and that nothing is broken. In the checkout, a bug doesn’t just ruin the test: it costs real sales for as long as the test is running.

How to read checkout test results

Most checkout changes are only seen by shoppers who reach the checkout, so it makes no sense to read the result across all of your site traffic: the effect gets diluted by visitors who never saw it. Intelligems lets you filter down to only the shoppers who reached that stage (under User Behavior, “Started Checkout”). The exception is shipping tests, like a free shipping threshold: the result shows up in the checkout, but the change is communicated across the whole site (product pages, the cart, the announcement bar), so we read those on all traffic.

We also split results by device, because the same change can move in opposite directions on mobile and desktop. In several tests that moved USPs or trust blocks above the fold, mobile won, and desktop lost. At one supplement brand, moving the USPs above the fold lifted conversion by 3.2% on mobile and cut it by 5.5% on desktop. At another, a trust block above the form won 3.4% on mobile and lost 4.2% on desktop. In those cases we shipped the change on mobile only. If we had only looked at the total, we would have either missed the extra revenue on mobile or rolled out something that was costing us revenue on desktop.

Finally, don’t stop at conversion rate. To decide whether a test won or gets discarded, we also look at revenue per visitor, AOV, and profit per visitor. And since the checkout gets less traffic than the rest of the site, tests there can take longer to reach a result than on other pages, so it’s worth planning the duration before you launch.

An Intelligems episode on testing the Shopify checkout, including why placement and results can differ on mobile and desktop; that part starts at 13:30

What you still can’t test in the checkout

It’s also worth knowing what you still can’t test, so you don’t waste time on it. A lot of clients ask us to test the checkout layout: one page versus three steps. That isn’t possible. Shopify only allows one active layout, so you have to pick one, and you can’t change the order of the steps either. You also can’t compare a “clean” checkout against your current one: if your checkout runs several apps that don’t go through Checkout Blocks or Intelligems, you can’t build a version without them and test it against what you have today. And, as we mentioned, apps that aren’t integrated can’t be tested, and without Shopify Plus there are no blocks inside the checkout.

Checklist: starting to test your checkout

• Start by moving what you already have before adding anything new.

• Change one thing per test.

• Check every variant on mobile and desktop.

• Read results on shoppers who reached the checkout, and split them by device.

• Measure revenue or profit per visitor, not just conversion rate.

• If you’re not on Shopify Plus, start with shipping and pricing.

Federico Reyes Arce is a CRO and Growth Strategist at DTC Pages, a CRO agency for Shopify brands. For the full breakdown of what to test in the checkout, read our Shopify checkout optimization post.