Why Does My CAC Go Up When I Run A/B Tests?

You launch a test. A day or two in, you check your ad account. CAC is up. CPMs look off. And the test is the most obvious thing to blame. It's the one thing that changed.

So you pause it. Or you stop testing altogether.

This happens a lot. And while the instinct is understandable, the conclusion usually isn't right. Here's why.

CAC Is a Cost Metric. It Doesn't Tell the Whole Story.

CAC is calculated as ad spend divided by orders. It's order-denominated. That means any test that trades conversion rate for order value will show CAC rising. Not because something went wrong, but because the math works that way. Some paid media teams track this as blended CAC, others as NCPA (new customer acquisition cost) specifically. Either way, the same logic applies.

Run a shipping threshold test that raises the bar for free shipping. Fewer people qualify, so conversion rate dips slightly. CAC goes up. But average order value increases for the customers who do buy. Whether that tradeoff helped or hurt the business isn't visible in CAC alone.

That's what profit per visitor is for. It holds conversion rate, order value, and margin together in one number. It's the metric that actually answers the question.

ROAS is worth watching too. If a test decreases CVR while increasing AOV, CAC goes up but ROAS goes up at the same time, because you're generating more revenue per dollar of ad spend. Watching only CAC in that scenario means watching a cost metric rise while a revenue efficiency metric improves, and drawing the wrong conclusion from it.

The harder question is separate: can running a test itself degrade your ad platform performance, independent of what the test results show? That one deserves a more honest answer.

The Relationship Between Tests and Ad Performance Isn't Linear

Here's where it gets genuinely interesting. The relationship between your on-site test results and your paid channel performance isn't a clean 1:1, and it's worth understanding why.

When a variant decreases conversion rate... even slightly, even while AOV goes up... ad platforms can penalize you with higher CPMs. That's real. Platforms like Meta reward higher conversion rates with more efficient delivery, so the inverse can happen too. A test that wins on average revenue per visitor might not translate into the ROAS improvement you'd expect on paper, because a CVR dip triggered a CPM increase that absorbed part of the gain.

There's a second dynamic that's harder to prove but difficult to dismiss entirely: when a test raises your revenue per order and a platform can see that your customers are worth more, it may simply charge you more to reach them. They're businesses optimizing their own P&L too.

The honest read is that this relationship is complicated, and the data can be messy enough that it's easy to confuse correlation with causation.

A lot of it comes down to timing

Part of what's happening is a timing problem. You launch a test, then you open your ad account an hour later and something looks off. CAC spiked. CPMs ticked up. And because you just launched a test, it's the first thing you blame. But paid channel metrics are noisy by nature. CAC fluctuates throughout the day, across days of the week, with auction competition, with creative fatigue, with iOS signal loss, with a hundred things that have nothing to do with what's on your site. Looking at your ad accounts within hours of starting a test and drawing conclusions is rarely meaningful... you're pattern-matching on noise.

Actually, seasoned practitioners recommend running a test for at least one to two weeks to capture full buying cycles. The mismatch between how quickly people check their ad accounts and how long a test actually needs to run is probably responsible for a lot of the fear around this.

That said, the comparison between variants holds regardless. Because Intelligems and other properly designed testing tools randomize visitors from the same traffic pool, both variants get the same traffic quality. If CPMs rise during the test window, both groups absorb it equally. The comparison stays clean. A variant that wins on profit per visitor still wins, because the ad spend component cancels out across both sides.

For this to break down, you'd need platform degradation so extreme and so unevenly distributed that it completely overwhelms the on-site signal. That's a very high bar.

Not All Tests Work the Same Way

It's worth making a technical distinction that often gets lost in this conversation, because the type of test you're running matters quite a bit.

Price tests and shipping tests in Intelligems operate at the Shopify infrastructure level, through Shopify's Cart Transform Functions and Third Party Rate Carrier API respectively. They're server-side by nature. The URL doesn't change, and there's no redirect. From the platform's perspective, there's no signal that typically triggers re-learning.

Content tests use client-side JavaScript that modifies the page after it loads. The URL stays the same, so ad platforms see no redirect, but there is script running in the browser. Whether that's noticeable to a platform depends on the scale of the changes.

Split URL tests are a different situation. When a visitor gets redirected to a different URL... say, from /collections/welcome to /collections/welcome-v2... that redirect is visible. Ad platform pixels fire on a different URL. If the destination page is substantially different from the original, there's a more plausible mechanism for platforms to register a change and potentially adjust delivery.

A lot of the concern that circulates about A/B testing and paid performance stems from experiences with redirect-based testing tools, where this effect is most plausible. That history tends to get applied broadly to all testing approaches, which isn't accurate.

How to Actually Check What's Going On

If your CAC is moving during a test and you want to know whether the test is the cause, here's a practical approach.

You can run a check in GA4, or whatever analytics tool your paid team already uses. Pull your paid channel's session data and compare conversion rate and revenue per session for the two to four weeks before the test against the same window during it. If those numbers held steady, the alarm is probably coming from daily noise in your ad account. If they moved in the same direction as your CAC, you have a real signal worth digging into.

We've run this analysis for several brands that raised the concern. No anomalies in paid social during the test periods. The traffic spikes that appeared showed up across paid, organic, and direct channels simultaneously. That pointed to something external rather than anything the tests introduced.

If CAC does increase during a test, run the calculation directly: did profit per visitor improve by more than CAC increased? A CAC increase alongside a PPV increase might be a trade-off worth making. A CAC increase with flat or declining PPV is a different conversation.

If You Want to Randomize Traffic at the Ad Platform Level

If you're still not convinced after all of this, there's an option worth knowing about. Intelligems lets you assign specific URL rules to specific variants, including UTM-based rules. That means you can route traffic from a particular source directly to a particular variant and analyze its performance in isolation.

The logic is about separating two things that often get conflated: randomization and measurement. Ad platforms can split an audience. But measuring what that audience actually did — revenue, margin, profit per visitor — is better done with data that comes directly from your store. Ad platform reporting runs through its own attribution models, which aren't always designed to give you an objective read on your site's performance. Intelligems analyzes results from Shopify data directly, so the measurement is clean and unaffected by how the platform accounts for conversions on its end.

The catch, and it matters: this only works properly if you're doing true randomization at the ad platform level. That means running a proper 50/50 random split within the same audience pool, not two separate ad sets targeting different audiences. The moment you let Meta or any ad platform optimize each creative to a different audience, you've lost the ability to isolate the variable. You're comparing two different groups of people, not two different experiences.

Done correctly, that means running an A/B test that evenly and randomly splits the same audience pool at the ad platform level.

From there, Intelligems' audience targeting lets you assign UTM-based rules to specific variants. Meta traffic from creative A goes to Variant A. Meta traffic from creative B goes to Variant B. The result is probably the cleanest view available of how each experience affects CAC, because you're controlling both the randomization and the attribution.

It's more setup. But if the CAC question is important enough to your team to warrant it, it's the right way to get a real answer. If you want help configuring it, reach out to your Intelligems Customer Success manager and they'll walk you through it.

The Real Cost of Not Testing

Every change you roll out to your site can affect your paid channels, whether you tested it or not. A new landing page, a price change, a different offer... all of it can trigger whatever platform adjustments follow. Testing doesn't introduce that risk. It gives you a way to measure it.

And the math on opportunity cost cuts the other way too. Even if a test temporarily affects your CAC in some hard-to-prove way, what you learn from that test can be capitalized on for months — sometimes years. Insights about pricing, shipping thresholds, or page experience don't just improve your website. They inform your ad creative, your offer strategy, your retention emails. Stopping testing to protect short-term CAC means forgoing that compounding value entirely.

CAC going up during a test is worth paying attention to. But correlation isn't causation — paid channel metrics are noisy, and attributing a move directly to an on-site test is harder to prove than it looks. And even if you could prove it, the better question is what it costs you to keep making changes blindly.