Should I Keep Running Tests During a Sale?

The concern about running an underperforming variant during your biggest sale is valid. But that's a reason to be selective about what you test, not to stop testing. Offer tests are worth running during peak periods because what you learn applies directly to the next one. Here's what that looks like in practice.

The Instinct to Pause All Testing During a Sale Is a Misread

Here's what most brands are worried about: if you're running a test during your biggest sale of the year, some portion of your traffic is seeing a variant that hasn't been validated. If that variant underperforms, you're giving away revenue during the period when it costs the most. That's the fear, and it's a reasonable one.

But the risk is more specific than it seems. Once you understand which tests actually carry it, the instinct to pause everything starts to look like unnecessary caution.

There's also a practical upside most brands miss: sale periods bring higher traffic, which increases your statistical power. What might take three weeks to read clearly in August can be readable in days during a major sale.

Not every test is equal during a promotion. A price change or navigation update will behave differently when purchase intent is already elevated. What wins during Father's Day might not hold in August. Those learnings are harder to apply outside of a promotional context, and that's a real limitation worth respecting.

But offer tests are a different story. When you test how your offer performs during a sale, you're generating data that applies directly to the next sale. Father's Day teaches you something for 4th of July. Labor Day teaches you something for BFCM. It's exactly the right data for the question you're asking.

What to Test: Your Offer, Three Ways

We've seen thousands of offer tests run across Shopify brands. A few things become clear when you look at that many experiments: three variables tend to drive the most meaningful differences in how an offer performs. (For a broader look at holiday testing setup, The Ultimate Guide to A/B Testing During the Holidays is worth a read.)

1. Who gets what

Your email subscribers who clicked through a Father's Day promotion aren't the same visitor as cold paid social traffic finding your site for the first time. Sale periods also shift the overall audience mix, bringing more new visitors than usual, people encountering your brand in a promotional context with no prior relationship. Different intent, different price sensitivity. Showing everyone the same offer is a guess.

Intelligems lets you target by traffic source across all test types: show a different banner, offer, or experience to email traffic versus paid, and measure which version drives more profit per visitor. The audience distinction often turns out to matter as much as the offer itself.

2. Which mechanism

Percent off, dollar off, free gift, free shipping, buy X get Y, buy 1 get 1 free. The same underlying value can perform very differently depending on how it's framed. A free gift and a 15% discount might cost you roughly the same, but one can create a response the other doesn't.

Testing the mechanism during Father's Day or Labor Day tells you something real before BFCM, when the traffic is highest and the cost of guessing is largest.

3. How much

Every promotion has a minimum effective discount: the smallest offer level that actually changes buying behavior. If 15% drives the same purchase rate as 25%, the extra 10 points is margin you gave away for nothing.

Smaller sale periods are the right time to find that number. Lower stakes, real promotional traffic, and an answer that carries forward to every sale after.

The Variable Most Brands Overlook

All three of those variables share a common failure mode: if the offer isn't clearly visible, you're not measuring the offer. You're measuring whether people noticed it. Customers who don't understand what's on offer won't change their behavior. They'll act the way they would have anyway, which means you end up discounting people who were going to buy regardless.

The offer needs to show up where people are making decisions: a homepage banner, a product page label ("qualifies for free gift"), a cart progress bar showing how close they are to unlocking the incentive. When you're testing offer mechanics, make sure visibility is consistent across both variants.

Offer Experiences handles the injection logic so getting placements live before the sale doesn't require a developer.

IntelliJAMS EP 061: whether to A/B test during a sale period, and what to focus on if you do.

What to Watch in the Results

Conversion rate will move during sale periods almost regardless of what you do. Higher intent, deadline pressure, more traffic. All of it inflates conversion. That makes it a noisy signal for anything you're testing.

The number that tells you something useful is profit per visitor. Did the variant drive more profitable purchases, or just more purchases? A deeper discount can win on conversion while losing on profit per visitor if you gave away more margin than the behavior change was worth.

Stop Guessing. Start Knowing.

Sale periods are a chance to run the tests that actually matter. The brands that arrive at BFCM knowing their minimum effective discount, their best-performing mechanism, and which audiences respond to what didn't figure it out in November. They ran the experiments when the stakes were lower.