Should I A/B Test Less Now That I'm Using AI?

AI has made it easier and faster to introduce changes to your store. A new variant, new copy, a new layout... now updated in minutes instead of days. It's tempting to read that as a reason to run fewer A/B tests, since you can just ship and fix it later. But it's actually the opposite. When shipping gets easy, you ship more, and every change you push live without measuring is a bet you're placing blind. More bets, less certainty on each one. That's why the case for testing gets stronger as AI gets more mature. Here's how to think about it.

The old limit on testing was how fast you could build

For years, the thing that capped how much a brand could test wasn't ideas. It was execution. Research, briefs, design, development, QA... every step took a person and real time. So brands rationed. You tested the handful of things that felt worth the effort and shipped everything else on instinct.

AI is removing that limit. Tools can now draft the brief, build the variant, and help analyze results. The cost of trying something has dropped dramatically, and that's genuinely good news. It also quietly changes the math in a way that's easy to miss.

Doing more raises the stakes of doing it wrong

When it was expensive to ship a change, you shipped fewer of them, and each one usually got some scrutiny. When it's nearly free, you ship a lot more, and the temptation is to skip the scrutiny... you can always change it back, right?

The catch is volume. Ten unmeasured changes a quarter is a manageable amount of guessing. A hundred unmeasured changes a quarter is a store drifting in a direction nobody actually chose. The faster you can act, the more it matters whether you're acting on something real. Speed without measurement isn't progress. It's just motion, and motion can carry you the wrong way as easily as the right one.

Victor and I spent the whole first episode of Live with Intelligems on this exact shift, and it's the one idea that stuck with me most: the business case for experimentation has never been stronger. The cost and time to ship something new is the lowest it has ever been. So the thing that's actually scarce now isn't the ability to build... it's knowing whether what you built helped the business move in the intended direction.

AI is trained on everyone's store, not yours

There's a deeper reason, and it's baked into how these tools work. AI is very good at producing what generally works. Ask it for a higher-converting product page and you'll get the patterns it has seen a thousand times: an urgency banner, trust badges, a punchier headline. Reasonable defaults. But "generally works" is exactly the trap experimentation exists to help you escape.

What lifted profit for another brand was optimized for their customers, their margins, their catalog. Dropping the same change into your store is a bet that their shoppers and yours behave the same way. Sometimes they do. Often they don't. The only way to tell which is to run it against your own traffic and watch what your customers actually do.

This is the part worth sitting with. AI raises the floor on what a change looks like, but it can't tell you whether that change was right for your business. The generic best practice you copied from a LinkedIn thread doesn't get more true just because an AI generated a cleaner version of it.

EP 040: why a real experimentation mindset starts from your own business and your own data, not the best practices everyone else is copying.

How to keep measurement in step with shipping

If AI is going to help you ship faster, the fix isn't to slow down. It's to point some of that saved time back at measurement so the two stay in balance.

The cleanest way to do that is to build inside a tool that measures by default. Intelligems' AI visual editor lets you describe the change you want and builds the variant for you, and because it lives inside the testing platform, that variant ships as an experiment instead of a blind change. The thing you spun up in minutes gets measured without any extra step.

Beyond the tooling, a few habits keep speed and measurement in step:

• Decide what a change has to prove before it goes live. Not everything needs a formal experiment, but anything touching price, offers, or checkout usually does. Those are the changes where being wrong is expensive.

• Measure profit, not just whether the number moved. A variant can lift conversion and quietly cost you margin. Watching profit per visitor keeps you from celebrating a change that actually made you less money.

• Let the data settle. AI can build a test in minutes, but it can't make your customers decide any faster. Give it the two to three weeks it typically needs to reach a high likelihood of beating control before you call it.

None of this is about adding friction for its own sake. It's about making sure the volume AI unlocks is pointed somewhere useful.

Speed stops being the advantage

When building was hard, being fast was an edge. Now that everyone can be fast, speed stops being what sets you apart. What's left is knowing... reading your own data well, understanding what a result means for your store, and being right more often about what to try next.

That's the part AI hands back to you, not the part it takes away. If it clears the operational work off your plate, the highest-value thing you can do with that time is get sharper at the judgment underneath it. The brands that pull ahead won't be the ones shipping the most. They'll be the ones who know which of those shipments actually worked.