When Can I End My A/B Test?

You're staring at your test results. Variant B shows 82% chance to win. But what does that actually mean? Should you act now? Wait for 95%? Keep checking until something feels certain?

The real question? Am I confident enough to decide?

It's tempting to treat statistical significance like a finish line to cross. It's either significant or it's not. Binary. But that's not how confidence works in business. Confidence is a spectrum. And the level you need depends entirely on what's at stake.

What Does "Chance to Win" Actually Mean?

Confidence is a spectrum, not a finish line

When your test shows "82% chance to win," it means that based on the data collected so far, there's an 82% probability that Variant B would outperform Control if you ran this test many times over. That's not certainty, but it's a strong signal.

Think about it like a weather forecast. When the meteorologist says "85% chance of rain," you pack an umbrella. You don't need 100% certainty to make a good decision. And if it doesn't rain, you just carry an umbrella for nothing. No big deal.

The same logic applies to your test results. 80% confidence means you'd be right 8 out of 10 times if you made this decision repeatedly. That's a strong signal for most business decisions.

How Do I Read My Confidence Interval?

Don't just look at the middle number

Your test dashboard shows more than just "Variant B is up 5%." It shows a range of possible outcomes. This range is your confidence interval, and it's one of the most valuable pieces of information in your results.

Let's say your test shows Variant B with a 5% improvement in profit per visitor. But the confidence interval shows it could be anywhere from a 2% decrease to a 12% increase. That range tells you something important: even in the worst realistic scenario, you'd only be down 2%.

Now flip the question. Instead of asking "Is this statistically significant?" ask yourself: "Am I okay with the worst case?"

If the worst realistic outcome is a 2% decrease and the best case is a 12% increase, that's probably worth pursuing even at 75% confidence. You're not betting on certainty. You're betting on acceptable risk.

Why Is 80% Confidence Often Enough?

Most business decisions are two-way doors

The 95% confidence threshold exists for a reason. It came from medical trials and academic research where the stakes demanded near-certainty. Getting a drug wrong could cost lives. Getting a research finding wrong could mislead science for decades.

Testing your prices isn't life or death. You can change them back tomorrow.

This is what Jeff Bezos calls "two-way doors." Most business decisions are reversible. You walk through the door, see what's on the other side, and if you don't like it, you walk back. For two-way doors, 80% confidence is often plenty.

One-way doors are different. Launching a product that's impossible to recall. Signing a long-term contract. Making an irreversible public commitment. For those decisions, aim higher.

This episode dives deeper into confidence vs. certainty and why most business decisions don't require 95% confidence.

But What If I Have High Traffic?

Time captures different customers, not just more customers

If you're getting 100,000 sessions a day, waiting two weeks can feel absurd. That's 1.4 million sessions. Surely you have enough data after a few days?

But more sessions doesn't mean you've seen all the patterns. Monday shoppers behave differently than Saturday shoppers. Week one might catch payday for half your customers. Week two catches it for the other half. Someone browsing on Tuesday might not buy until the following weekend.

Two weeks captures different types of customers and different buying contexts, not just more data. A test that runs Monday through Friday misses entirely different shopping behavior than one that captures two full weekends.

High-traffic sites actually have an advantage here. You'll hit statistical confidence faster within those two weeks. But cutting the time short means your confident result might only reflect one slice of your customer base.

There's another threshold that matters: order volume. High confidence percentages can be misleading if they're based on a tiny slice of your normal business.

If you typically get 10,000 orders a month and you're making a decision based on 50 orders, that sample doesn't represent your real customer behavior. Early results are noisy. What looks like a clear winner in the first few days might flip completely once you've captured enough orders to reflect how your store actually performs.

The good news? If you're hitting 80%+ confidence after one week and the worst case is acceptable, you can make a call sooner. Just know what you might be missing.

What's the Simplest Framework for Deciding?

Ask yourself: what's the worst case if I'm wrong?

When you're staring at your results wondering whether to act, run through this simple framework:

Step 1: Look at your chance to win.

What's the probability that your variant beats control? Above 80% is a strong signal. Above 90% is very strong.

Step 2: Look at the range of outcomes.

Don't just look at the middle number. Look at the best case and worst case in your confidence interval. Your analytics should show you this range.

Step 3: Ask "Am I okay with the worst case?"

If the worst realistic outcome is acceptable, you have permission to act. A 2% downside with 15% upside is almost always worth taking.

Step 4: Ask "Can I change this back if I'm wrong?"

Most pricing decisions are reversible. If you can undo the change, you don't need certainty.

Step 5: If yes to both, act.

Don't wait for perfection. Act when the downside is acceptable.

Common Mistakes When Reading Results

Perfect confidence is the enemy of good decisions

Only looking at the middle number. The single number ("profit is up 5%") hides important information. Look at the range. Even if you're "losing" in the middle, the worst case might be totally acceptable.

Waiting for 95% on reversible decisions. That's overkill. If you can change your prices back tomorrow, 80% confidence is often enough. Save 95% for decisions you can't undo.

Ignoring the range of outcomes. Two tests can both show "85% chance to win" but have very different risk profiles. One might have a tight range (3-7% improvement). The other might have a wide range (-2% to 15%). Same confidence, very different decisions.

Not considering what happens if you're wrong. The question isn't just "Am I right?" It's "What happens if I'm wrong?" If the answer is "I change the price back," that's a two-way door. Act faster.

Stop Waiting for Certainty. Start Deciding.

You might be checking your tests obsessively, waiting for a moment when the results suddenly become "done." You want someone to tell you it's safe to act.

That moment rarely comes. Confidence is a spectrum, not a finish line. The right threshold depends on the stakes, not some arbitrary academic standard.

Make confident decisions:

• Check your confidence level. Aim for 80%+ on reversible decisions, higher for permanent ones.

• Look at the range of outcomes. Best case to worst case, not just the middle.

• Ask if you can change it back. Most pricing decisions are two-way doors.

• Consider what happens if you're wrong. Is the downside acceptable?

• Act when downside is acceptable. Not when you're 100% certain.

Don't wait for certainty. Know when you're confident enough!

Ready to stop waiting and start deciding? When you're ready to make confident calls on your pricing and testing, let's get you testing beyond what's typical.