Should I pause or end my A/B tests for Black Friday?

If a test has to stop for Black Friday, pause it when you will restart it within days, and end it when it would sit idle for weeks. A pause can hold a result steady through a short break, but over weeks more returning shoppers come back unrecognized and can land in a different group.

If your team wants the site left alone for Black Friday and Cyber Monday (BFCM) while a test is still live, decide what happens to that test first.

If a test has to stop, pause it for a short sale and end it if it would sit idle for weeks

The deciding question is how long the test will sit idle.

• A sale that lasts a few days is a short break. Pause the test and pick it back up when the sale ends.

• Some brands stop all site changes for weeks before BFCM. A test that would sit idle that long is better ended and launched again afterward.

A short pause is safe when shoppers keep their group and sale-day orders stay out of the results

A pause stops a test without deleting it.

New visitors are not added. Shoppers already in a group stop seeing the test until it resumes, then see their version again.

For a pause to protect your result, two things must hold:

• Shoppers keep their group.

• Orders placed during the pause do not count.

In Intelligems, both hold for every test type. If you use another tool, check that it does the same.

In a long pause, a shopper whose cookie expires can come back in a different group

A test keeps each shopper in their group by remembering them in the browser, and that memory expires. In Intelligems, Safari and every iPhone browser delete the cookie after seven days without a visit, while Chrome keeps it for about a year.

A Safari or iPhone shopper who stays away through a long pause comes back unrecognized once the test resumes. They get assigned again and can land in a different group than before.

A long pause also uses up calendar time. Intelligems advises finishing a test within five weeks, because shoppers switch devices and clear cookies. A pause of several weeks spends that time while the test collects no orders.

A pause can leave test prices live, so check prices and shipping before you pause

A pause does not always put your store back the way it was, so check what a paused test leaves live in your tool.

In Intelligems:

• Pausing a price test stops the test prices, but Shopify keeps the highest price from the test unless you choose to revert prices when you pause. A percentage-off sale would then take its discount off that higher price.

• Pausing a shipping test does put your original rates back.

You can schedule a pause for a content test, but a price or shipping test has to be paused by hand.

End the test on your original version, then relaunch it as a new test after BFCM

Ending closes a test cleanly.

When you end a price test in Intelligems, you pick which group’s prices stay in Shopify. Choosing the control group leaves your original prices exactly as they were.

After BFCM, launch the test again as a new one. It starts over and assigns every visitor fresh.

List each live test with its restart date and bring the list to Intelligems before you stop any of them

Write down every live test, what it changes and when you plan to restart it.

Tests you will pick back up right after a short sale get a pause. The rest get an end.

If you test with Intelligems, bring that list to Intelligems support before you stop anything. Support can:

• set a price or content test back to paused if it gets ended by mistake (an ended shipping test cannot be restarted);

• turn on scheduled pauses and ends for content tests, if you do not see the option to schedule.