Ecommerce Strategy
Which of my apps are actually earning their keep?
Hide an app from some shoppers.
Keep the apps your store would miss and cut the rest. To tell which is which, hide each app from part of your visitors and compare profit per visitor across the whole store. If the group without the app makes as much or more, the app isn’t earning its keep. If the group with it makes more, check that the gain covers what the app costs you.
In December a lot of teams look at their app list and decide which tools to keep for next year. Often that call rests on who on the team uses a tool and a general sense that it helps. A test replaces that sense with your own numbers.
A widget can keep selling and still lower your store’s revenue
An upsell app’s report counts the add-ons it sold. What it can’t count is what those shoppers would have bought without it. Testing inside the app has the same blind spot: an upsell tool can compare two of its own offers, but it can’t show what having an upsell there at all does to the rest of the store.
An ecommerce operator on Intelligems’ podcast counts upsell tools among the easy ones to judge, because their sales show up directly. A footwear brand’s cart drawer shows how a widget’s sales can mislead.
The brand tested three versions of the drawer: the cross-sell it already ran, a different add-on and empty space. The empty drawer made more revenue per visitor than the original. The store’s own theory is that the add-on sent shoppers off to browse, and some never came back to check out. The second add-on sold far more often than the first, yet order value came out almost the same.
On the product pages, early results pointed the other way: hiding the same add-on there looked like a loss. One store, two placements, two different answers, and neither would have shown up in a report of what the add-on sold.
Hide the app for some shoppers and compare profit per visitor
The check is an ordinary A/B test with the app as the change:
Hide the app’s element for one group and leave the site as it is for the other.
Compare profit per visitor across the whole store and on the pages where the app appears. Profit per visitor shows whether the store made more after product costs, something conversion rate alone can miss.
When you’re choosing between two apps, add a group that gets neither. App A against app B only tells you which of the two does better; the empty group shows whether either one earns its place.
Count what the app does later. An email pop-up collects addresses that pay off after the test ends, so weigh its signups per visitor from your email tool next to the profit result.
Wait for enough orders. Intelligems’ guidance is at least 300 orders per group and seven days before reading a result, and even then a clear answer isn’t guaranteed.
You don’t have to build it by hand. If you use Intelligems, open Atlas in the app, or connect Claude or ChatGPT through the Intelligems MCP, and ask in a sentence: “Set up a test that hides the reviews widget from half my visitors.” Before you start it, run Atlas QA: it checks the setup and browses your store the way a shopper would. It’s a first check, not a guarantee, so look the test over yourself, then start it. If your team prefers to write it, a few lines of CSS in a Styles and JavaScript experience hide a widget without touching your live theme.
The split doesn’t have to be even. If hiding an app from half your shoppers feels like too much risk, keep it for most visitors and hold it back from a smaller group.
Then weigh the result against the fee. Say your store gets 40,000 visitors a month and the app costs $400 a month. The group with the app has to make at least a cent more profit per visitor just to pay for it. Less than that, and the app is costing you money.
An Intelligems podcast episode on deciding which apps to keep for next year; an ecommerce operator explains hiding an app’s element to test it, starting at 2:36.
Start with the doubtful app the most shoppers see
Two things decide which app to check first. One is doubt: the app your team can least explain the value of. The other is reach: how many shoppers actually see it.
A widget on a few pages is easy to hide, but few shoppers see it, so its test collects orders slowly. When the footwear brand hid its add-on on the few product pages that showed it, the test was slow to give a clear answer. Something most visitors meet, like a sitewide email pop-up, gets there sooner.
When doubt and reach pull apart, start with the app more shoppers see. You’ll have an answer while there’s still time to act on it.
Some apps can’t be tested this way, and some don’t need a test
A few kinds of app fall outside this check:
Checkout apps such as shipping protection, gift wrap or donations depend on the app itself. In Intelligems, one can be shown to one group and held back from another only if it can read a cart attribute, a tag the test adds to the cart to mark the shopper’s group. Checkout testing also needs Shopify Plus.
An app change that’s small, low-risk and one you’re already confident about can skip the test and go straight to every shopper.
Tools you’ve stopped using don’t need a test at all. Their scripts can keep firing after you stop using them, so remove them.
Choose in December, test once your site opens up
Many stores don’t touch their site in November until the holiday sale is over. If yours is still frozen, use these steps to have the first test ready for when it opens up:
List the apps you pay for and mark the ones nobody can show are helping.
Write down what each one should move: orders, order value, signups.
Load your product costs so the test shows profit.
Pick the first app and have its test built as a draft now, so it’s ready to start once your site opens up again.
Then let each result decide, and move on to the next app on your list. As the footwear drawer showed, the answer can change with where a widget sits, so test each app where your shoppers meet it.
Ecommerce Strategy
AB Testing
Content Testing
