Which A/B testing tool should your Shopify store use?

Pick a Shopify A/B testing tool in four steps. Start with what you want to test, because price, shipping, and checkout tests need tools built for them. Then price each option at your own traffic, since most apps charge by visitors or orders. Read what reviewers complain about, especially flicker and page speed. Finally, make sure the tool reports revenue for each group alongside conversion rate. shopifyabtesting.com runs the first three steps for every app in the Shopify App Store.

Full disclosure before going further: Cascayd sponsors shopifyabtesting.com. Cascayd isn't ranked in its lists, and the sponsorship doesn't change how apps are scored [shopifyabtesting.com].

Start with what you want to test

Tools split along the kind of change they can test. Here is roughly what each kind involves on Shopify:

What you want to test	What the tool has to do
Theme or template	Show visitors a different theme, page template, or section layout
Content	Swap headlines, images, copy, or badges inside a page
Price	Charge two groups different prices for the same product, through to checkout
Shipping	Show two groups different rates or free shipping thresholds
Offer	Test bundles, discounts, or gifts with purchase
Checkout	Change the checkout itself

Price and shipping tests are the demanding ones. The tool has to change what shoppers actually pay, consistently, from the product page to the order. A tool that only swaps what shoppers see can't run them.

Checkout tests depend on your plan as well as the tool. Apps that customize the information, shipping, and payment steps of checkout are available only on Shopify Plus [Shopify]. If you're not on Plus, rule checkout testing out before you compare anything.

On shopifyabtesting.com, the "I want to test" filter narrows 73 apps to the ones whose listings say they can run your kind of test.

Price it at your traffic

Most Shopify testing apps charge by monthly visitors or orders, so the headline price on a listing is usually the entry tier. Your bill rises as your traffic crosses each tier.

Enter your monthly visitors, orders, and revenue on shopifyabtesting.com and every price on the site updates to the cheapest plan that covers your numbers [shopifyabtesting.com]. Do this before you shortlist anything. It changes the order fast.

Shopify's own Rollouts tool is the free option. Its Experiment mode needs the Grow plan or higher, and it tests theme edits, whole themes, and checkout configurations [Shopify]. Read the section on revenue below before you rely on it.

Read the complaints behind the stars

Most of the established testing apps rate between 4.5 and 5 stars, which makes the stars close to useless for choosing. The written reviews are where the differences show up.

shopifyabtesting.com read all 1,136 written reviews across 73 apps and tagged each one against 12 topics, including price, billing, support, setup, flicker, page speed, bugs, and reporting. A five-star review that says "great app, a bit pricey" still counts toward the price complaint rate [shopifyabtesting.com].

Two complaint topics deserve the most weight.

Flicker. Some tools load the original page, then swap in the variant a moment later. Shoppers see the change happen. That flicker becomes part of the variant, and it can sink an idea that would otherwise have won.

Page speed. Every testing app adds code to your store, and it costs you money on every visit, test or no test. At Amazon, a 100 millisecond slowdown cut sales by 1% [Kohavi et al., 2014]. Uninstalling an app doesn't automatically remove its code from your theme either [Shopify], so trialing five tools can leave you slower than when you started.

Make sure it reports revenue for each group

A test is only as good as the number it's judged on. Conversion rate alone misleads, because a change can lift conversion rate while lowering order value. A 10% off banner can push conversion rate from 2.5% to 2.8% and still lose money if average order value drops from $80 to $70.

The number you want is revenue per visitor for each group. Check a tool's reporting before you buy it. Shopify Rollouts is the clearest example of the gap: its theme experiments report conversion rate, bounce rate, reached checkout rate, and add to cart rate, with no revenue figure, and you can't add one [Shopify].

While you're in the reporting, check how the tool decides a winner. A dashboard that lets you stop whenever the variant looks ahead invites false winners, because checking repeatedly and stopping at the first good result inflates the error rate well past the stated confidence level [Johari et al., 2017]. Good tools either fix the sample size in advance or use statistics built for continuous monitoring.

Coming from Google Optimize

Google shut Optimize down on September 30, 2023 [Google], and 15 App Store reviewers mention moving to a Shopify app afterwards. shopifyabtesting.com lists where they went.

If you're coming from Optimize, expect to change how you build variants. Optimize edited pages in the browser. Many Shopify apps instead switch themes and templates, or change prices and shipping inside Shopify itself, so you plan tests around how your theme is built.

When a tool isn't the answer

A tool only helps if your store has the traffic to reach a result and someone with time to run it. At a 2% conversion rate, detecting a 10% lift takes roughly 80,000 sessions per group. Below about 50,000 sessions a month, most tests won't reach a clear answer, and fixing obvious problems directly is a better use of money. The sample size calculator runs the numbers for your store.

If you have the traffic but not the team, the DIY vs agency guide walks through that decision. Cascayd is a performance A/B testing service that picks, builds, and runs the tests for you, and only counts verified winners toward its fees. See how the Shopify CRO service works.