A/B testing best practices for reliable decisions

A reliable A/B test has four parts: a hypothesis based on something you observed, one primary metric chosen before launch, a sample size and runtime planned in advance, and a decision made only once the test has finished. Skip any of them and a random swing in the numbers can end up shipped as a permanent change to your store.

Most tests will not produce a winner. A well-run test still tells you what to do next, even when the answer is "keep what you have."

Start with a question the test can answer

A good hypothesis names the change, the metric it should move, and the reason you expect it to. Here is one for a product page: "If we move delivery times and the returns policy up beside the add to cart button, more visitors will buy, because the page answers their last question before they scroll for it." You can test that. "Make the page better" you cannot.

A short template keeps hypotheses honest:

For [audience], changing [current experience] to [proposed experience] should affect [primary metric] because [evidence or user insight].

The evidence can come from funnel drop-off, support emails, customer interviews, or session recordings. Write down what you expect to learn even if the variant loses. For page-level ideas, the landing page optimization checklist is a good place to look for problems worth testing.

Decide what success means before launch

Pick one primary metric that represents the decision you need to make [Kohavi et al., 2020]. A product page test might use revenue per visitor. A cart test might use completed checkouts per cart view. Define the denominator, who counts, and the time window, so everyone calculates the result the same way. The revenue per visitor guide shows the formula, a worked example, and how to calculate it in Shopify and GA4.

Add guardrails for things that must not get worse, such as return rate, refund rate, page errors, or average order value. Supporting metrics can explain why a change behaved the way it did. They should never replace the primary metric once you see which one looks best.

Before any traffic enters the test, check the event definitions, both versions, how visitors are assigned, and conversion tracking. If a visitor can switch versions between visits, or one version fires an event differently, the result measures your setup instead of your shoppers.

Plan sample size and runtime up front

Four inputs decide how much data a test needs: your current baseline, the smallest lift worth acting on, your significance threshold, and statistical power. Set them before launch and run the numbers in a sample size calculator. If the required sample is out of reach for that page, pick a busier page, a bolder change, or a different way to learn. Running longer will not rescue a test that can never reach a useful sample.

A minimum number of days does not replace sample planning. The window should still cover the cycles that matter to your shoppers, such as weekdays versus weekends or a monthly campaign. Do not call a winner because the graph looks good on day three, and do not keep a test running just to wait for the answer you wanted. Checking a running test repeatedly and stopping at the first good-looking result makes false winners far more likely [Johari et al., 2017].

Testing several elements at once needs even more traffic, because a multivariate test splits visitors across every combination. Read when multivariate testing makes sense before you add variables.

Change one main thing and protect the comparison

Keep the two versions different in one meaningful way. If the headline, image, offer, and form all change together, a lift tells you something worked but not what.

Run overlapping tests only when you understand how their audiences and changes interact. Keep each visitor in the same version, check that traffic is split as planned, and log any other site or campaign changes during the test. If the split drifts away from 50/50, treat the result as suspect. A sample ratio mismatch usually invalidates a test outright [Fabijan et al., 2019].

Read the result, then make a decision

Look at the size of the effect and the uncertainty around it, along with the "winner" label. A small lift can be statistically significant and still not worth the cost of shipping. A big lift on too little data can still be noise. Compare what you got against the decision rule you wrote before launch, then check the guardrails.

If the evidence supports the change and no guardrail got worse, ship it and keep watching the metric after rollout. If the result is inconclusive, keep the current version or gather better evidence. Either way, record what you tested, on whom, how you measured it, what happened, and what comes next. Never promote a segment or secondary metric to the main result after you have seen the data.

Prioritize tests by evidence and impact

When the backlog is longer than your capacity, score ideas by likely impact, confidence in the evidence, and effort. A simple impact × confidence ÷ effort score works if the team agrees what each scale means. Treat the score as a way to sort the list. It will not predict results.

Start with problems at a busy step that have evidence behind them. Most ideas lose: across thousands of experiments, changes rarely produce a big positive impact on key metrics [Kohavi et al., 2014]. An unclear shipping policy or a mismatch between an ad and its landing page is a better use of traffic than a button color picked because it was easy to build. The CRO audit framework turns observations like these into test ideas, and the Shopify A/B testing guide covers what is practical to test on a Shopify store.

Get the tests run for you

Doing all of this well takes research, design, development, and patient analysis. Cascayd is a performance A/B testing service for Shopify stores. It finds the opportunities, builds and runs the tests, and reads the results, and you only pay if it makes you more than it costs. See how it works.