DIY vs agency A/B testing for Shopify: how to choose

A/B test your Shopify store yourself if you have the traffic to reach a result and someone with real time to research, build in your theme, QA every variant, and read the statistics. Hire help when you have the traffic and not the people, and favor pricing tied to results, because most tests lose. Under roughly 50,000 sessions a month, neither route gets many clear answers, so fix obvious problems directly instead.

Check your sessions in Shopify first

Traffic decides whether testing works at all, whoever runs it. Open Analytics in your Shopify admin and look at monthly sessions, then the conversion rate breakdown report, which shows how many sessions add to cart, reach checkout, and buy [Shopify].

Those two numbers tell you what a test can detect. A store converting at 2% needs roughly 80,000 sessions per group to detect a 10% relative lift at 95% confidence and 80% power, and about 21,000 per group for a 20% lift. At 50,000 sessions a month, the bigger swing is readable in under a month and the smaller one takes more than three. The sample size calculator runs this for your own numbers.

A store with 15,000 sessions a month should not be paying anyone for A/B tests yet. Put that money into clear fixes instead, like turning off forced account creation at checkout or showing shipping costs on the product page, and into talking to customers. The CRO audit framework shows how to find them.

How DIY testing works on Shopify

If the traffic is there, doing it yourself usually means one of two setups. A testing app from the Shopify App Store adds its own script to your storefront, splits visitors, and reports results for a monthly fee. Or a developer builds the split into your theme code and sends the data to your analytics, which is the most flexible option and the easiest to get subtly wrong.

Shopify also has a built-in option called Rollouts, whose Experiment mode can split visitors between two versions of a theme on the Grow plan or higher [Shopify]. It reports conversion, bounce, reached checkout, and add to cart rates, with no revenue figure, and you can't add one [Shopify].

That gap matters whichever setup you use. A free shipping threshold that lowers conversion rate while raising order value looks like a loser on conversion rate alone. Make sure your setup reports orders and revenue for each group, so you can judge tests on revenue per visitor.

What one Shopify test actually takes

Whichever setup you pick, a single test touches five jobs:

1. Research. Reading Shopify's funnel reports, watching session recordings, and turning a drop-off into a hypothesis worth the traffic.
2. Design. Mocking up a variant of a product page, collection, or cart drawer that changes one meaningful thing.
3. Build. Editing a duplicate of your theme, or the Liquid behind it, without breaking apps that inject into the same sections.
4. QA. Checking the variant on phones, with discount codes, Shop Pay, and every currency you sell in.
5. Analysis. Reading the result correctly and deciding what ships.

Plenty of founders and marketers can do one or two of these well. The trouble starts when one person does all five on the side of a full-time job, because the shortcuts land where they hurt most.

The statistics are the classic trap. Checking a test every day and stopping when the variant pulls ahead makes false winners far more likely than the confidence level suggests [Johari et al., 2017]. A traffic split that drifts away from 50/50 usually invalidates the whole test [Fabijan et al., 2019], so check the split on every test.

Speed is the Shopify-specific trap. Shopify lists installed apps among the biggest factors in storefront speed, and uninstalling an app doesn't automatically remove its code from your theme [Shopify]. Every test app you try and abandon can leave weight behind. That weight costs money on every visit. At Amazon, a 100 millisecond slowdown cut sales by 1% [Kohavi et al., 2014]. Your theme's web performance report marks app installs and theme updates on its charts, so you can see when a change slowed the store down.

What doing it yourself costs

Testing apps cost a monthly fee. The people are the expensive part.

Building and running tests well takes developer and designer skills at minimum. In the US, the median web developer earned $92,650 in May 2025, and web and digital interface designers earned $104,000 [BLS]. Add analysis skills and you are looking at one senior hire or a small team, before a single test has run.

The cheaper-looking version is asking your existing team to fit tests around their real jobs. Tests then wait behind product launches and sales, fewer of them run, and the shortcuts above creep in.

DIY still makes sense in two situations. You have a team that already ships theme changes every week and someone comfortable with experiment statistics. Or your traffic is low, the work is mostly research and direct fixes, and a part-time effort covers it. The A/B testing best practices guide covers what that person needs to get right.

What hiring help gets you

Outside help brings three things most in-house teams lack: a process that has already run many tests, people who work in Shopify themes and testing setups every day, and throughput. More tests means more chances to find a winner. Booking.com runs some 25,000 tests a year [Thomke, 2020]. No Shopify brand needs that volume, but the principle scales down. A store that runs three tests a month learns three times faster than one that runs one.

The catch is incentives. When you pay someone to act on your behalf, their interests and yours never line up perfectly, and economists have a name for the cost of that gap: agency costs [Jensen & Meckling, 1976]. With a monthly retainer, the provider gets paid whether your revenue moves or not. Plenty of retainer firms do good work anyway. The contract still rewards activity, so watch the gap between a busy roadmap and a growing store.

Most tests lose, so pricing matters

At Microsoft, experiments on well-designed ideas found that about a third improved the metric they targeted, a third made no difference, and a third made it worse [Kohavi et al., 2009]. Plan for a similar split on your store.

That ratio changes the maths on how you pay:

Pricing model	What you pay for	Who carries the risk of losing tests
Hourly	Time spent	You
Monthly retainer	Access to a team	You
Pay for performance	Verified winning tests	The provider

On a retainer, two out of every three tests cost you money and return nothing directly. Under a performance model, those tests cost you nothing, and the provider only earns when a test produces a measured revenue lift. That pushes the provider toward the ideas most likely to move revenue, and toward measuring the result honestly, since a fake win is one they cannot defend.

Questions to ask before you hire anyone

• What metric decides a win? You want revenue per visitor, or a clear reason for something else. Conversion rate alone can reward a change that lowers revenue.
• How is a win verified? Ask for the confidence threshold, the minimum test length, and whether tests run in full weeks.
• Who builds the variants? If the answer involves your developers, count their time as a cost.
• What gets added to my theme? Ask which scripts or apps go on the store, and how they come off when the work ends.
• What happens to a losing test? It should be switched off quickly and cost you as little as possible.
• Do you guarantee a lift? Anyone who promises a specific uplift before seeing your data is selling certainty that A/B testing cannot give.

A quick way to decide

Your situation	Best route
Under about 50,000 monthly sessions	Fix clear problems directly and research customers. Hold off on paid testing.
Enough traffic, plus a team that ships theme changes weekly and knows the statistics	Run tests in-house, with a setup that reports revenue for each group.
Enough traffic, no spare team capacity	Hire help, and favor pricing tied to results.

Cascayd is a performance A/B testing service built for that third row. It suits Shopify stores with enough traffic for tests to reach a clear result, usually somewhere around $2M in annual sales or 50,000 monthly visitors. Cascayd finds and builds the tests with no dev work from your team, judges each one on revenue per visitor, and only counts verified winners toward fees. There is no setup fee or retainer, and no promise of a specific lift. See how the Shopify CRO service works, or if you're not sure where your store sits, start with a free Shopify CRO audit.