How long should you run an A/B test? The honest arithmetic, and what to do when you don’t have the traffic

The short answer

An A/B test should run until it reaches the sample size you worked out before it started, rounded up to whole weeks, and for at least one full week. You read the result once, at the end. Not when it first looks good, and not “two weeks” because that’s what everyone says.

How long that takes depends on three things: how often your visitors buy now, how big a difference you want to be able to see, and how many visitors you have. The rest of this article shows the sums, then what to do when they come out longer than you’d like.

Step 1: work out the sample size

The sample size is the number of visitors each version needs before the result can be trusted. There’s a well-known rule of thumb for the number of visitors each version needs, at the usual 5% significance level and 80% power (Zhou et al. call it “a widely adopted rule of thumb”; Evan Miller popularized it):

visitors per version ≈ 16 × p × (1 − p) ÷ δ²

Here p is your current conversion rate and δ is the difference you want to detect, in absolute terms. The smaller the difference, the more visitors you need, and it grows fast: halve δ and you need four times the visitors.

Here’s the same store with different lifts in mind, using the full formula:

The rule of thumb is fine for a first guess. The table uses the full formula, the same one in our free A/B test calculator, so your own numbers there will match. The terms are in the glossary: minimum detectable effect, statistical power, sample size.

What about “1,000 visitors per version”?

It’s a common rule, and it’s far too small for most stores. At a 2% conversion rate, 1,000 visitors per version is about 20 orders each. With that, the smallest difference you can reliably detect is more than +100%: the new version would need to more than double your sales before the test could tell. Few changes come anywhere near that. Run 1,000-visitor tests and you’ll mostly be reading noise.

Step 2: run whole weeks

People shop differently on a Monday morning than on a Sunday evening. A test that runs Tuesday to Friday sees only one kind of shopper. That’s why Optimizely advises running tests for a minimum of one business cycle (seven days). Adobe’s documentation rounds test lengths up to the nearest whole week. And Microsoft’s experimentation team, which runs thousands of tests, notes that “the typical duration of an A/B test is 7 days” and warns about day-of-week effects.

Whole weeks also help with the novelty effect: regular visitors sometimes click on something just because it’s new, and the effect fades. Microsoft’s team describes a “rapid decline of estimated treatment effect across date segments” as a sign of it. If your new version wins big in week one and less each week after, believe the later weeks. More in the glossary.

Step 3: don’t stop early

Peeking is checking a running test and stopping it as soon as the result looks good. It’s where most store-owner tests go wrong. You check the test every morning. On day 6 the new version is ahead and the tool says “significant”. You stop and ship it.

The problem: the 5% false-positive rate behind “significant” only holds if you look once, at the planned end. Every extra look is another chance for random noise to cross the line. In one of Evan Miller’s examples, checking repeatedly and stopping at the first significant result pushed the false-positive rate to 26.1%, instead of the 5% you think you have. The more often you check, the higher it goes. Spotify’s engineers found that even a single extra look makes a 5% test behave closer to 10%. Researchers at Stanford and Optimizely put it bluntly: standard results are “wholly unreliable” if you decide when to stop by watching them.

There are legitimate ways to look early, but they need statistics designed for it (sequential tests, which some tools use). With an ordinary test, the rule is simple: set the length, then leave it alone. The glossary calls this peeking.

Step 4: check the split before you read the result

Before you look at which version won, check that each one got its share of visitors. If a 50/50 test shows 10,000 visitors on one side and 9,400 on the other, that gap is far too large to be chance (the odds are well under 1 in 10,000). Something is broken: a redirect, a script that fails on some phones, bots counted on one side.

Microsoft’s team is clear that such a sample ratio mismatch “typically invalidates the A/B test”. Don’t try to rescue the result. Find the cause, fix it, and run the test again.

When you don’t have the traffic

Most small stores will look at the table above and wince. Thirteen weeks for one test is a long time. You have five honest options:

1. Test bigger changes. A redesigned product page can plausibly move conversion by 30%; a new button colour can’t. In the table, going from +20% to +30% cuts the sample size by more than half.

2. Measure a step closer to the change. Far more visitors add to cart than buy, so add-to-cart reaches a reliable answer much sooner. At an 8% add-to-cart rate, spotting a 20% lift takes about 4,900 visitors per version, against 21,100 for orders at 2%. At 500 visitors a day that’s about 3 weeks instead of 13. Optimizely’s advice for low-traffic sites is the same. One catch: more add-to-carts don’t always mean more orders, so keep an eye on orders too.

3. Test one change against the original. Every extra version needs its own share of visitors. In one worked example by Seer Interactive, a test needed 61,000 visitors with two versions, 91,000 with three and 122,000 with four. More versions also mean more chances of a false winner: Adobe calculates an 18.5% chance of at least one false positive with five versions.

4. Fix the length, then see what you can detect. Turn the sum around. A store with 500 visitors a day can give each version about 7,000 visitors in four weeks. At a 2% conversion rate, that’s enough to detect a lift of roughly +36%, and no smaller. If the change you have in mind won’t plausibly do that, testing it for four weeks will only tell you “no clear difference”.

5. Don’t test. A broken checkout button, a missing delivery cost or a page that’s unreadable on a phone doesn’t need a test; it needs fixing. And when you don’t know what to change, research works on small traffic where tests can’t: ask the shoppers who leave, or watch recordings of real visits.

How long is too long?

Long tests have their own problems. People clear their cookies and switch between phone and laptop, so over months the same person can end up seeing both versions. Seasons change, and a test that straddles Black Friday measures Black Friday. Shopify’s own Experiments end after 90 days by default.

A practical limit: if your sums say a test needs more than about two months, don’t run it. Change the plan instead, using the options above.

Before you press start

• Write down your current conversion rate, the smallest lift worth finding, the visitors per version and the length in whole weeks.
• Pick one main metric (orders per visitor, or add-to-cart on a small store) and one guardrail, such as revenue per visitor, so a “win” that loses money gets caught.
• Read the result once, at the end.
• Check the split first. If it’s off, the result doesn’t count.

If the arithmetic says most of your ideas can’t be tested, that’s useful too: it tells you to spend your time on the changes big enough to show. This guide to raising your conversion rate shows how to find them.