How long should I run an A/B test?

You launch a test, check the results after a day, and one subject line is clearly winning. Do you call it? Most people want to. Most people should wait.

How long you run an A/B test depends on two things working together: reaching your required sample size and covering enough calendar time to catch day-of-week behavior differences. Get both right, and your results actually mean something.

Start with sample size, not a time limit

So before you send a single email, figure out how many recipients each variant needs. The rough formula pulls from three inputs: your baseline open (or click) rate, the minimum lift you actually care about detecting, and your target confidence level.

Let's say your baseline open rate is 25% and you want to detect a 5-percentage-point improvement (so 30% vs 25%). At 95% confidence, you'd need roughly 1,500 to 2,000 recipients per variant. Smaller lift you want to detect means bigger sample size required. That's just statistics doing its thing.

Most ESPs have built-in calculators, or you can use a free online sample size tool. Plug in your numbers before you start, not after you peek at results.

Convert sample size into calendar days

Now once you know the sample size per variant, divide by your typical daily send volume to get a rough duration. If you need 2,000 per variant and you send to 1,000 people a day total, you're looking at at least four days per variant (or you split a single send and wait for responses to accumulate).

And Here's a quick example to make it concrete:

• Baseline open rate: 25%
• Lift you want to detect: 5 percentage points
• Confidence level: 95%
• Required per variant: ~1,800 recipients
• Your daily send volume: 900 emails
• Estimated test duration: about 4 days

That's your floor, not your target.

Always run at least a full week

Even if your sample size fills up in two days, don't stop there. Open and click behavior shifts across the week. Tuesday subscribers behave differently from Saturday ones. A test that only captures Wednesday and Thursday tells you about Wednesday and Thursday, not your audience.

Now one full week is the practical minimum for most campaigns. Two weeks is safer if you send to a smaller list and your daily volume is low. If your test window spans a major holiday or a product launch, note that externally and treat the results with some skepticism (because something else drove the behavior, not just your subject line).

What to do if significance never arrives

Sometimes you run a test for two weeks and the results are still too close to call. That's actually a valid finding. It means the two variants perform roughly the same for your audience, so pick either one and move on. You haven't failed. You've learned there's no meaningful difference, and that's worth knowing.

What you shouldn't do is keep extending the test indefinitely hoping the numbers will separate. If they haven't after a reasonable window, they probably won't. Call it inconclusive and redirect that energy into testing a bigger change next time.

Not sure how to read your results once the test wraps? Check out what statistical significance actually means before you decide what to do with the data.