What confidence level should I aim for (e.g., 95%)?

Most email A/B testing guides tell you to aim for 95% confidence and leave it at that. But that answer only means something once you understand what you're actually trading off when you pick a number.

Confidence level is the threshold you set before calling a winner. At 95%, you're saying there's only a 5% chance that the difference you saw was a fluke. At 90%, that risk goes up to 10%. At 99%, it drops to 1%. The higher you go, the more certain you are, but the longer your test needs to run and the bigger your list needs to be.

95% is the right default for most tests. It's the standard for a reason. It keeps false positive risk low enough to act on, without requiring an enormous sample or weeks of waiting. For subject line tests, CTA copy, preview text, or sender name tweaks, 95% is where you should start.

90% is fine for low-stakes, directional tests. You're not making an irreversible decision. You just want a nudge on which way to go with a small list or a fast-moving campaign. Know that you're accepting more risk of a false positive, and treat the result as a hint, not a verdict.

99% is worth the extra wait for high-stakes decisions. Think completely new templates, pricing language, a major change to how you frame your value proposition, or anything you'd roll out to your entire list permanently. The cost of being wrong is high enough to justify the bigger sample size and longer runtime.

Speaking of which: confidence level alone doesn't tell you how many subscribers you need or how long to run the test. Those depend on your list size, your expected open or click rate, and the minimum difference you actually care about detecting. A test comparing two subject lines where one performs 2% better than the other needs far more data to call reliably than one where the difference is 15%.

A rough rule of thumb: at 95% confidence with a typical open rate around 20-30%, you generally want at least 1,000 recipients per variant before trusting the results. Smaller than that and you're reading noise. Check the tools that calculate statistical significance automatically, most good ESPs include this, or you can use a free calculator and plug in your numbers before you send.

So one more thing worth knowing: confidence level and test duration are connected. If your list is smaller, you might need to run the test for longer to collect enough data to reach your target confidence. That's why test duration is its own question worth thinking through before you start.

If you're unsure how to size your test or interpret the results, we're happy to help. Drop a question on the SOS hotline and we'll walk you through it.