How to interpret statistical significance in email tests?
Still have a question, spotted an error, or have a better explanation or a source we should cite?
You're running an A/B test on your email subject lines. Version A gets 22% open rate. Version B gets 21.5%. Which one won? Your gut says A, but your gut is a terrible statistician. That small difference might be pure luck. That's where statistical significance comes in. It tells you whether the difference is real or just random noise.
And Here's the basic idea. You set a confidence level, usually 95%. That means you need enough evidence that there's only a 5% chance the difference you're seeing happened by accident. Without that bar, you're basically guessing. You might roll out a "winner" that isn't actually better. Then you waste budget on something that doesn't work.
Three factors affect whether you'll reach significance. First, sample size. More data means more confidence. Second, how big the difference is. If A is 25% and B is 18%, that's a 7-point gap. Significance comes faster. A 0.5-point difference needs way more volume. Third, your baseline conversion rate. Improving something that already converts well is harder to prove than improving something that barely works. A test moving from 2% open rate to 2.1% needs a massive sample. Moving from 5% to 6% is easier to confirm.
And don\'t eyeball results. Use a significance calculator. Plug in your sample sizes and conversion numbers. The calculator returns your confidence level. If it says 98%, you're good. If it says 40%, you need more data. Keep the test running. Never stop early just because one variant looks ahead. Early stopping is how you fool yourself with false winners.
Related: pre-send testing, dark mode testing.
Contributors
Who worked on this answer
Every name links to their profile. Every company links to their site. Real people, real accountability.
