How do I run a statistically valid A/B test?

You've picked your variable, you've got a hypothesis, and you're ready to test. But here's where most email A/B tests quietly fall apart: the math gets skipped. Without the right sample size and a clear finish line, you're not running a test. You're flipping a coin with extra steps.

Here's how to do it properly.

Start with a real hypothesis

Don't just say "I want to test subject lines." Say what you expect to happen and why. Something like: "Adding the recipient's first name to the subject line will increase open rates by at least 5% because it signals relevance." That number matters. It's your minimum detectable effect (MDE), the smallest improvement worth acting on. If you'd only change your approach for a 1% lift, set it at 1%. If a 10% lift is the threshold that would change your strategy, use 10%. Be honest with yourself here.

Calculate your sample size before you start

This is the step everyone skips. You need to know how many subscribers each group (A and B) needs before a result is trustworthy. The inputs are three things:

• Your baseline rate. What's your current open rate or click rate? Say it's 22%.
• Your MDE. Let's say you want to detect a 3 percentage point improvement (so 22% vs. 25%).
• Your confidence level. Most marketers use 95%, meaning there's only a 5% chance the result happened by random chance.

Plug those into a free sample size calculator (Evan Miller's is the most widely used). For the example above, you'd typically need around 2,500 subscribers per group, so 5,000 total just to detect a 3-point lift with 95% confidence. That surprises a lot of people. If your list is smaller, you either need to detect a bigger effect or accept a lower confidence level.

Test one variable only

If you change the subject line and the send time in the same test, you won't know which one moved the needle. One variable per test. That's it. (Yes, multivariate testing exists, but it requires much larger sample sizes and is a separate conversation.)

Split randomly and let it run

Your ESP should handle random assignment automatically. Don't hand-pick who goes into which group. And once the test starts, don't peek and stop early because one version looks like it's winning. Early stopping is one of the most common ways valid tests produce false results. Set your end point in advance and stick to it.

Read the significance, not just the percentages

And a 23% open rate vs. a 22.1% open rate might look like a win. It might be random noise. What you need is a p-value below 0.05 (for 95% confidence) or a confidence interval that doesn't cross zero. Most ESPs report a "winner" without surfacing this properly, so it's worth running your numbers through a significance calculator after the fact to confirm what you actually found.

One more thing worth knowing: a test that finds no significant difference is still useful. It tells you the effect either doesn't exist or is smaller than your MDE. That's real information. Don't treat a "no winner" result as a failed test.

If you're not sure where to start, take a look at what's actually worth testing before you run the numbers. Or if your tests keep showing no winners and you suspect it's a list quality issue, that's a different problem worth looking at separately.