What is a p-value?

You ran an A/B test on your subject lines. Variant B got a higher open rate. Now your testing tool shows a p-value of 0.08 and you're wondering whether to ship it or start over. Sound familiar?

The p-value tells you how likely it is that your results happened by pure chance, assuming there was no real difference between the two variants. A p-value of 0.05 means there's a 5% chance the gap you're seeing is just random noise. A p-value of 0.01 means only a 1% chance. Lower is stronger evidence.

The most common threshold in email testing is p < 0.05. That's not a law of nature, it's just a widely agreed-on line in the sand. Cross it, and most people call the result statistically significant. Stay above it, and you don't have enough evidence yet to be confident the difference is real.

So what does a p-value of 0.08 actually mean for your decision? It means roughly an 8% chance you're looking at random variation, not a genuine win. That's not terrible, but it's also not great. Here's a practical framework:

• p < 0.05 and a meaningful difference in results: Ship the winner. You've got solid evidence and a real-world impact worth acting on.
• p between 0.05 and 0.10: Treat this as a signal, not a verdict. Run the test longer or on a bigger sample before committing.
• p > 0.10: The data doesn't support a clear winner yet. Reset, revisit your sample size, or rethink what you're testing.

One thing the p-value does not tell you is whether the difference actually matters for your business. A subject line that lifts open rates by 0.2% might be statistically significant with a huge enough list, but it's not going to change your quarter. That's the difference between statistical significance and practical significance, and it's worth keeping both in mind before you declare a winner.

Also worth knowing: p-values are sensitive to how long you run your test and your total sample size. A small list will almost never produce a p-value below 0.05 even if one variant is genuinely better. That's not a flaw in your test, it's just math telling you it needs more data. If you're unsure whether your sample size is big enough to trust your results, that's the first thing to check before you read too much into the p-value at all.