What is statistical significance in A/B testing?
Still have a question, spotted an error, or have a better explanation or a source we should cite?
Statistical significance measures how confident you can be that observed differences between test variants reflect real effects rather than random variation.
When a result is statistically significant, it means the probability of seeing such a difference by chance alone is below your threshold, typically 5% (corresponding to 95% confidence).
Significance does not mean the difference is large or important. A tiny improvement can be statistically significant with large enough samples. Conversely, large apparent differences may not be significant with small samples.
Achieving significance requires adequate sample size, sufficient test duration, and a real underlying difference to detect.
Statistical significance answers "is this difference real?" not "is this difference meaningful?" You need both questions answered before acting on results.
Contributors
Who worked on this answer
Every name links to their profile. Every company links to their site. Real people, real accountability.
