What sample size is needed for MVT?
Still have a question, spotted an error, or have a better explanation or a source we should cite?
You want to test three elements at once: subject line, hero image, and CTA button. Makes sense in theory. But before you hit send, there's a math problem sitting between you and reliable results.
MVT splits your audience across every combination you're testing, not just two variants. So the sample size question isn't "how many people do I need for a test?" It's "how many people do I need for each combination?"
Here's the basic rule: each combination needs enough recipients to reach statistical significance on its own. If a standard A/B test needs around 5,000 recipients per variant to be meaningful, then an 8-combination MVT needs roughly 40,000 total. That math scales fast.
The number of combinations comes from multiplying your variables. Two variables with two options each gives you 4 combinations. Three variables with two options each gives you 8. Add a fourth variable and you're at 16. This is why MVT gets complicated quickly, even when the concept feels simple.
Four things directly affect how large your sample needs to be:
- Number of combinations: More combinations mean more recipients needed. It's not additive, it multiplies.
- Baseline conversion rate: Lower baseline rates (say, 1% click rate vs 10%) need bigger samples to detect a real shift.
- Minimum detectable effect: If you want to catch a 2% improvement, you need a much larger sample than if you're trying to catch a 20% improvement.
- Confidence level: Wanting 99% confidence instead of 95% pushes your sample size up further.
Use a sample size calculator built for factorial designs, not a standard A/B testing calculator. The two aren't interchangeable. Standard calculators don't account for the interaction effects between variables that MVT is actually designed to measure.
Insufficient sample size is the most common MVT mistake (and probably the most frustrating one, because the test runs, you get data, and none of it is actually trustworthy). If your list size can't support the number of combinations you're testing, you're better off running sequential A/B tests instead. You'll learn more from two clean A/B tests than from one underpowered MVT.
Not sure if your list is big enough? Drop your numbers into the AI prompt below and get a rough answer in seconds.
Contributors
Who worked on this answer
Every name links to their profile. Every company links to their site. Real people, real accountability.
