What are common errors in A/B testing?

You ran an A/B test, picked the winner, rolled it out to your full list, and... nothing changed. Maybe things got worse. Sound familiar? That's usually a sign the test itself was broken, not your instincts.

A/B testing email is genuinely useful, but only when you do it right. Here are the mistakes that quietly kill your results.

Testing too many variables at once. If you change the subject line, the preview text, and the send time in the same test, you won't know which one moved the needle. One variable per test, full stop.

Stopping the test too early. You check the results after a few hours, version B is winning by a nice margin, so you call it. The problem is that early data is noisy. Stopping too early inflates false positives and you end up making permanent decisions based on temporary flukes.

Skipping statistical significance. A 3% open rate difference sounds real. It might just be random chance. Statistical significance tells you whether your result is likely real or likely noise. Acting on raw percentages without checking this is the most common way smart marketers fool themselves.

Running tests on too small a sample. Tiny sample sizes produce unreliable results every time. Before you run a test, figure out how many subscribers you need for a meaningful read. Most ESPs have sample size calculators built in, or you can find free ones online. (If your list is under a few thousand, some tests just aren't worth running.)

Ignoring what else was happening. A test that runs over a major holiday, a news event, or a day when your ESP had a sending delay is a compromised test. External factors skew your data in ways you can't control for after the fact.

Testing without a hypothesis. "Let's try a different subject line" is not a hypothesis. "Our subscribers open more when the subject line references a specific benefit rather than a generic teaser" is. Random testing produces random learnings.

Not documenting what you find. Your future self will run the same test again in six months and make the same mistake, unless you write down what you tested, what happened, and what you concluded. It doesn't need to be fancy. A shared spreadsheet works fine.

The common thread across all of these is patience. Good A/B testing is slow, methodical, and often underwhelming in the moment. The results compound over time, and that's where the real value is.

Now if you want a second pair of eyes on a test you're planning or trying to make sense of, our SOS hotline is free and we're happy to think through it with you.