How can I A/B test personalization strategies?

You've decided personalization is worth testing. Great. But "I'll just A/B test it" is where a lot of senders stall out. What exactly do you test first? How big does your list need to be? How long do you run it before trusting the result?

Here's a practical framework to actually run these tests well.

Start with one variable at a time

This sounds obvious, but it's the rule most people break. If you test name personalization in the subject line AND a behavioral product recommendation in the body at the same time, you won't know which one moved the needle. Pick one thing. Test it. Then move to the next.

Good starting points, in rough order of complexity:

• Personalized vs. no personalization. The baseline. Does adding the recipient's first name in the subject line actually lift open rates for your audience?
• Personalization placement. Subject line vs. body vs. both. Where it appears changes how it lands.
• Personalization type. Name vs. location vs. past behavior vs. browsing history. These perform very differently depending on your audience and email type.
• Personalization depth. A first name vs. a full dynamic content block with product recommendations. More data-driven doesn't automatically mean better results.
• Fallback content. What shows when you don't have the data. If 20% of your list is missing first names, your fallback matters. Test it.

Sample size and duration

This is where most tests fall apart. Gut-checking after 200 sends is not a test. As a rule of thumb, you want at least 1,000 recipients per variant to start drawing conclusions from open and click data. For conversion metrics (purchases, sign-ups), you'll often need 2,000 to 5,000 per variant because those events are rarer.

Run your test for at least 48 to 72 hours before calling a winner. Send timing affects behavior, and a test that closes 4 hours after send only captures the most engaged readers, not your full list. Many ESPs like Mailchimp, Klaviyo, and Brevo let you set a hold-out window before auto-sending the winner. Use it.

Which metrics to track

Match the metric to what the personalization is trying to do.

• Subject line personalization. Track open rate (or click-to-open rate if your ESP adjusts for Apple Mail Privacy Protection).
• Body personalization. Track click rate and click-to-open rate, not just opens.
• Recommendation blocks or dynamic content. Track clicks on those specific elements and downstream conversions. Revenue per email sent is the cleanest signal if your ESP supports it.
• Fallback content. Compare the segment with missing data against the segment with full data. A good fallback should close that gap.

Statistical significance, practically speaking

So a 1% lift after 300 sends means almost nothing. You need enough data that the difference you're seeing isn't just noise. Most ESPs show a confidence percentage in their A/B results. Don't declare a winner below 90% confidence, and 95% is a safer bar for anything you plan to roll out permanently.

If you don't have a big enough list to hit those thresholds, run your test across multiple sends (same variant, same audience segment) and pool the results before deciding. It takes longer, but it's more honest.

One more thing worth knowing

Personalization results don't always transfer across segments. A name-in-subject-line test that lifts opens for your cold subscribers might do nothing for your loyal buyers who already know you well. Measuring the actual uplift from personalization across different segments is worth doing once you have a baseline result you trust.

Not sure which test to run first, or whether your list is large enough to get clean results? Our SOS hotline is free and we're happy to think through the setup with you.