What’s the right way to A/B test segmentation?
Still have a question, spotted an error, or have a better explanation or a source we should cite?
A/B testing segmentation is different from A/B testing a subject line. You're not changing what the email says. You're changing who gets it, or what rules determine who gets it, and measuring whether the new logic produces better outcomes than the old one.
The structure: Create a control group that receives your current segmentation strategy, and a test group that receives the new approach. Keep everything else constant: same content, same send time, same cadence. The only variable you're testing is the segmentation logic itself.
Random splits are critical. If you split non-randomly (say, putting your most engaged subscribers in the test group), you're testing audience quality rather than segmentation effectiveness. Use a list-level random split before applying any segment criteria. Your ESP should support this. If it doesn't, you can split by even/odd subscriber ID or another pseudo-random method.
What to measure: engagement metrics (click rate is more reliable than open rate since Apple MPP inflates opens), deliverability signals (complaint rates, bounce rates), and downstream business outcomes (revenue, conversions, retention). Segmentation effects often show up in deliverability before they show up in engagement, because better segments produce better reputation signals.
Run the test long enough. Segmentation effects compound over time. A single campaign might not show a meaningful difference, but after 6-8 sends the patterns become clearer. Four to six weeks is usually the minimum. If your sending frequency is weekly, that's 4-6 data points per group. If your frequency is monthly, you may need to run it for a quarter.
One honest reality: segment testing requires large enough groups to reach statistical significance. If your total list is 5,000 subscribers and you're splitting them across two test conditions, each group of 2,500 might not produce statistically reliable differences in click rates. The smaller the segment, the noisier the data. Set expectations accordingly.
Document what you learned and use it. The best segmentation tests produce a finding you can apply to your next segmentation decision. For the metrics that tell you whether a segment is healthy enough to be worth testing, the segment health metrics guide is a good prerequisite read.
Contributors
Who worked on this answer
Every name links to their profile. Every company links to their site. Real people, real accountability.
