What are common segmentation biases in testing?

Your A/B test on two segments shows the winning variant had a 22% higher click rate. That's significant, right? Maybe. But if one segment skewed toward your most recent subscribers and the other toward long-tenured ones, you weren't testing your email, you were testing your subscriber base.

The most common problem is selection bias: the people in your segments aren't random samples, they're self-selected by behavior or acquisition source. If your "high engagement" segment naturally contains early adopters who click anything, and your "low engagement" segment contains people who signed up from a cold lead magnet, any test between them measures the audience, not the message. Good segmentation tests compare like to like by splitting a single segment randomly rather than testing across segments with different audience profiles.

Recency bias sneaks in when you test at different points in time. If you send version A on Tuesday morning and version B on Friday afternoon, you've introduced timing as a variable. Run your tests concurrently, or at minimum during similar time windows. The same logic applies to survivor bias: a segment that's been active for a year has already shed unengaged contacts. It's not a representative sample of your full list, it's the people who made it through your previous sends.

Sample size is where many segmentation tests fail quietly. Small segments produce noisy results. A 30% click rate from 100 people doesn't mean what a 30% click rate from 5,000 people means. Before treating any result as a signal, make sure your segment is large enough to detect a real difference. Most segmentation testing frameworks include a sample size calculator to use before you start.

The fix isn't to stop segmenting, it's to test within segments rather than across them. Take your "active subscribers" segment, split it randomly 50/50, and test your two variants within that group. You've controlled for audience characteristics and you can trust the result. Once you've got a winner, you can roll it out to similar segments with reasonable confidence. Click rate is your primary signal here, since opens are inflated by Apple Mail Privacy Protection since iOS 15 and don't give you clean data.