How to A/B test deliverability by segment?
Still have a question, spotted an error, or have a better explanation or a source we should cite?
Testing deliverability is different from testing conversion. You're not just measuring clicks and opens. You're measuring whether your email actually arrived in the inbox vs. spam vs. not at all. That requires a different approach.
Step 1: Set up seed testing before you start. A seed test sends your email to a panel of known addresses across major mailbox providers, then reports where it landed (inbox, spam, or not delivered). Tools like GlockApps or Litmus can do this. Without seed testing, you're measuring opens as a proxy for inbox placement, which isn't the same thing. Open rates don't tell you if Gmail filtered you to spam.
Step 2: Split your segment randomly, not by engagement level. If you split by engagement (variant A gets highly engaged subscribers, variant B gets everyone), you're testing audience quality, not your variable. Random splits ensure both groups are comparable. The key is changing only one thing between variants: subject line, content length, image ratio, or send time.
Step 3: Run the same test across multiple segments. A subject line that performs well with your most engaged subscribers might fail with your re-engagement group. Testing within a single segment tells you what works for that segment. Testing across multiple segments tells you what's generally reliable.
One honest caveat: measuring inbox placement requires larger sample sizes than measuring conversion. Placement signals are noisier (mailbox providers are inconsistent), and the differences between variants are often small. You typically need thousands of sends per variant to get conclusive results. If your segments are small, treat test results as directional rather than definitive.
What to measure: inbox placement rate from seed testing, open rate (with the caveat that Apple MPP inflates opens), click rate, complaint rate (through feedback loops if you have them), and unsubscribe rate. Track these per variant and per segment. Differences in complaint rate especially are signal worth acting on.
If a specific content pattern consistently produces better inbox placement across multiple segments and multiple tests, that's a real finding. One-off results in a single test aren't.
Contributors
Who worked on this answer
Every name links to their profile. Every company links to their site. Real people, real accountability.
