How do I measure segment performance against a control group?

You've built a segment you're proud of. High-value customers, lapsed buyers, a specific geography. But how do you actually know if targeting them made a difference? That's where a control group earns its keep.

Here's how to set one up and read the results honestly.

Step 1: Split before you send

Before you send anything, randomly pull a portion of your full list and hold them back. This is your control group. They don't get the targeted send. They might get nothing, or they might get your standard broadcast. Whichever fits your test setup. The key word is random. If you cherry-pick who goes into the control, you've broken the test before it started.

A clean 80/20 split (80% targeted segment, 20% control) works well for most list sizes. If your segment is small (under 1,000), lean toward 70/30 so the control group stays statistically meaningful.

Step 2: Send the same content to both groups

This is the part people skip, and it ruins the whole experiment. If your segment gets a different subject line or a different offer, you're no longer testing segmentation. You're testing content. Send identical emails to both groups. The only variable should be who received it.

Step 3: Calculate lift

Once results are in, the formula you want is simple:

Lift = (Segment result − Control result) ÷ Control result × 100

Say your targeted segment converted at 4% and your control group at 2.5%. That's a 60% lift. That number tells you segmentation added real value. Not just that your email performed well overall.

Track the same metrics for both groups: open rate, click rate, conversion rate, and revenue per recipient if you can. One metric alone can be misleading (opens lie more than clicks do, especially post-Apple MPP).

Step 4: Check if the difference is real

Still a 60% lift sounds great. But if each group only had 50 people, that difference could be noise. Sample size matters a lot here. As a rough guide, aim for at least 500 contacts per group before drawing conclusions. The more similar your groups are in size, the more trustworthy the comparison.

Most ESPs won't calculate statistical significance for you automatically, but you can run a quick chi-square test on your conversion numbers using any free online calculator. If your p-value is above 0.05, the result isn't reliable enough to act on yet.

How to set this up in your ESP

The process varies by platform, but the pattern is the same across most tools:

• Mailchimp: Use the A/B test or Multivariate campaign builder. You can designate a holdout segment manually by tagging a random subset and excluding them from the send.
• Klaviyo: Has a native holdout group feature inside flows. For campaigns, create a control segment using random sampling via the segment builder.
• Brevo: Use A/B testing with a percentage split, or manually create a suppressed control list before sending.
• ActiveCampaign: Build a control segment using list-level conditions and exclude it from the automation or campaign.

So if your ESP doesn't support holdout groups natively, the manual approach works fine. Tag a random sample as "control," suppress them from this send, and compare their baseline behavior afterward.

What to do with the results

If your segment significantly outperforms the control, your targeting logic is working. Document it and build on it. If the lift is tiny or reversed (it happens), that's genuinely useful too. It means your segmentation criteria might need rethinking. Or the segment genuinely doesn't behave differently from the rest of your list.

Either way, you've got real data instead of a guess. That's the whole point of a control group.

If you're still figuring out how to build the segments themselves before you test them, check out the question on testing segment effectiveness first. And if you want a second set of eyes on your results or your setup, we're happy to take a look.