How to test content vs engagement impact separately?

You've probably wondered at some point whether your emails are landing in spam because of what you're writing or because of who you're sending to. These are two different problems, and mixing them up leads to fixes that don't fix anything.

Here's how to separate them cleanly.

First, define your engagement tiers

Before you run any test, you need two clearly defined groups. "Engaged" means people who have opened or clicked at least once in the last 90 days. "Disengaged" means people who haven't opened or clicked in 90 to 180 days (or longer). You're not guessing here. Pull these segments from your ESP's reporting and lock them in before you touch anything else.

If you're in Mailchimp, Klaviyo, or most other platforms, this is a standard segmentation filter. If your platform doesn't track engagement history at the subscriber level, that's a separate problem worth solving before you try to test anything.

Test 1: Does content matter? (Fix the audience, vary the content)

Take your engaged segment and split it in half. Both halves get the same send time and the same sender name. Half A gets subject line version 1. Half B gets subject line version 2 (or a different format, different copy length, whatever you're testing).

Any inbox placement difference you see between A and B is coming from the content, not the audience. Because both groups have the same engagement history, you've controlled for that variable.

What to measure: inbox vs. spam rate using seed addresses placed in both groups, plus real open rates from each segment. If the open rate gap is large but inbox placement is similar, you're looking at a subject line issue, not a deliverability one.

Test 2: Does engagement history matter? (Fix the content, vary the audience)

Send the exact same email, word for word, to your engaged segment and your disengaged segment on the same day. Don't change a thing between the two sends. Same subject line, same body, same sender.

Now compare inbox placement. If the engaged group lands in the inbox and the disengaged group hits spam, you've confirmed that placement differences are driven by recipient history, not your content. That tells you the fix is list hygiene and re-engagement, not rewriting your emails.

What about seed addresses?

Seeds are useful but limited here. A seed inbox has no engagement history at all. So seeds can tell you whether a given mailbox provider is routing your content to spam, but they can't tell you how real disengaged subscribers are being treated. You need actual subscriber segments for that half of the test.

Run seeds inside both groups to check per-provider placement, but don't rely on them alone to answer the engagement question.

A note on sample size

This works best when each group has at least 1,000 real subscribers. Smaller samples make it hard to tell whether a difference in placement is real or just noise. If your list is smaller, run the test over multiple sends before drawing conclusions.

If you're not sure your list is clean enough to segment reliably, it's worth validating it first. A list full of inactive or invalid addresses will skew both groups and muddy your results. You can run it through Review My Emails to sort the actives from the deadweight before you start.