Should I test personalized vs. non-personalized versions?

And Here's a question that surprises a lot of senders: what if your personalized email actually performs worse than the plain version? It happens more often than you'd think. Personalization is a hypothesis, not a guaranteed win. The only way to know if it's working for your audience is to test it directly.

But The good news is that a personalized vs. non-personalized test is one of the cleaner A/B tests you can run. You have a clear control (no personalization) and a clear variant (personalization applied). You're not guessing about the baseline.

What to measure

Don't stop at open rate. Open rate tells you if personalization in the subject line caught attention, but it doesn't tell you if it drove value. Track these in order of importance for your goals:

• Open rate for subject line personalization (first name, location, etc.)
• Click rate for body personalization (product recommendations, dynamic content)
• Conversion rate for any personalization tied to a purchase or action
• Revenue per email if you're in e-commerce (this is the number that actually justifies the effort)
• Unsubscribe rate because if personalization feels creepy or off, you'll see it here

Sample size matters more than you think

A lot of senders run a test on 200 people, see a 2% difference, and call it a win. That's not a result. That's noise. You need enough recipients in each group for the result to be statistically significant, meaning the difference you see is real and not just random variation.

A rough rule of thumb: aim for at least 1,000 recipients per variant for open rate tests, and more for click or conversion tests where fewer people take action. Most ESPs like Mailchimp, Klaviyo, or Brevo have built-in A/B testing that calculates significance for you. Use it. Don't eyeball percentages.

Test one variable at a time

Now if you're testing first-name personalization in the subject line, keep the email body identical in both versions. If you change the subject line AND add a product recommendation block, you won't know which change drove the result. Isolating your variables is the whole point of a controlled test.

Segment your results before you conclude anything

Personalization often performs differently across segments. A first-name subject line might lift open rates for new subscribers but do nothing for your long-time loyalists who already recognize your brand. Run the same test across different segments before assuming the result applies to everyone on your list.

Also worth testing separately: name personalization, behavioral personalization (based on past purchases or clicks), and location-based personalization. One type might help while another actively hurts. You won't know unless you test them independently. This connects directly to measuring the actual uplift each type delivers.

What bad personalization looks like in results

So if If your personalized version shows a higher unsubscribe rate, a lower click rate despite a higher open rate, or a flat conversion rate, that's your audience telling you something. Either the data quality is off (wrong names, stale behavior signals) or the personalization feels intrusive rather than helpful. Both are fixable, but only if you're measuring. Not sure your list data is clean enough to personalize reliably? That's worth sorting out before you run the test. Stale or incomplete data will skew your results and make good personalization look like it's failing.