How to test bounce handling accuracy?

You set up bounce handling, your ESP says it's working, and you trust it. But how do you actually know it's classifying bounces correctly? A misconfigured bounce processor quietly suppresses valid addresses, lets bad ones through, or mixes up a temporary deferral with a permanent failure. Left unchecked, those errors stack up fast.

Here's a practical walkthrough for testing your bounce handling end to end.

Step 1: Send to known test addresses

The cleanest way to start is by sending to addresses with predictable outcomes. Mailtrap and similar sandbox tools let you trigger specific SMTP responses on demand. You want to cover at least these five scenarios:

• A non-existent address (expect a hard bounce, typically a 550 5.1.1 "user not found" response)
• A full mailbox (expect a soft bounce, typically a 452 or 4.2.2 code)
• A domain with no MX record (expect a DNS failure bounce)
• A reputation-based rejection (expect a block bounce, usually a 5.7.x code)
• An auto-responder address (this should NOT trigger a suppression)

For each test, note the raw SMTP response code your ESP receives. That's the ground truth you'll compare against in the next step.

Step 2: Check how each bounce was classified

Pull up your ESP's bounce logs and look at how it labeled each response. Did the 550 come back as a hard bounce? Did the 452 land as a soft bounce or did it get flagged as a hard one by mistake? Did the auto-responder trigger a suppression when it shouldn't have?

Now the difference between simulated and real bounce testing matters here. Some ESPs classify sandbox bounces differently than production ones, so you may need to run a second pass with real addresses you control (a domain you own where you can delete a mailbox, for example).

Step 3: Verify suppression behavior

After your test sends, check your ESP's suppression list. Hard bounce addresses should appear there. Soft bounce addresses generally shouldn't be permanently suppressed after a single failure. And the auto-responder address? It should be completely absent from the suppression list.

And this is where a lot of systems quietly break. If your ESP is suppressing soft bounces on first hit, you're almost certainly losing valid subscribers (of course, that's the kind of thing you'd only notice by testing it).

Step 4: Measure your classification accuracy

Once you have results, calculate three numbers:

• Correct classification rate: What percentage of test bounces were labeled the right type?
• False positive rate: How many valid addresses got suppressed when they shouldn't have?
• False negative rate: How many invalid addresses were not suppressed when they should have been?

There's no universal benchmark, but if your false positive rate is above 1-2%, that's worth investigating. Valid addresses being dropped costs you real reach.

Step 5: Run sample audits on live traffic

One-off testing only tells you your system worked on that day. Build a habit of pulling a random sample of recent bounces (say, 50-100 per month) and reviewing the raw SMTP codes manually against the classifications your ESP assigned. You're looking for unclassified or miscategorized bounces that slipped through.

Edge cases show up in production that don't appear in sandboxes. A response from a legacy corporate mail server might return a non-standard code your ESP has never seen before and quietly drops into the wrong bucket.

If you're seeing high bounce rates that don't match what you'd expect from your list, it's worth running a full list validation before assuming it's a classification problem. We clean lists at RME if you want a second opinion on what's actually in there ;)