How to test or audit a validation vendor?

Auditing a validation vendor is the same job as auditing any vendor that grades your data. You build a known-answer test, run it through the vendor blind, then score the vendor against truth. If you skip this step you are trusting marketing copy with your sender reputation.

Here is the procedure we use at RME when we benchmark a vendor.

Build a seed list of 1,000 to 2,000 addresses

Mix the seeds in known proportions so you can score precision and recall by category. A useful split:

• 40% known-good addresses. Real mailboxes you own or control across Gmail, Outlook, Yahoo, and at least one Google Workspace and one Microsoft 365 tenant. These must come back as valid. Anything else is a false positive that will cost you real subscribers.
• 15% hard bounces you have confirmed. Pull addresses that returned an SMTP 550 5.1.1 "user unknown" from your last send. The vendor should flag every one of these. See RFC 5321 section 4.2.2 for the reply code definitions.
• 10% role accounts. info@, sales@, support@, postmaster@, abuse@. These are deliverable but high-risk. A good vendor labels them as role, not valid.
• 10% disposable domains. Pull current lists from mailinator.com, guerrillamail.com, tempmail variants. Disposable lists change weekly so use fresh ones.
• 10% catch-all domains. Domains you own where every address accepts mail. The vendor cannot prove a specific mailbox exists on a catch-all. Honest vendors say "accept-all" or "risky." Vendors that return "valid" on every catch-all address are guessing.
• 5% spamtraps you control. Set up addresses on a domain you own, never publish them anywhere, then drop them into the test list. A vendor that returns "valid" on an address that has never received legitimate signup traffic is not running deep checks. Do not use real third-party traps. That gets you blocklisted.
• 5% throttled or greylisting servers. Pick small business domains on hosts known to rate-limit verification probes (some Plesk and cPanel setups, certain regional ISPs). A vendor that calls these "invalid" because the SMTP probe timed out is going to nuke deliverable subscribers from your list. This is one of the "silent" hygiene issues that bleeds your list quietly.
• 5% typo domains. gmial.com, hotnail.com, yaoo.com. Good vendors flag and suggest corrections.

Run the test twice, a week apart

Upload the same list to the vendor on day 1 and day 7. Compare verdicts row by row. If more than 2% of addresses flip verdict between runs without you changing anything, the vendor's internal data is noisy. We have seen vendors flip 8 to 12% of verdicts week to week, which means their "valid" label is closer to a coin flip than a verdict.

Also send a 10% sample through a second vendor in parallel. Disagreement rate above 15% on the same addresses means at least one of them is wrong. Usually both are partly wrong.

Score against truth

Build a confusion matrix for each category. The pass thresholds we use:

• False positives on known-good: under 1%. Anything higher will shrink your sendable audience for no reason.
• True positives on confirmed bounces: above 95%.
• Catch-all handling: 100% labeled as accept-all or risky, never "valid."
• Disposable detection: above 90% on a list refreshed in the last 30 days.
• Run-to-run consistency: above 98% identical verdicts between day 1 and day 7.

Check the method, not just the result

Ask the vendor in writing whether they do live SMTP RCPT TO probes, MX record lookups only, or pattern matching against historical bounce data. SMTP probing is the only method that can confirm a specific mailbox exists at this moment, and even that is unreliable against catch-alls and Microsoft 365 tenants that accept everything at the connection layer. The Microsoft 365 behavior is documented in their admin docs on directory-based edge blocking. A vendor that claims 99% accuracy on M365 domains without explaining how is overselling.

Also ask whether they suppress trap-suspected addresses or just label them. "Label only" puts the risk on you.

What to do with the results

If the vendor passes, you have a new baseline. Re-run the audit every quarter, especially after they push platform updates. If they fail on catch-alls or run-to-run consistency, treat their "valid" verdict as a soft signal and combine it with engagement data before sending. If they fail on known-good, switch vendors. False positives are the most expensive failure mode because you never see the revenue you lost.

This is the same logic that sits behind why list hygiene matters for deliverability. A bad validator is worse than no validator because it makes you confident in dirty data.