How to test a vendor’s accuracy?

You've narrowed down a shortlist of email validation providers. They all claim 98%+ accuracy. So how do you actually figure out which one is telling the truth?

The answer is a benchmark file. You build a small, controlled list of addresses where you already know the right answer, run it through each vendor, and see who gets it right. It's not complicated, but the details matter.

Step 1: Build your benchmark file

Your benchmark file needs to include every address category a validator should handle. Aim for 50 to 200 addresses total, with representation across these types:

• Known valid addresses. Real inboxes you control or know are active. Try addresses like captain@deepcurrent.io or a test account you've created at Gmail or Google Workspace.
• Known invalid addresses. Addresses at real domains that don't exist. Think noreply-zzz99@gmail.com or a username you know was never created.
• Role addresses. Info@, support@, admin@, sales@. These should flag as role-based, not personal.
• Disposable addresses. Addresses from throwaway services like Mailnull, Trashmail, or Guerrilla Mail. A good vendor keeps an updated database of these domains.
• Catch-all addresses. Domains configured to accept everything. These are notoriously hard to classify correctly and reveal a lot about a vendor's depth.
• Syntax errors. Obvious malformed addresses (missing @, double dots, invalid TLD). Every vendor should catch these. If one misses them, stop right there.

Step 2: Run the same file through multiple vendors simultaneously

Don't run vendor A today and vendor B next week. Validation results can shift over time as domains change. You want a snapshot from the same moment so the comparison is fair. Pick two or three vendors from your shortlist and upload the file to all of them within the same hour if possible.

Step 3: Build a comparison matrix

Export each vendor's results and line them up side by side. For each address in your file, note:

• What classification did each vendor return?
• Does that classification match what you know to be true?
• Where do the vendors disagree with each other?

Disagreements on catch-all addresses are expected. Disagreements on clear-cut valid or invalid addresses are a red flag.

Step 4: Watch out for inflated valid counts

Some vendors game their accuracy numbers by defaulting unknown addresses to "valid" rather than "unknown" or "catch-all". This makes their valid rate look high, but it means you're keeping risky addresses in your list. If a vendor returns almost no unknowns or catch-alls, that's suspicious. A realistic vendor will acknowledge the grey zone rather than pretend it doesn't exist.

And this is also where inflated accuracy claims hide, so it's worth knowing what to look for before you sign a contract.

What counts as "good" accuracy?

On the syntax errors and known valid addresses, you should expect 100%. No excuses there. On catch-alls, any honest vendor will admit the result is probabilistic. The real differentiator is how they handle the grey zone. Do they tell you honestly, or do they just mark everything valid and move on?

If you'd rather not build this file from scratch, or you want a second opinion on a vendor you're already using, talk to us. We've seen what the benchmark results look like across a lot of providers, and we're happy to share what we know.