Executive Summary: Every business verification vendor will quote you a match rate. Almost none will tell you the denominator, the applicant population it was measured on, or how a partial match was scored. That number is not portable to your book. The only figure that predicts what a provider will do inside your underwriting flow is the one you produce yourself, on your own applicant names, with a labelling protocol written down before you saw any results. This is the methodology: how to draw the test set, how to build the name variants that break matchers, how many records you need before the result stops being noise, where to set acceptance thresholds, how to price a false negative, and how to compile it into a scorecard that survives procurement.
Why Do Vendor-Published Match Rates Fail to Predict Your Match Rate?
A published match rate is a ratio, and a ratio without a stated denominator carries no information. The failure is structural: the vendor measured something real, on a population that is not yours.
Two different quantities travel under the single label "match rate." Precision is "the ability of the classifier not to label as positive a sample that is negative," or true positives over true positives plus false positives. Recall is "the ability of the classifier to find all the positive samples," or true positives over true positives plus false negatives.[1] Record linkage uses the same split, defining precision as "the proportion of predicted matches which are true matches" and recall as "the proportion of matches that are correctly predicted."[2] A vendor quoting 97 percent has given you one of these and withheld the other, and the two move in opposite directions as a threshold shifts.
Which Population Did the Vendor Measure?
Vendor benchmarks are typically run against registry-sourced names, meaning the query string was drawn from the same database that holds the answer. That construction guarantees a high number and tests nothing your intake will encounter. Your applicant names arrive from a broker portal, a bank statement header, or a signed application, carrying dropped suffixes, trade names, ampersands rendered as "and," and the occasional entity that was administratively dissolved four months before it reached you. Provider selection across the broader market is covered in the guide to Secretary of State API solutions and their business match rate accuracy; this post is the test you run after you have a shortlist.
What Does a Partial Match Count As?
The scoring rule is where most informal evaluations quietly break. A candidate list holding the right entity at position four is a success for recall and a failure for automation. A record with the correct legal name and a registered agent from a prior filing is correct on the field you searched and stale on the field you will underwrite against. Decide the rule before the run, write it into the labelling instructions, and apply it identically to every vendor. The split between exact-identifier resolution and fuzzy name resolution, and the separate denominators each requires, is worked through in the companion post on company lookup APIs versus company search APIs.
How Do You Build a Test Set That Reflects Your Actual Applicant Population?
Draw the test set from files your team has already resolved, because those are the only records where you know the answer. Formal accuracy analysis depends on "a sample of labelled (ground truth) matches, which may have been produced (for example) by human beings."[3] Ground truth is a prerequisite for any edge metric, and clerical labelling is how it gets made.[4]
Where Should the Seed Records Come From?
Pull from five buckets and keep them tagged, because per-bucket results are the part of the output that changes a decision.
• Funded and resolved files. Applications where a human confirmed the entity and stored the state entity ID. These are your positive controls and should be the largest bucket.
• Declined files with a confirmed entity. Declines carry messier names, since a share of them were declined for reasons tied to the entity record itself.
• Files resolved in manual review as no-registration. Your true negatives, and the only way to catch a vendor that manufactures a plausible match for an entity that does not exist.
• Files abandoned mid-review. Tag and exclude these from the scored set. You do not know the answer, and including them silently inflates whichever error rate they fall into.
• Recent formations under 90 days old. State indexes lag filing, and this bucket separates a vendor problem from a registry problem.
Fifteen to twenty percent of the scored set should be known negatives. Without them, a vendor that returns a confident wrong answer on every unregistered applicant scores the same as one that correctly returns nothing.
What Stratification Actually Changes the Result?
Stratify by state first, weighted to your actual origination mix. State response behavior varies enough that a test set skewed toward fast states will overstate every timing figure you produce. Stratify second by entity type, since officer and agent field availability differs by form. Stratify third by name shape, separating single-token names, names containing an ampersand, names with a legal-form suffix, and names where the applicant supplied a DBA. A 200-record set stratified three ways yields thin cells, which is a reason to size up and not a reason to skip the stratification.
How Do You Label Without Poisoning the Result?
Two reviewers, blind to which vendor produced which response, labelling from the state record and not from the vendor output. Measure their agreement with Cohen's kappa, "a statistic that measures inter-annotator agreement," running from -1.0 to 1.0 where "the maximum value means complete agreement" and zero "means chance agreement."[10] A kappa below roughly 0.7 on the first pass means the labelling instructions are ambiguous, and the honest response is to rewrite them and relabel instead of averaging the disagreements away.
A benchmark whose labelling rules were written after the first results came back is a description of those results. The rules have to be frozen before the first API call, and the freeze has to be a file with a timestamp.
Which Name Variants Belong in the Test Set, and How Do You Generate Them?
Seed names come from your files. Variants are generated deliberately, because the variant distribution is what separates a matcher that handles real intake from one that handles clean queries.
Legal-form suffixes are the highest-yield category and the most standardized. ISO 20275 exists to specify "the elements of an unambiguous scheme to identify the distinct entity legal forms in a jurisdiction," and the February 2026 code list covers "more than 3,600 entity legal forms across more than 200 jurisdictions."[8] For a US test set that means each entity's suffix has a small, enumerable set of written forms, and every one of them shows up in intake.
Which Transformations Should You Apply to Each Seed Name?
Apply these deterministically so the variant set is reproducible across vendors and across quarters.
• Suffix expansion and contraction. LLC, L.L.C., "Limited Liability Company," and the suffix dropped entirely. Four variants from one seed.
• Punctuation stripping. Remove commas, periods, apostrophes, and hyphens, then test stripped and unstripped forms separately.
• Ampersand substitution. "&" to "and" and back. Intake systems disagree constantly, and registries store one form.
• Article and stop-word removal. Drop a leading "The" where the registered name carries one.
• Truncation at the field limit. Cut the name at 30 and at 40 characters, reproducing what a legacy origination field does to it.
• DBA substitution. Replace the legal name with the trade name the applicant wrote on the application, keeping the true entity ID as the label.
• Single-character corruption. One transposition and one omission per seed, the minimum realistic typo load from manual rekeying.
That produces roughly ten to twelve variants per seed. Score each against the same ground-truth entity ID and report the per-transformation failure rate alongside the pooled number. The pooled number tells you a vendor is at 88 percent. The per-transformation breakdown tells you the misses are concentrated in DBA substitution, which is an intake-form change on your side and not a vendor switch.
How Many Variants Are Too Many?
Variants inflate apparent sample size without adding independent information, since ten variants of one entity are ten correlated observations. Report two figures: the entity-level match rate, where a seed counts as matched if any variant resolved correctly, and the variant-level rate, which is the automation-relevant number. When they diverge sharply, the vendor can find the entity and your intake path cannot reach it.
How Large Does the Test Set Need to Be Before the Result Means Anything?
Two hundred entities is the working floor for a single pooled figure, and it is not enough for per-state or per-transformation claims. The arithmetic is standard proportion testing. The NIST/SEMATECH handbook sizes a proportion test from the baseline rate, the difference you care about, and the two error rates; its worked example of detecting a 0.10 shift from a 10 percent baseline at 5 percent significance yields "N ≥ 102," rising to 112 with the continuity correction.[6]
What Can 200 Records Support, and What Can They Not?
Two hundred supports a statement like "Vendor A resolved 88 percent and Vendor B resolved 81 percent, and the gap exceeds sampling error at our tolerance." It does not support "Vendor A is weak in Georgia" when Georgia contributed eleven records. The same handbook states the validity condition for the normal approximation on a proportion, requiring N above 30 and "min{Np₀, N(1-p₀)} ≥ 5."[7] At a 5 percent expected failure rate, that condition alone puts a floor of 100 records on any cell you want to make a claim about. Per-state claims need per-state cells, which is how a 200-record pilot becomes an 800-record production benchmark once the pilot shows the vendors are close.
How Should Sample Size Change After the First Run?
Size the second run against the gap you observed. When two vendors land three points apart on a 200-record set, the difference sits inside the noise and the decision has to rest on cost, coverage, or support. When they land twelve points apart, further sampling will not reverse the ranking and the money is better spent testing the leader against your slow states. Coverage depth by state, which drives most of the per-state variance, is mapped in the state coverage checklist for Secretary of State APIs.
What Should Your Acceptance Thresholds Be, and Where Do the Numbers Come From?
Thresholds are a policy decision informed by a measurement, and the measurement is a sweep. A threshold sweep shows "how the number of false positives and false negatives varies depending on the match threshold chosen."[3] Run it on your labelled set and plot precision and recall against the score, adding specificity, negative predictive value, accuracy, F₁, F₂, F₀.₅, and the Matthews correlation coefficient if your tooling produces them.[5] Precision rises as you raise the cutoff and recall rises as you lower it, and the sweep shows the shape of that trade on your data instead of in the abstract.
What Do Published Defaults Tell You About Reasonable Bands?
Published matcher defaults give you a sanity range. OpenSanctions ships a default score threshold of 0.7 with the algorithm "calibrated to be used with this value," and notes that "for sanctions screening with low tolerance for false positives, this could be raised to 0.8 or even 0.85."[9] Cobalt's SOS Search returns a 0.0 to 1.0 confidence score on every result with three published bands: 0.8 to 1.0 is high confidence and most customers auto-accept, 0.5 to 0.79 is moderate and gets flagged for review, and below 0.5 is likely a different entity. Those bands are a starting position, and your sweep tells you whether your applicant mix justifies raising the auto-accept floor.
What Should You Demand From a Vendor at This Stage?
Ask four questions and treat non-answers as answers. Does the score have a published calibration, or is it an unlabelled internal number. Is the candidate list truncated before scoring, and at what limit. Is a no-result response distinguishable from a source timeout. Does the score mean the same thing in every state. Sandbox procedures for putting those questions to a live endpoint are in the pre-purchase SOS API test checklist.
How Do You Put a Dollar Figure on a False Negative?
The cost model turns an accuracy chart into a procurement decision, and it starts by rejecting the assumption that the two error types cost the same.
A false negative is a fundable applicant whose entity the system failed to find, costing the expected margin on a deal that dies in review or closes late out of a manual queue. A false positive is a confirmed match against the wrong entity, costing the loss content of funding against a record that describes a different company plus the file-reconstruction work if it surfaces during an examination. In most alternative lending books the per-event false-positive cost is larger and the false-negative volume is higher, so neither dominates by inspection.
Which Line Items Belong on Each Side?
Build the model on your own figures and keep it to six inputs so it stays auditable.
• False-negative unit cost. Expected contribution margin per funded deal, times the share of manually recovered files that fail to close because of the delay.
• Manual recovery cost. Reviewer minutes per adjudication, times the loaded hourly rate, times the touches an average recovery takes.
• False-positive unit cost. Expected loss given funding against a misidentified entity, weighted by your recovery rate on those files.
• Remediation cost. Hours to reconstruct a decision file when an examiner questions the verification, which is where audit-grade evidence changes the number.
• Volume. Applications per month passing through the verification step.
• Base rates. Observed false-negative and false-positive rates from your benchmark, per vendor.
How Does the Cost Ratio Select Your Scoring Metric?
Use the cost ratio to pick one comparison metric so vendors can be ranked without argument. The F-beta measure is "a weighted harmonic mean of the precision and recall," where beta above 1 weights recall more heavily and beta below 1 weights precision more heavily.[1] In record linkage terms beta is set so that "recall is β times more important than precision," and F₂ is the standard choice when missed matches hurt more than wrong ones.[2] Set beta to the square root of your false-negative to false-positive cost ratio, round to 0.5, 1, or 2, and state the choice in the scorecard. A team whose false positives cost four times a false negative ranks on F₀.₅ and will pick a different vendor than a team ranking on F₂.
How Do You Measure Manual Review Rate Without Guessing?
Manual review rate is the operational number that shows up in headcount planning, and most teams estimate it from memory. Measuring it means instrumenting the decision point before the benchmark starts, since the data does not exist retroactively. Log a fixed record per call, at decision time and not after a reviewer has cleaned it up. The SOS Search response carries the fields you need in one payload:
{
"status": "complete",
"statusCode": 200,
"requestId": "abc123-def456",
"results": [
{
"title": "ACME CORPORATION",
"filingDate": "2015-03-15",
"status": "Active",
"entityType": "Corporation",
"registeredAgent": {
"name": "CT Corporation System",
"address": "1209 Orange St, Wilmington, DE 19801"
},
"sosUrl": "https://icis.corp.delaware.gov/ecorp/...",
"screenshotUrl": "https://screenshots.cobaltintelligence.com/..."
}
],
"possibleAlternatives": []
}What Belongs in the Log Line?
Capture the `requestId` for reconstruction, the length of `possibleAlternatives` at decision time, the confidence score as returned and not bucketed, the retrieval path used, the wall-clock latency, the adjudication outcome, and the reviewer minutes spent. Manual review rate is adjudications divided by calls, computed per vendor and per state. Reviewer minutes times loaded rate gives you the annualized operating cost difference between two vendors that looked three points apart on accuracy, and that difference is frequently the larger number.
How Do Slow States Distort the Measurement?
Record timing per state, because pooled latency hides the tail that drives review queues. Most states return a live Cobalt lookup in 10 to 30 seconds, Delaware runs 15 to 30 seconds, and Oregon can take up to five minutes. Requests exceeding 30 seconds return asynchronously through a `retryId` you poll or a `callbackUrl` you register, so your benchmark script has to implement the async path or it will record artificial failures on the states that matter most:
curl --location 'https://apigateway.cobaltintelligence.com/v1/search?searchQuery=Acme%20Corp&state=oregon&liveData=true&callbackUrl=https://yoursite.com/webhook' \
--header 'x-api-key: Your_API_Key' \
--header 'Accept: application/json'Two further constraints belong in the benchmark budget. Delaware charges a state fee for online status, $10 for the tier that returns the current status screen and $20 for the tier that adds the last five filings, franchise tax assessment, and total authorized shares, and that fee passes through at cost on every Delaware record in your test set.[11] Officer data availability also varies by state because the API returns what the state publishes and nothing beyond it, so a per-field completeness score will show legitimate gaps that are registry limitations and not vendor failures. Oregon's own public interface splits business lookup across a basic name search, a separate advanced name search, and a distinct active-registrations search, which illustrates the variation any aggregating layer has to absorb.[12]
What Does the Finished Scorecard Look Like When You Run It?
The output is one page per vendor with seven scored lines, each traceable to a number your benchmark produced: entity-level match rate, variant-level match rate, top-hit precision, recall across the full candidate list, manual review rate, median and 95th-percentile latency by state, and false-answer rate on the known-negative bucket. Weight them by your cost model using the beta you derived, and record the weights in the same file.
How Should the Weights Be Set?
Weight manual review rate at least as heavily as raw match rate, because it converts directly into salaried hours and vendors do not publish it. Treat the known-negative false-answer rate as a gate instead of a score: a vendor above your tolerance there fails regardless of the other six lines, since a system that confidently identifies entities that do not exist has an error mode no threshold will fix. Give latency a weight proportional to how much of your volume sits in slow states, which your stratification already told you.
What Decision Does the Scorecard Support?
It supports a ranked shortlist with a stated tolerance, and rerunning the same measurement two quarters later against the frozen test set to detect drift. It does not support a credit policy. Cobalt Intelligence returns the state record, the normalized status, the confidence score, and a timestamped screenshot; what a 0.62 confidence match means for a $180,000 advance sits with your underwriting team and your regulator, and no benchmark output changes that. Where the verification step belongs relative to the rest of the stack is laid out in the KYB API guide for lending risk assessment and the walkthrough on verifying business registration status via API.
Teams that run this process usually find the accuracy gap between shortlisted vendors is narrower than expected and the manual review gap is wider. That inversion is the finding, and it is only visible to a team that measured its own reviewer minutes instead of reading a match rate off a slide.
.jpg){kind=small}
.png)