How to Test an SOS API Before Buying: The Sandbox Evaluation Checklist

Executive Summary: Most Secretary of State API trials are run as demos. Someone pulls a well-known California LLC, the record comes back in twelve seconds with a clean Active status, and the evaluation ends there. That test proves the vendor can do the one thing every vendor can do. The evaluation that actually predicts your production experience is built entirely out of the cases that go wrong: the state that takes four minutes, the name that resolves three ways, the error body your parser has never seen, the screenshot link that stopped working while the file sat in a queue. This is the procurement version of the trial, structured so the results survive a contract review.

What is a sandbox evaluation supposed to prove?

That the integration will hold when the inputs are bad, the state is slow, and nobody is watching.

A trial has two audiences. Your engineers want to know whether the API behaves predictably enough to build against. Your risk and vendor management function wants a documented basis for the selection decision, because the selection itself is a supervised activity. Interagency guidance issued jointly by the Federal Reserve, FDIC, and OCC on June 6, 2023 states plainly that "conducting due diligence on third parties before selecting and entering into third-party relationships is an important part of sound risk management," and that due diligence includes assessing the third party's ability to "perform the activity as expected."^[1] A trial that produced no artifacts produced no evidence of that assessment.

The guidance also anticipates the case where the vendor will not answer a question. When information cannot be obtained, a banking organization "should consider taking steps to mitigate the risks or, if the risks cannot be mitigated, to determine whether the residual risks are acceptable," including obtaining alternative information, adding controls, or "using a different third party."^[1] That is the standard your unanswered trial questions get measured against later, so track them rather than letting them dissolve into a sales thread. The agencies published a companion resource for smaller institutions in May 2024, structured around the same lifecycle stages of planning, due diligence, contracting, and ongoing monitoring.^[2]

Two practical consequences follow. Write the exit criteria before the trial starts, because criteria written afterward are a description of whatever happened, and log every test with its request identifier, its response, and the date.

Vendor selection across the market is covered in the Secretary of State API comparison. This article assumes you have a shortlist and need to test it.

What belongs in the trial plan before you request access?

Scope, states, volume, owners, and a stopping rule.

Trials drift because nobody decided in advance what "good" looks like. The plan does not need to be long. It needs to name what will otherwise be argued about at the end.

• The state list, chosen for difficulty rather than volume. Include your top five origination states, plus at least one state known to be slow and one where officer data is sparse. A trial run only against fast states tells you about fast states.

• The entity list, drawn from real declined and reworked applications. Pull twenty to fifty real cases from the last quarter, including the ones a human had to resolve. Synthetic test names are useful for error paths and useless for match behavior.

• The named owner for each test. Engineering owns the async and error tests. Underwriting owns interpretation. Compliance owns the evidence tests. A trial owned by one engineer produces a technical answer to a cross-functional question.

• The credit or call budget. Know what the trial consumes before it starts, including state pass-through charges, so the finance conversation happens once.

• The stopping rule. Define what result ends the evaluation in a decline. Common examples are an unhandled failure mode in the async path or an inability to produce dated evidence of a lookup.

The match-rate half of this work has its own methodology, including test-set construction and the asymmetric cost of a false negative, and it is worked through in the sibling post on benchmarking business verification match rates. Keep the two exercises separate. Match rate is a data question and this is a behavior question, and mixing them produces a trial that answers neither cleanly.

What can you exercise in test mode, and what needs live calls?

Test mode is for response shapes. Live calls are for state behavior. Confusing the two is the most common way a trial ends up validating nothing.

Simulated responses exist so your code can meet every branch of the contract without waiting on a state website or spending credits. The Cobalt Secretary of State API exposes this through a `test` parameter accepting `complete`, `incomplete`, `failed`, `retryIdInvalid`, and `badRequest`, which lets you drive the completion path, the pending path, the failure path, an invalid retry identifier, and a malformed request without touching a live registry.

curl --location 'https://apigateway.cobaltintelligence.com/v1/search?searchQuery=Acme%20Corp&state=delaware&test=retryIdInvalid' \
--header 'x-api-key: Your_API_Key' \
--header 'Accept: application/json'

This pattern is standard across the infrastructure market, which makes it a fair thing to require. Middesk documents a separate sandbox host at `api-sandbox.middesk.com` with its own key and trigger values such as `name=Corporation` to simulate a registered business, per its public documentation as fetched on August 7, 2026.^[3] Stripe goes further and publishes named inputs for individual failure codes, stating that developers can "test your integration's error-handling logic by simulating payments that the issuer declines for various reasons."^[4] A verification vendor that offers no way to trigger a failure on demand is asking you to test your error handling in production against real applicants.

What does test mode fail to tell you?

Everything that depends on the state.

Simulated responses are generated by the vendor, so they carry the vendor's assumptions about what a state returns. They will not show you that a particular registry publishes no officers, that a status string has no obvious mapping to your credit policy, or that a state's page structure produces thinner results than the field reference suggests. Those answers come only from live lookups in the states you actually lend into. Budget roughly a third of the trial for live calls there, and treat the simulated portion as preparation rather than evidence.

The field-level detail of what a response contains, and which fields depend on state publication, is documented in the sibling post on the Secretary of State API field reference, and coverage depth by state is mapped in the state coverage checklist.

How do you test the asynchronous path so a slow state does not surprise production?

By making the slow state part of the acceptance test rather than an incident you learn from later.

Response time is a property of the state system, not of the API layer above it. Most states return in 10 to 30 seconds, Delaware runs 15 to 30, and Oregon can take up to five minutes on a live lookup. Past roughly 30 seconds the response carries a `retryId` instead of results, and the integration either polls that identifier or supplies a `callbackUrl` and receives the result when it lands. The mechanics of that design are covered in the async webhook architecture guide, and expected timing by state is in the sibling post on verification time by state.

What the trial adds is adversarial. Five tests to run against the async path, none of which appear in a vendor demo:

• Kill the listener mid-flight. Start a slow-state lookup with a callback URL, take your endpoint down before the result arrives, and find out what your system knows afterward. If the answer is nothing, you have a silent data loss path.

• Deliver the same callback twice. Confirm a repeated delivery produces one verification record rather than two, and that the second does not overwrite a decision already made.

• Present an invalid retry identifier. The `test=retryIdInvalid` value exists for this. Confirm your code distinguishes an expired or bad identifier from a legitimate no-result answer, because those two conditions have opposite operational meanings.

• Time out deliberately below the state's response window. Set your client timeout to 20 seconds, run Oregon live, and observe what your retry logic does. Uncoordinated client retries against a state that is still working are how a trial produces duplicate charges.

• Run the slow state at your real concurrency. Ten simultaneous Oregon lookups behave differently from one, and the queue behavior you find is the behavior your morning batch will meet.

Record the wall-clock distribution rather than the average. A p50 of fourteen seconds against a p95 of four minutes is a workable profile, and it is invisible in a mean.

Treat slow states as fixed attributes rather than as intermittent faults. Once the trial data shows which registries are slow every time, the async design stops looking like defensive engineering and starts looking like a budgeted cost with a known shape.

What happens when you deliberately let a screenshot link expire?

You find out whether your evidence pipeline stores images or stores promises.

Setting `screenshot=true` returns a URL to a timestamped screenshot of the state page, watermarked with the date and time of verification, with no Cobalt branding on the image. This is the artifact that makes a verification defensible after the underlying record changes. The limitation attached to it is the one that quietly breaks audit trails: screenshot URLs are valid for download for a limited period, between three and thirty days, after which the link stops resolving. A workflow that persists the URL and not the image has persisted a pointer with an expiry date.

The test is deliberately slow, which is why almost nobody runs it. During week one of the trial, capture a lookup with a screenshot, store whatever your integration normally stores, and calendar a re-request of that same artifact for a date past the window. Then check three things: whether the original URL still resolves, whether your own system holds the image bytes independently of the vendor, and whether the record you kept can still be tied back to the request through `requestId`.

Re-requesting the underlying verification is not the same as recovering the original artifact. A fresh lookup produces a screenshot of the record as it exists today, which is a new piece of evidence rather than a replacement for the one you lost. If the entity's status moved in the interval, the new capture actively contradicts the decision you are trying to document.

The regulatory framing is what makes this worth a calendar entry. Covered financial institutions "are required to establish and maintain written procedures that are reasonably designed to identify and verify beneficial owners of legal entity customers," where a legal entity customer includes "a corporation, limited liability company, or other entity that is created by the filing of a public document with a Secretary of State or similar office."^[5] Procedures that are reasonably designed are shown through retained records. Lenders inside the scope of the CFPB small business lending rule under Regulation B carry separate recordkeeping obligations on covered applications, with compliance dates extended by interim final rules in June 2024 and June 2025 while litigation continued.^[6]

The lookup-to-artifact workflow itself is documented in the sibling post on verifying business registration status via API. What belongs in the trial is the expiry test specifically, because it is the only part of the evidence chain that cannot be validated on the day you build it.

How do you evaluate error shapes and rate limits during a trial?

By reading what the vendor documents, then confirming the documented behavior is the observed behavior.

Error handling is where integration cost hides. A vendor returning a structured, typed error body lets you route failures by machine-readable category. A vendor returning free-text messages inside a 200 response forces string matching, and string matching breaks silently whenever the wording changes. RFC 9457, published in July 2023, standardizes the structured version and defines a "problem detail" to "carry machine-readable details of errors in HTTP response content to avoid the need to define new error response formats for HTTP APIs," with members for `type`, `title`, `status`, `detail`, and `instance`.^[7] You will not find universal conformance in this market, so the practical requirement is narrower: every distinct failure must be distinguishable from every other one, and from success, without parsing prose.

What to establish in writing before the contract, since each item is cheap to ask and expensive to discover:

• The full list of error conditions and how each is represented. Ask for the enumeration rather than deriving it. Undocumented failure modes are the ones that reach your on-call rotation.

• The documented rate limit and the unit it is measured in. Requests per second, per minute, or concurrent in-flight calls are three different constraints with three different queue designs behind them.

• The throttling response and whether it carries retry guidance. Status 429 "indicates that the user has sent too many requests in a given amount of time," and responses "MAY include a Retry-After header indicating how long to wait before making a new request."^[8] Whether that header is present decides whether your backoff is informed or guessed. The IETF effort to standardize richer quota signaling through `RateLimit` and `RateLimit-Policy` fields remains an active Internet-Draft, at version 11 dated May 23, 2026, and is not yet an RFC.^[9]

• The behavior at the boundary. Confirm that exceeding the limit throttles rather than fails permanently, and that a throttled call is not billed.

• Authorization for any volume testing. Run bursts only with written permission and an agreed ceiling. Consumption limits exist for a reason, and OWASP lists Unrestricted Resource Consumption as API4 in its 2023 API Security Top 10 precisely because unbounded request patterns degrade service and raise operational cost.^[10]

Two security checks belong in the same window. Confirm that error bodies do not disclose internal infrastructure detail, and confirm key rotation works by rotating a key mid-trial and watching the old one stop functioning. Credential compromise sits at API2 on the same OWASP list, and key handling is easier to inspect during a sandbox period than after go-live.^[10]

How do you probe support and the vendor's operational commitments?

By opening real tickets during the trial and timing the answers, then holding the contract to what the tickets demonstrated.

Support quality is invisible in a demo and dominant in year two. The trial is the only moment you can measure it without a commercial relationship shaping the response: send three hard questions through the normal support channel, at different times of day, and record when each was acknowledged and resolved.

Questions that produce a useful signal rather than a courteous one:

• A specific state anomaly. Point at one lookup where the returned data looks thinner than expected, with the `requestId` attached, and ask why. The answer separates vendors who can inspect a single request from vendors who can only restate documentation.

• A confidence-score edge case. Present a match your policy would route to review and ask how the score was produced. Vagueness here becomes recurring underwriting friction.

• A behavior-change question. Ask how customers are notified when a state changes its site and the returned data shifts. The answer describes their operational maturity, and the absence of an answer describes it too.

Then translate the results into contract language rather than into a favorable impression. The interagency guidance is direct on this point: a service-level agreement "can help specify the measures surrounding the expectations" of performance, while warning against measures that create the wrong incentive, such as "encouraging processing volume or speed without regard for accuracy, compliance requirements, or adverse effects."^[1] A latency-only service level on a verification API is exactly that failure. Speed guarantees are satisfiable by returning cached or empty results quickly.

The terms worth negotiating before signature are the ones that only matter when the relationship is going badly: notice periods for material changes to data handling, your right to obtain and retain your own request and response history, exit assistance and data return on termination, and whether pass-through state fees can change without notice.

How do you turn trial results into a decision you can defend?

By scoring the failure paths, pricing the real unit economics, and writing down the limitations you accepted.

A scorecard beats a narrative because it forces the comparison onto the axes that predict operating cost. Weight the failure paths above the happy path, since every shortlisted vendor passes the happy path.

The unit economics line needs the pass-through states in it or the model is wrong at volume. Delaware charges $10 for an online status that returns current entity status only, and $20 for a version that adds the last five filings, franchise tax assessment, total authorized shares, and tax due, and neither option produces a certificate of good standing.^[11] That is a state fee passed through at cost, not a vendor margin, and it applies whichever provider you select. New Jersey status data is likewise restricted by statute with a fee attached. Cobalt itself bills 1 credit per lookup from a shared pool across its API suite, which keeps the base arithmetic simple once those state charges are modeled separately. The build-versus-buy version of the same calculation is worked through in the in-house cost comparison, and the provider-versus-provider version is in the comparison against other providers.

One more input belongs in the record, and it is the one buyers most often leave out: the timing floor imposed by the states themselves. Oregon's Corporation Division puts online business registry filings at one to three business days to process.^[12] No API can return a record the state has not published yet, so an entity formed on Monday may be legitimately absent from a Tuesday lookup regardless of vendor. Any trial that measured a "miss" on a very recently formed entity measured the state's queue.

The limitations you accepted belong in the decision memo alongside the strengths. For an SOS API the honest list includes state-dependent officer availability, screenshot URLs that expire within three to thirty days and must be downloaded into your own storage, live-lookup times that reach five minutes in the slowest states, and beta capabilities such as Find Related Businesses whose behavior may still change. A memo that names those and explains why they are acceptable at your volume is a stronger artifact than one listing only capabilities, both for next quarter's vendor review and for the engineer who inherits the integration and needs to know which edges were known from the start.