A one-time result describes one day. Continuous assurance describes the product you use every day
Oct 10, 2026, 11:28:40 AM · Rick Bawcum
An AI product is not the same product from one month to the next. The vendor pushes an update. The underlying model is swapped for a newer one. A prompt is adjusted, a new feature is switched on, or someone in your organization changes a setting in the platform.
Each of those can change how the product behaves. Most arrive with a release note at best, and none of them comes with evidence that the product is still doing what it did when you bought it.
The gap a single assessment leaves
A one-time assessment is a photograph. It tells you how the product behaved on the day it was tested, which is useful but not enough. The question your Board or your members will ask is about today: is it still behaving within the guardrails we set?
That question has to be answered again each time something changes. It is the reason assurance for AI cannot be an annual event.
What continuous independent assurance means
The idea is simple to state.
- A baseline. Before anything else, there is a defined set of tests and a recorded result for the product as configured.
- Re-testing as things change. The same tests run again on a regular schedule, and on demand when the customer tells us the vendor has changed something, so a change is noticed instead of discovered.
- A comparison, not just a score. The question is what moved since the last result, and whether it moved inside or outside the guardrails.
- A status people can act on. When behavior drifts, the status changes, and the people responsible hear about it.
Independence matters at every step. If the party that ships the update also decides whether the update passed, the buyer is back to relying on the preparer.
Whose guardrails
The guardrails are the organization's, not the vendor's and not ours. They are the things your assistant must not say, the member data it must not touch, the questions it must hand to a person, and the tone it keeps when someone is upset. Writing them down is the first piece of work, and it is useful to a board on its own.
This includes your own choices. A setting an administrator selects in the platform is a change like any other, and the assurance should cover the product as you actually run it.
Who tells whom
The regular run sets the rhythm. Responding quickly to a specific change depends on a short chain of notice.
Vendors should be obligated, in their contracts, to tell their customers about material changes: product updates, model changes, new features and altered defaults. Customers should pass that on to Cimbiotic as soon as they hear, and that triggers a re-test on demand. The same goes for changes you make yourself in the platform.
If you buy AI for your organization, ask for the notice obligation before you sign, and name the person who receives it.
Where Cimbiotic fits
Cimbiotic is independent AI assurance for associations and nonprofits, and monitoring what organizations actually run is part of the work. CimAssure tests chatbots and agents against a versioned test library, re-tests on a regular schedule and on demand when a customer reports a vendor change, and reports an Assurance Statement of Green, Amber, Red or Not rated. Agent testing is available now. CimAssure is assurance, not a guarantee, and is not a certification, audit or attestation. Cimbiotic is aligning to ISO/IEC 17065 and is not currently accredited.
Where does your organization actually stand?
Seventeen questions, about five minutes, no sign-up to see your result.