Insights

"Our vendor tests it" is a product claim. What would Day 2 evidence look like?

Nov 12, 2026, 6:30:00 AM · Rick Bawcum

← All insights

A good answer that is not yet evidence

When you ask a vendor about its AI, "we test it" is often true. Good vendors do test. The difficulty is what that sentence can and cannot show a Board.

This is not an accusation. A vendor's testing describes how its product performs in general. Your Board is asking something narrower: how does the assistant behave for our members, with our content, today?

Four gaps between a product claim and Day 2 evidence

  1. Version. Which version did they test? Hosted AI changes. A result for last quarter's model describes a different system from the one answering now.
  2. Questions. Whose questions did they use: theirs, or the ones your members actually ask? Eligibility, fees, deadlines and anything touching money or rights are specific to you.
  3. Change. What did they do after the last model or content change? A test that ran once at launch does not show what happened since.
  4. Grader. Who graded the result? If the team that built the product also grades it, the result is an honest self-assessment. It is not independent.

For a fuller version of these questions, see Four questions to ask when a vendor says "we already test."

What a vendor cannot test for you

A vendor's test cannot exercise your configuration, your content and your members. It also cannot test how its product behaves next to the other AI you run. Many associations have several assistants from different vendors, each bought separately, and each vendor sees only its own part.

None of this makes vendor testing less valuable. It makes it one input. The Day 2 question is whether you also have a record that comes from the outside.

What to ask this week

Choose one vendor and ask, in writing: when did you last change the AI we use, what changed, and how do we find out next time? A vendor that can answer clearly is giving you something to put in the file. A vendor that cannot is showing you where the gap is.

Where Cimbiotic fits

CimAssure supplies the independent record. We test the assistant as a member would use it, against a versioned set of tests, keep every reply as evidence, and report a dated Assurance Statement. Cimbiotic neither builds nor sells the assistants it tests. It provides assurance, not a guarantee, and it is not a certification, audit or attestation.

Choose your next step:

Where does your organization actually stand?

Seventeen questions, about five minutes, no sign-up to see your result.

Take the free Scorecard Book a 30-minute call