Four questions to ask when a vendor says "we already test"
Oct 10, 2026, 11:28:38 AM · Rick Bawcum
Ask an AI vendor how they make sure their product behaves, and a good one will tell you they test it. Many will show you a dashboard.
That is a fair start. Treat it as evidence, and ask what the evidence covers. These four questions work in a sales call, an RFP or a renewal conversation, and none of them accuses anyone of anything.
1. Who chose the test cases?
If the vendor wrote them, the results show how the product performs on what the vendor expected. That is useful. It does not show what nobody thought to test. Ask whether anyone outside the company has reviewed the test set, and whether you can see the categories it covers.
2. Do the scenarios look like your organization?
Vendor test sets usually reflect the vendor's typical customer. An association has its own situations: member records, dues and renewals, advocacy and policy questions, volunteer and donor conversations, a board that will ask what happened. Ask which of those the tests include.
3. What does the product do when someone outside the company uses it?
Internal traces show what the vendor can see inside the system. Your members see something different, and the gaps tend to be where problems appear. Ask whether the product has been tested from the outside, the way a member would meet it.
4. How current is the result, and what happens if it changes?
A screenshot of a dashboard describes one day. Models, prompts and settings change. Ask how often the testing is repeated, who is told when a result gets worse, and whether you would hear about it.
What a good answer sounds like
A confident vendor welcomes these questions. They will show you what they test, say where the gaps are and explain how they respond to a bad result. Be wary of a vendor that treats the questions as a challenge, or that answers with a certificate instead of an explanation.
If the answers are thin
That does not mean the product is poor. It means you do not yet have a basis for comparing it with the alternatives. An independent assessment can supply one, and the person who has to explain the decision to a board or a funder usually wants exactly that.
Cimbiotic is independent AI assurance for associations and nonprofits. CimAssure tests chatbots and agents from the outside and reports an Assurance Statement of Green, Amber, Red or Not rated. It is assurance, not a guarantee, and it is not a certification, audit or attestation. Agent testing is available now.
Where does your organization actually stand?
Seventeen questions, about five minutes, no sign-up to see your result.