The AI you tested last spring is rarely the AI answering your members today, and in most cases nobody sent you a notice that it changed.
That is not a failing of any one vendor. Hosted AI is a service, not a product you install. The model, the instructions around it, the infrastructure that serves it and the features switched on can each change without your involvement. A test result is evidence about the system on the day it ran. It is useful evidence, and it is dated.
For an association, the practical consequence is plain. A pre-launch test, a vendor questionnaire or a demo that went well in the spring does not tell your Board what the chatbot is doing in October.
Change arrives through more doors than most boards expect. Five are worth naming.
1. The model is retired or replaced. Vendors retire models on their own schedule. OpenAI's help page on retiring models lists GPT-4o, GPT-4.1, GPT-4.1 mini, o4-mini and GPT-5 as retired in ChatGPT on February 13, 2026, the GPT-5.1 family on March 11, 2026, and GPT-4o in Custom GPTs for Business, Enterprise and Edu on April 3, 2026. Custom GPTs built on a retired model move to the closest current equivalent. The page does not state a notice period. A chatbot you tested on one model can be answering on another without anyone at your organization having changed a setting.
2. The same model name behaves differently over time. In a study by Chen, Zaharia and Zou (arXiv 2307.09009), GPT-4's accuracy on a prime-versus-composite task fell from 84% to 51% between March and June 2023. The authors were measuring behavior on one task, and the result should not be read as a general verdict on the model. The point it supports is narrower: an identical product name does not mean identical behavior.
3. An update shifts behavior in a direction nobody intended. In April 2025, OpenAI rolled back a GPT-4o update that made the model noticeably more flattering and agreeable. OpenAI said it had focused too much on short-term feedback. The update was reversed. For the weeks it was live, anyone relying on an earlier test was relying on a different system.
4. Infrastructure faults degrade quality quietly. Anthropic's postmortem of three recent issues, published September 17, 2025, describes three bugs between early August and mid-September 2025. In the worst hour, on August 31, a routing error affected 16% of Sonnet 4 requests. Anthropic said detection came mainly from user reports and that its own evaluations had not captured the degradation. It committed to running quality evaluations continuously in production. Disclosure: Claude, the AI assistant that helped draft this post, is built by Anthropic.
5. Your own configuration and features move. Switching on a new tool, connecting a new data source, editing the system prompt, changing the knowledge base or enabling a vendor's new feature all change what the assistant does. These changes are made by your staff or your vendor with good intentions, and rarely trigger a retest.
None of these requires bad faith. Each is an ordinary part of running a hosted AI service. Together they mean the system you tested is a moving target.
You cannot stop a vendor from changing its service. You can decide how you find out and what you do next.
None of this needs a large team or a large budget. The first pass takes an afternoon. What earns trust is repeating it on a schedule.
Re-testing shows you that behavior changed. It does not tell you why, and it does not prevent the change. A fixed question set also covers only the questions you thought to ask. Testing on a schedule narrows the gap between a change and your awareness of it. It does not close it.
This is the problem CimAssure is built around. We test chatbots and agents against a versioned set of tests and report the result as an Assurance Statement: Green, Amber, Red or Not rated. Because the tests are versioned and the statement is dated, a re-test is comparable with the one before it. Agent testing uses test records and Cimbiotic test members, never real member data.
If you run an AI assistant today, begin with step one: write down what you tested and when. If you do not have that record, you have your answer to the question in the title.
The Trust Readiness Scorecard has 17 questions and shows where your oversight is strong and where it is thin. If you would like a second set of eyes on a specific assistant, the CimAssure Baseline is a fixed-price, three-week review.
Choose your next step: