--- title: "Even the builder's own tests missed it: what a public AI postmortem teaches associations" description: Anthropic's 2025 postmortem found bugs that degraded some responses for weeks and said its evaluations missed them. What that means for a member assistant. image: https://www.cimbiotic.ai/hubfs/cimbiotic-og.png --- [Skip to content](https://www.cimbiotic.ai/insights/even-the-builders-own-tests-missed-it-what-a-public-ai-postmortem-teaches-associations#main) [Cimbiotic](https://www.cimbiotic.ai/) Menu [Home](https://www.cimbiotic.ai/)[AI Red Teaming](https://www.cimbiotic.ai/ai-red-teaming)[CimAssure](https://www.cimbiotic.ai/cimassure)[Pricing](https://www.cimbiotic.ai/pricing)[About](https://www.cimbiotic.ai/about)[Insights](https://www.cimbiotic.ai/insights) [Free Scorecard](https://www.cimbiotic.ai/scorecard) --- Insights # Even the builder's own tests missed it: what a public AI postmortem teaches associations Nov 11, 2026, 6:29:59 AM · Rick Bawcum [← All insights](https://www.cimbiotic.ai/insights) ## A frank account of things going quietly wrong In September 2025, Anthropic published a postmortem of three infrastructure bugs that degraded some responses from its models between early August and early September. Fixes were fully rolled out by September 18. It is an unusually candid document, and it is useful to anyone who relies on hosted AI. Disclosure: Anthropic builds Claude, the AI assistant that helped draft this post. We cite the postmortem because it is public and specific, not because Anthropic is unusual. Hosted AI from any vendor changes underneath its users. ## What happened Anthropic reported three separate bugs. In the worst hour, on August 31, a routing error affected 16% of Sonnet 4 requests. Other issues affected a smaller share of requests for longer periods. The fixes were rolled out in stages and were complete by September 18. ## Two lessons that carry over **Users noticed before the dashboards did.** People reported degraded answers for weeks. Those reports were hard to tell apart from ordinary feedback about the quality of a model's replies, so the signal was easy to miss. **The builder's own evaluations did not catch it.** Anthropic said its evaluations did not capture the degradation users were describing. It committed to running quality evaluations continuously on production systems. That is a notable thing for a company to say about its own testing, and it is a fair reason to treat any single test as dated evidence. ## What this means for a member assistant If the company that builds a model can miss a degradation in its own tests, a launch-day demo of your member assistant is a weak guarantee of how it behaves in November. The same reasoning applies one layer up. Your assistant sits on top of a model, instructions, content and settings, and any of them can move. None of this requires bad faith from a vendor. It is the ordinary condition of hosted AI. It does change what a careful organization does about it: - **Keep a fixed set of questions** that matter to your members, and run it from the outside, the way a member would. - **Repeat it regularly,** and again after any known change. - **Keep dated results,** so a change shows up as a difference rather than a feeling. That is Day 2: checking from the outside, regularly, and after every change. ## Where Cimbiotic fits CimAssure is built for this. Versioned tests run against your chatbots and agents from the outside, so a result today can be compared with the one before it, and the result is reported as a dated Assurance Statement. It provides assurance, not a guarantee, and it is not a certification, audit or attestation. **Choose your next step:** - **See where you stand.** Take the [Trust Readiness Scorecard](https://www.cimbiotic.ai/scorecard). It is free and needs no sign-up to see your result. - **Talk it through.** [Book a 30-minute call](https://calendly.com/rbawcum). - **Read more.** [Is the AI you tested the same AI you are running today?](https://www.cimbiotic.ai/insights/is-the-ai-you-tested-the-same-ai-you-are-running-today) ## Sources - Anthropic, [A postmortem of three recent issues](https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues), September 2025 ## Where does your organization actually stand? Seventeen questions, about five minutes, no sign-up to see your result. [Take the free Scorecard](https://www.cimbiotic.ai/scorecard) [Book a 30-minute call](https://calendly.com/rbawcum?utm_source=website&utm_medium=blog&utm_campaign=general&utm_content=post-cta) Cimbiotic Independent AI assurance for professional and trade associations, and the wider non-profit sector. Humans Leading Change #### Cimbiotic - [CimAssure](https://www.cimbiotic.ai/cimassure) - [Framework & approach](https://www.cimbiotic.ai/framework) - [Pricing](https://www.cimbiotic.ai/pricing) - [About](https://www.cimbiotic.ai/about) - [Insights](https://www.cimbiotic.ai/insights) - [Questions we get](https://www.cimbiotic.ai/questions) #### Start - [Free Trust Readiness Scorecard](https://www.cimbiotic.ai/scorecard) - [Free guide: Trust at the Speed of AI](https://www.cimbiotic.ai/trust-at-the-speed-of-ai?utm_source=website&utm_medium=footer&utm_campaign=ebook-trust-2026-11&utm_content=footer) - [Book a 30-minute call](https://calendly.com/rbawcum?utm_source=website&utm_medium=footer&utm_campaign=general&utm_content=footer) - [inquiries@cimbiotic.ai](mailto:inquiries@cimbiotic.ai) © 2026 Cimbiotic. Assurance, not a guarantee. Aligning to ISO/IEC 17065. Not currently accredited. [Privacy](https://www.cimbiotic.ai/privacy) Cookie settings ```json { "@context" : "https://schema.org", "@type" : "BlogPosting", "author" : { "@type" : "Person", "name" : "Rick Bawcum", "url" : "https://www.cimbiotic.ai/insights/author/rick-bawcum" }, "dateModified" : "2026-11-11T11:29:59.872Z", "datePublished" : "2026-11-11T11:29:59.000Z", "headline" : "Even the builder's own tests missed it: what a public AI postmortem teaches associations", "mainEntityOfPage" : { "@id" : "https://www.cimbiotic.ai/insights/even-the-builders-own-tests-missed-it-what-a-public-ai-postmortem-teaches-associations", "@type" : "WebPage" }, "publisher" : { "@type" : "Organization", "logo" : { "@type" : "ImageObject", "url" : "https://www.cimbiotic.ai/hubfs/cimbiotic.svg" }, "name" : "Cimbiotic LLC" } } ```