A model upgrade broke the AI in two different ways.
Moving to Sonnet 5 first took every generator offline. After that fix, the model often sent tool arguments in the wrong shape, such as a list wrapped inside a string. On one prompt, every reply was malformed.
Why it was hard, what I decided, how AI helped
- Why it was hard
- The content in the broken replies was correct; only the encoding was wrong. Some retries happened to succeed, which made a reworded query look like a fix when it was luck.
- My approach
- I asked for a probe that measures the failure rate instead of guessing, and for every idea to be tested at a sample size large enough to trust.
- How AI helped
- Claude Code wrote the probe and replay scripts, tested the candidate fixes, and wrote the tolerant decoder.
- The decision
- Stay on Sonnet 5 and make the bad shapes harmless, rather than switch models and change the product's writing voice.
- Evidence
- commits 7876f3a, 2ee1a9b · PR #178





