When an Upstream API Changes, Your Agent's Tools Break Silently

The upstream change that hurts your agent will almost never throw an error. Unfortunately, the whole toolkit we have for API breaking changes detection assumes it will.

You know the standard advice. Providers version their APIs. Consumers pin a version, read the changelog, and run contract tests in CI. If you do all that, a breaking change turns into a planned migration and you avoid the outage. I believed this for years, and it’s a perfectly good model when a break makes the compiler complain or turns a test red.

Agents don’t work that way. When a tool an agent depends on changes shape, the agent does what language models are trained to do with odd input, which is keep going. The field it expected comes back null, so it decides nothing was found. The selector matches nothing, so there must be nothing to click. The run finishes and the log says success. You never see an error. What you get is behavior that looks plausible and is wrong, and it can stay wrong for weeks.

I didn’t get this from a paper. I got it from my own logs.

Doesn’t API Versioning Already Solve Breaking Changes?

No. Versioning protects you from the changes a provider admits are breaking. The ones that hurt agents tend to be changes nobody on the provider’s side would think to call breaking.

Stripe is the standard example of versioning done well. In Stripe’s own description, each account is pinned to whatever API version was current at its first request, and backwards-incompatible changes ship behind new dated versions. It’s a careful system and I have no complaint with it. Notice what it assumes, though. The provider decides what counts as a break, and the provider applies the label.

Even under the strictest conditions, breaks are common. Xavier, Brito, Hora and Valente at the Federal University of Minas Gerais looked at 317 Java libraries, 9,000 releases and 260,000 client applications. 14.78% of API changes broke backward compatibility. Breaks got more frequent over time, and the libraries that broke most often were the bigger, more popular and more active ones. That study is from 2017, and it’s about Java, where a break at least stops the build. JSON APIs don’t have a compiler. If followers_count becomes followersCount, nothing fails at build time. The field just stops showing up.

Then you have upstream changes that have nothing to do with schemas at all. On February 1, 2023, Twitter announced it would end free API access on February 9, with paid tiers reportedly starting at $99 a month. That June, Reddit’s new pricing ($12,000 per 50 million requests) pushed Christian Selig to shut down Apollo on June 30. He put the cost at roughly $20 million a year at Apollo’s usage. Neither change was versioned. These were policy decisions, and everything built on those APIs absorbed them however it could. Apollo at least died loudly, in public, on a known date. Most integrations aren’t that lucky.

Won’t Contract Tests Catch API Breaking Changes?

Contract tests catch drift between your code and your fixtures. They can’t see drift between your fixtures and the live system, and that’s the drift that reaches production.

A contract test checks a response against a schema somebody wrote down. When the provider changes and your fixture doesn’t, the test keeps passing, because it’s checking the past. That’s why I keep calling the failure silent. Every check you control was written against the world as it looked on the day you wrote it.

My X agent is my best example. It ran a growth loop where every post, reply, follow and like went to Telegram first, and I approved each one by hand. For about six weeks, four hardcoded target accounts that no longer existed ate 88% of the bot’s daily comment attempts. Every run logged success. Nothing raised an error. The tool did exactly what the code said: look up the account, find nothing to reply to, return cleanly. Any contract test I could have written would have passed, because the contract still held. The world it described had moved on.

The same thing happened in places I could see even less of. The Telegram Skip button didn’t work for months. Every skip got recorded as a timeout, so my approval stats were wrong that whole time, and I was making decisions off them. The long-form article lane lost 8 of 20 generations to JSON parse errors, and each failure quietly ate a 48-hour slot. None of these were exotic. Each time, one side changed, the other side kept saying everything was fine, and nothing was watching the gap between them.

Can’t an LLM Agent Just Adapt to API Changes on Its Own?

It adapts, and that’s the problem. An agent that works around a broken tool also hides the break from you.

The research says this fairly openly. Berkeley’s 2023 Gorilla paper (arXiv:2305.15334) was built partly around the fact that API docs change and a model trained on stale docs will confidently call the wrong thing. Its fix is retrieval: hand the model current docs at call time. That’s useful when the docs are current. It doesn’t help when the live API has drifted away from its own documentation, which is the case I actually care about.

Sierra’s τ-bench (arXiv:2406.12045, 2024) found something worse. With tools working exactly as specified, GPT-4o succeeded on fewer than 50% of tasks. In the retail domain its pass^8, meaning it succeeded on all eight attempts at the same task, dropped below 25%. That’s how inconsistent agents are on stable tools. Now let the tools change underneath them, and the extra variance is indistinguishable from normal agent noise. When the agent does a bit worse this week, you blame the prompt, or the model. The upstream API doesn’t look any different, so it’s not on your list of suspects.

If you remember one thing from this post, make it this: an agent turns an upstream breaking change from an exception into a behavior, and nobody gets paged for a behavior.

Two X suspensions taught me that. The first came on 2026-07-21. The lockout check fired and returned True, but the caller skipped one action and kept looping for another 3.5 days. I fixed it with a process exit plus a sentinel file only a human can clear, since launchd’s KeepAlive would just respawn a plain exit. I considered the matter closed. Then X suspended the account again on 2026-09-22, a week after I raised the reply cap from 16 to 150, and this time the check never fired. The page had changed, none of its known selectors matched, and so the check concluded “not locked out.” I stopped the bot by hand. My safety mechanism was also a tool sitting on an upstream interface, and it broke just as quietly as everything else.

Isn’t This Just Monitoring, Which Everyone Already Does?

Not really. Standard monitoring tells you whether a service is up. An agent needs to know whether the service still means what it meant yesterday.

Uptime checks, latency graphs and error-rate alerts all assume failure shows up in transport, as a 500, a timeout or a spike. Every failure in this post came back 200 OK. The four dead accounts resolved. The lockout page loaded. The article endpoint answered. Put a dashboard on any of them and it would have been green the entire time.

What would have caught them is comparison over time: this week’s response shape against last week’s, the fields the agent relies on against the fields that actually came back, a selector’s hit count against its own history. That’s a different job from monitoring. It’s closer to version control for somebody else’s system. You keep snapshots of what the upstream used to return, you diff them, and you alert on the diff.

So I’m arguing that breaking-change detection should stop being the provider’s job, done once in a changelog, and become the consumer’s job, done continuously against what you actually observe. The provider has no idea which of its fields your agent quietly depends on. You’re the only one who knows.

How Do You Detect API Breaking Changes Before They Reach Production?

Record what the upstream actually returns. Diff it on a schedule against what it returned before. When the diff touches something your tools depend on, alert a human. That’s the idea behind APIDiffGuard, which I built after the X agent taught me the same lesson one time too many.

The scope is deliberately small. It monitors API responses, detects schema drift, compares versions, and alerts developers before an integration breaks. The design choice I care about most is that it diffs observed responses and doesn’t trust the spec on its own. It can import an OpenAPI document, but a spec is a promise and a response is what actually happened, and my entire argument is that the two drift apart.

The diff engine ships as its own MIT-licensed package, @apidiffguard/diff. The full app is MIT too, and you can self-host it. The hosted Cloud version is the paid option for teams that would rather not run Supabase, schedules and alerting themselves. I split it this way because the comparison logic shouldn’t be locked up. If consumer-side drift detection is the right idea, anyone should be able to drop it into their own agent harness. There are also some free tools that need no login (a JSON diff, a formatter and a validator) if you just want to compare two payloads by hand.

A few trade-offs exist because the tool fetches URLs that other people hand it. Outbound fetches block private hosts, re-validate redirects and cap response size. A drift monitor that can be aimed at your internal network isn’t a monitor. It’s a vulnerability. The /api/* fetch routes also require an authenticated session and are rate-limited.

Here’s what it doesn’t solve: semantic drift where the shape stays the same. If a provider keeps a field called status but changes what "active" means, the schema diff comes back empty and the break is still real. My four dead X accounts were partly this kind. The lookup’s structure was fine. What had quietly stopped being true was the meaning, “this account exists and is worth replying to.” A schema differ won’t catch that. You catch it by pairing the diff with outcome counters, such as how many attempts per target actually produced anything, and treating a sharp change there as drift too. I don’t think anyone has fully cracked this, and that includes me.

What Changes If This Is Right

If upstream breaks reach agents as behavior and not as errors, responsibility changes hands. The changelog is no longer the source of truth, just one input among several. Any team running an agent in production picks up a job it used to leave to vendors: keeping a record of what each upstream returned and diffing that against what the agent sees now.

It also changes what “the agent is working” means. A run that finishes cleanly doesn’t prove anything anymore. Success logs become claims you check against measured outcomes. Safety checks like a lockout detector need more drift monitoring than anything else, because when they fail quietly, nobody is supervising the system at all.

And it changes how you explain incidents. When an agent gets a little worse, don’t start with “what did the model do?” Start with “what changed upstream that nobody told us about?” I spent six weeks blaming my own strategy for flat growth while four accounts that didn’t exist anymore soaked up 88% of the work. The bot did what I told it to, in a world that had changed around it, and nothing in my stack was built to notice.