Hallucination in Production Agents: The Real Risk Profile Nobody Discusses Honestly

The agent that hallucinates 3% of the time is not your biggest liability. The reviewer who has stopped reading its output is.

I went looking for a clean answer to a simple question: what is the actual LLM hallucination production risk once you strip away the vendor demos and the doom posts, and just look at what happens when these systems touch real customers, real money, real legal exposure? I expected to find a number. I found something else — a pattern in how companies respond to the number that turns out to be riskier than the number itself.

What Is The Actual Hallucination Rate You’re Shipping With?

There isn’t one rate. There’s a range that depends entirely on domain, and the domains where hallucination is most likely are the ones where the consequences are worst.

Vectara has run an open hallucination leaderboard since 2023, scoring models on faithfulness to source documents in summarization tasks. The best-performing frontier models sit around 0.7%–1.5% hallucination on that narrow, document-grounded task. That number gets cited constantly, and it’s real. It’s also close to the best-case scenario: short context, source text right there, one job.

Pull the same models into an open-ended domain and the number stops looking reassuring. Stanford’s RegLab and Institute for Human-Centered AI ran general-purpose LLMs against real legal questions in 2024 and found hallucination rates between 58% and 82% depending on the model and the type of legal query — often stated with total confidence, and often in response to questions the model should have recognized it couldn’t answer (Dahl et al., “Large Legal Fictions,” 2024). That’s not a fringe case. Legal, medical, and financial queries are exactly the domains agentic products are being built for right now, because that’s where the labor cost is high enough to justify automating it.

So the first wrong turn is assuming there’s a single hallucination rate you can benchmark against and clear. There isn’t. The rate is a function of how far your agent’s task sits from “summarize this document,” and most valuable production use cases sit pretty far from that.

Okay, So Just Put A Human In The Loop — Doesn’t That Solve It?

No. The evidence says a human reviewer’s ability to catch AI errors degrades sharply the moment the AI is right most of the time — which is exactly the condition under which you’d feel safest adding one.

This was the part that actually surprised me, because “add a human checkpoint” is the default answer everyone reaches for, including me, including in systems I’ve built. It sounds like free insurance. The research on automation says otherwise. Raja Parasuraman and Dietrich Manzey’s review of decades of human-automation studies found a consistent effect they call automation complacency: as an automated system’s reliability increases, human monitoring of that system’s output decreases, and errors slip through at higher rates precisely because the system is usually right (Parasuraman & Manzey, Human Factors, 2010). Skitka, Mosier, and Burdick found the same thing earlier, in a decision-making context closer to what an “approve this AI output” reviewer actually does: people default to trusting the automated recommendation and stop independently verifying it, even when instructed to remain vigilant.

Translate that into a Slack channel where someone approves 200 AI-drafted replies a day, and the mechanism is obvious. The first week, they read every one. By week three, an output that looks fluent and confident gets a reflexive approve, because the false-positive base rate has trained them to expect fluent and confident to mean correct. The reviewer hasn’t gotten lazy. Their attention has been correctly, predictably recalibrated by the system’s own reliability. That’s the mechanism, not a character flaw, which is what makes it so hard to fix by telling people to “just pay more attention.”

So the human-in-the-loop isn’t a safety net that costs you review-time and buys you risk reduction. Past a certain volume, it’s a safety net that costs you review-time and buys you the appearance of risk reduction, while the actual catch rate quietly falls toward zero.

What Is The “Safe” Choice Actually Costing, Then?

The safe choice — keep a human gatekeeping every AI action, or don’t ship the agent at all — costs you the thing you can’t easily see: the gap between what a well-scoped autonomous system could have caught and what your degraded human checkpoint actually catches, plus everything you didn’t build while you were staffing that checkpoint.

Here’s a number that made me stop and reread it. Gartner predicted in mid-2025 that over 40% of agentic AI projects would be scrapped by the end of 2027 — not primarily because the underlying models were unreliable, but because of unclear business value, escalating costs, and inadequate risk controls bolted on after the fact (Gartner, June 2025). Read that next to the complacency research and a different story shows up. It’s not that the cautious, heavily-gated projects survive and the reckless ones get canceled. A huge share of the cautious projects get canceled too, buried under review overhead that never gets redesigned once it stops working, dragging out a build cycle until the business case evaporates.

Meanwhile the review layer itself has a cost that never shows up on the same ledger as the hallucination it was meant to prevent. Every human-reviewed AI output is a human who isn’t doing something a human is uniquely suited for. That’s a real headcount cost, and it’s usually invisible because it gets absorbed into “just part of the process” rather than booked as the price of avoiding a specific, nameable risk. Companies that never run this math end up with the worst of both: a review process expensive enough to slow them down, and — per Parasuraman and Skitka — not actually reliable enough to catch the failure mode it exists for.

That’s the quotable version of it: the cost of the cautious path isn’t the caution, it’s that most caution is theater with a real invoice attached.

Doesn’t Air Canada Prove Agents Are Just Too Risky For This?

No — it proves that unbounded scope is too risky, which is a different and much more useful finding. In Moffatt v. Air Canada (2024 BCCRT 149), a customer relied on a fabricated bereavement-fare policy invented by Air Canada’s support chatbot, and the tribunal held the airline to it, ruling that a company is responsible for the representations of its automated agents the same way it’s responsible for a human employee’s (CBC News, Feb. 2024).

That case gets cited as the canonical argument against deploying agents at all. I expected, going in, that it would hold up as that argument. It doesn’t, on closer read. The tribunal wasn’t ruling that chatbots are inherently untrustworthy — it was applying an old, boring liability principle that predates LLMs by a century: you own what your agent says on your behalf, human or otherwise. The actual failure in that case was scope, not model choice. Nothing bounded the bot to verified policy text; it was free to generate plausible-sounding policy on demand, and it did.

Compare that to Mata v. Avianca (2023), where a lawyer submitted a brief with six fabricated case citations generated by ChatGPT and got sanctioned for it (Reuters, June 2023). Same root failure: a claim generated with no grounding step and no verification step before it reached someone who relied on it. In both cases the fix that would have prevented the outcome wasn’t “don’t use an LLM” — it was “don’t let the LLM assert anything it can’t cite back to a real source,” which is an architecture decision, not a caution level.

That’s the second wrong turn I had to back out of. I went in assuming the fix for hallucination risk was more oversight. The case law says the fix is narrower authority — an agent that can only say things traceable to a document, a policy, a record — paired with verification that happens whether or not a tired human is still reading closely. That’s a harder thing to build than “add an approval step.” It’s also the only version of the fix that doesn’t decay the way human vigilance does.

So Where Does That Leave Me?

Narrower than I expected, and less resolved than I’d like. I’m fairly convinced the instinctive “safe” move — throw a human reviewer in front of every agent output — is mostly a cost with a shrinking safety return, not a real control, past whatever volume makes the AI right often enough for complacency to set in. I’m fairly convinced the actual lever is scope and grounding, not headcount.

What I haven’t resolved: at what volume does that complacency curve actually bend, and does it differ meaningfully by task type — I found the mechanism, not the threshold. And there’s an uncomfortable recursive problem I don’t have an answer for: if you replace human review with an automated verification layer, what stops that verifier from suffering its own version of complacency once it’s right often enough? I don’t think anyone has actually measured that yet. I went looking for the risk profile of hallucination in production. I came back with a second, unmeasured risk profile for whatever you build to guard against the first one.