What Happens When the AI Gets It Wrong and You Do Not Notice

The bug was there for three weeks. Every time someone ran the test suite, it passed. Every time someone reviewed the PR, the logic looked right. The function did exactly what the comment said it did — and the comment had been written by the same model that wrote the code. It was only when the production environment diverged slightly from the test setup that anything surfaced at all, and by then the damage was already downstream.

I want to talk about that gap. Not the failure itself — the silence before it.

Why Does AI Code Feel So Convincing Even When It Is Wrong?

There is something about the way a language model writes code that makes it harder to scrutinize than code written by a human. A human who is uncertain will often signal it — a vague variable name, an awkward comment, a TODO left in place. A model never hedges. It produces clean, confident prose whether it is solving a problem it knows well or hallucinating an API that does not exist. The syntax is perfect. The indentation is perfect. The error handling looks thorough. Everything looks like the work of someone who knew what they were doing.

This is not a coincidence. These models were trained on human code that was valued for correctness, and the surface features of correct code — consistency, structure, confident naming — are easy to learn. The deeper semantics are harder. So you get code that looks right more often than it is right, and those two things are very different.

I have seen this play out more times than I expected. The bugs that come from generated code are rarely obvious — they tend to be subtle: off-by-one errors in buffer allocation, insecure default parameters, logic that passes obvious test cases but cracks under edge conditions. The kind of thing a careful developer might miss not because they are careless but because the code looks right.

The researchers who have studied this call it “asleep at the keyboard.” I think of it differently. It is not that developers are falling asleep. It is that the material we are reviewing has changed, and our review instincts have not caught up.

What Changes When You Trust the Machine More Than You Mean To?

There is a concept from aviation called automation bias — the tendency of people working with automated systems to defer to machine output even when their own judgment should override it. The failure modes break into two kinds: missing that the automation is wrong at all, and following automated guidance even when something already felt off.

Both happen with AI-generated code, and the second one is the stranger of the two. I have done it myself — read a model’s explanation of its own code, found something that nagged at me, and then accepted the explanation rather than the nag. The model was articulate. The model had a reason. My concern felt like nitpicking.

The problem is that language models are very good at producing articulate reasons for things that are wrong. If you ask a model to explain a bug it introduced, it will often explain it confidently in terms of the surrounding logic — correctly describing what the code does while missing that what it does is not what you wanted. This is not the model lying. It is the model reasoning within the frame of its own output, which is the only frame it has.

The human reviewer is supposed to provide a different frame. That is the whole point of review. But if the reviewer has absorbed the model’s frame before they start reading — if they are thinking this code was generated by an AI, so let me check for obvious errors rather than this code is a black box to me, what is it actually doing — the review degrades into proofreading.

How Do the Errors Actually Sneak Through?

The categories of AI coding errors that slip past review share some common features. They are rarely syntactic — those get caught by linters and compilers before a human even sees them. The ones that make it through tend to fall into a few quieter categories.

Logic errors that are locally coherent but globally wrong. The function does what it says on the label, and what it says on the label was the wrong thing to build. The model interpreted an ambiguous requirement in one of its valid interpretations and you did not notice the interpretation was made.

Silent data corruption. A model will sometimes produce code that processes data correctly in the happy path and drops or transforms it incorrectly at an edge — null inputs, empty arrays, timezone-naive datetimes. The test suite does not cover the edge because the person writing the tests was looking at the function signature the model provided, not at the full space of inputs the function might encounter.

Dependency misuse. Models sometimes call library functions with incorrect argument order, deprecated parameters, or version assumptions that do not match the project’s lockfile. This category has grown more common as training data ages while libraries do not.

Security assumptions. The model assumes the input has already been sanitized, or assumes the caller will handle the credential rotation, or assumes a subprocess will fail safely. These assumptions are often implicit — not stated anywhere in the code — which means they are invisible to a reviewer who does not know to look for them.

What links these categories is that they are all invisible at the surface. The code reads fine. The error lives in the gap between what the code says and what the surrounding system requires, and that gap is exactly where review breaks down when the reviewer has partially outsourced their model of the system to the model itself.

What Does a Review Process Actually Need to Change?

The short answer is that you cannot review AI-generated code the way you review human-generated code, because the failure modes are different.

Human code fails in recognizable ways — the developer was confused about something, rushed something, misread the docs, carried a wrong mental model from a previous project. You can often reverse-engineer the confusion from the error. AI code fails in statistical ways, which means the error does not reveal a misunderstanding you can correct. It reveals a gap between the training distribution and the problem at hand. The mitigation is different.

What has helped me: treating the model’s output as a first draft that is confident about everything, including the parts it should not be confident about. Reading the code in execution order rather than in the order it was written. Running the tests but also asking: what test would have to exist for this to fail? Asking the model to try to break its own function — sometimes it finds something, sometimes it does not, but the exercise forces you out of the frame.

The structural thing that matters is keeping a human who did not generate the code in the review chain, and making sure that human has enough context to have an independent model of the problem. If the reviewer’s only source of information about what the code should do is the code itself and the model’s comments, the review is circular. You are checking the model’s output against the model’s description of its own output. That is not review — it is a confidence interval.

I have noticed this in myself and in teams I have worked with: as output volume rises, review time does not rise with it. If anything, it compresses. The productivity gains are real. So is the risk they paper over.

What Should You Actually Do With This?

Review every AI-generated function as if it were written by a competent developer who did not fully understand the surrounding system — because that is, roughly, what it is.

Keep a separate mental model of what the code should do before you read what the code does. If you cannot hold that model independently, the code is not reviewable. That is a signal to go back to the prompt, not to ship it.

Do not let the model review itself in isolation. Its explanations are fluent. Fluency is not accuracy.

The tool is genuinely powerful. I use it every day. But there is a version of this that ends badly, and it ends badly quietly — three weeks of silence before the production divergence, the bug that passed every test because the tests were written from the same frame as the code. The answer is not to distrust the model. The answer is to distrust the feeling that you have already checked.

That feeling is fast. Checking is slower. One of those is worth keeping.