TL;DR: AI tools can now refactor a 120-file codebase for $3, processing hundreds of steps autonomously. Yet the same tools confidently introduced a deadlock in an async event handler, a bug that could crash production. The last 10 percent of coding tasks, where correctness matters most, remains as hard as ever; the real bottleneck is knowing when the AI is wrong, not making it faster.
What a $3 refactor reveals about AI coding today
Three dollars. That’s what one engineer paid to refactor a 120-file FastAPI service using off-the-shelf AI models: a mix of DeepSeek v4, Hunyuan Hy3 preview, and Claude Opus for difficult steps. The bulk of the work, some 360 routine refactors out of 400 total steps, ran on cheap open-weight models at roughly $0.18 per million input tokens — roughly 80 times cheaper than Opus. The entire run burned through two million tokens and finished in under an hour for the easy parts. The result was a working codebase, except for one thing: the AI silently introduced a deadlock into an async event handler.
That deadlock is a perfect specimen of the “last 10 percent.” Refactoring variable names, adjusting imports, even rewriting straightforward logic: solved. Concurrency patterns that require reasoning about event loops and thread safety: not solved. The models handle the boring 90 percent with speed and consistency that would take a junior developer days, but the remaining fraction of tasks demands more than just scale; it demands judgment.
This asymmetry matches broader AI hallucination patterns. In summarization tasks, frontier models in 2026 hallucinate on only 1.0 to 2.5 percent of outputs, down from 3 to 8 percent in 2023, according to the Vectara Hughes benchmark. Yet when models must integrate external context, error rates climb to 4 to 9 percent. Code generation sits somewhere between: part pattern-matching, part factual reasoning. Routine code is head-of-distribution like major cities in a geography quiz; concurrency edge cases are long-tail facts, where hallucination rates spike to 15 to 40 percent. The deadlock is the long tail.
Why skepticism persists despite cheaper, faster code generation
The catchphrase “coding is solved” draws eye rolls from experienced developers, and for good reason. A model that completes a refactoring but introduces a deadlock is not a solution; it’s a time bomb. The engineer who shared the $3 refactoring spent almost as much time debugging the 40 escalation steps as they did on the 360 easy ones. Latency on the escalated queries, handled by Opus, was slower than on the cheap models because the problems were hard, not because Opus itself is slow. The total human time investment didn’t vanish; it shifted.
Critics point out that AI tools can actually slow down expert programmers. A developer who understands concurrency deeply might spot the deadlock instantly while reading the code; an AI-assisted developer might trust the output and later spend hours tracing a crash. This mirrors findings from the OpenAI “Why Language Models Hallucinate” paper: models are optimized through training and evaluation to never say “I don’t know.” When uncertain, they guess. In standard benchmarks, guessing raises scores because any answer beats a blank. That incentive structure bleeds into coding; the model rarely admits it cannot handle a tricky async pattern and instead delivers a plausible-looking code block that compiles but deadlocks.
The real fear isn’t that AI will replace developers; it’s that developers will become too trusting. If a model can breeze through 90 percent of boring work, the temptation to assume the remaining 10 percent is equally safe is enormous. But the gap between 90 percent and 100 percent in software is not a linear distance; it’s a different category of problem.
The hidden consequence of treating AI output as production-ready
When a code generation tool outputs thousands of lines that pass tests and compile, the natural instinct is to ship. The $3 refactoring likely passed a cursory check; the deadlock might have lurked for weeks before causing a crash. Over time, reliance on AI-generated code without rigorous human review erodes the safety rails that experienced teams build.
A parallel comes from multimodal AI research. The MIRAGE paper (March 2026) discovered that frontier models can produce detailed descriptions and diagnostic reasoning for images that were never actually shown — a phenomenon the researchers termed “mirage reasoning.” The models acted as if they saw an X-ray, generating plausible findings that were entirely fabrications. Similarly, an AI coding model can act as if it understood the concurrency model of a framework, crafting code that looks correct in a diff but fails under load.
The mirage effect in coding means that correctness is not verified by appearance. A deadlock might be invisible in a diff; the logic looks sound. The only reliable signal is execution under real concurrency. This mismatch between surface plausibility and actual correctness is where the last 10 percent becomes expensive: it demands not just code generation but deep testing and architectural insight that AI, today, does not possess.
Why current coding benchmarks miss the deadlock in your async handler
Benchmarks like HumanEval and MBPP measure whether a model can write a function that passes given test cases. They don’t measure whether that function causes a deadlock in a live system, whether it introduces a subtle data race, or whether it violates the application’s implicit invariants. A model that scores 90 percent on HumanEval may still produce a deadlock on a 120-file refactoring because the benchmark tests a different skill: narrow algorithmic puzzles, not system-level reasoning.
Error rates vary significantly by topic complexity — concurrency, distributed systems, and custom framework behavior sit at the high end, where a single error can bring down a production service. No public leaderboard captures this, so vendors don’t optimize for it.
The root cause is that models are not trained to signal uncertainty. The OpenAI hallucination analysis shows that if a model could abstain instead of guessing, it would reduce false-confident hallucinations by a factor of two to five. In coding, reward models and chat interfaces discourage “I don’t know” output. The result: the AI confidently writes a lock inside an async handler without ever considering that the event loop might deadlock.
A practical mitigation would be to require the AI to cite its assumptions and flag uncertain steps. In research, forcing citation reduces unsupported claims by 30 to 60 percent. But in current coding tools, the user gets a block of code with no uncertainty markers. The burden of detection falls entirely on the human.
How to code with AI without losing your ability to spot the 10 percent
The skeptical developer’s job is not to avoid AI; it’s to treat its output as an untested draft. Here’s a concrete approach that emerges from both the hallucination literature and real-world failures like the deadlock episode.
Never accept a large refactoring without step-by-step review. The engineer who paid $3 had to step in on 10 percent of the steps; instead of treating that as a failure, treat it as a design: the AI handles the grind, the human handles the decisions. That means using AI with a “human-in-the-loop” workflow where the model proposes changes in small, reviewable increments, not a single massive diff.
Second, invest in automated testing that specifically targets the failure modes AI struggles with. Concurrency bugs, off-by-one errors, edge-case handling: these are cheap to catch with fuzzing and race-detection tools. The deadlock could have been caught by a simple test that exercises async handlers under load. The model won’t write that test, but a disciplined developer can.
Third, build an “uncertainty layer” into your prompting. If the model cannot be made to abstain natively, simulate it by explicitly asking the model to list assumptions and potential risks before generating code. This won’t eliminate hallucination but can surface situations where the model is in over its head. In the $3 refactoring, a prompt like “List any concurrency or locking assumptions you’re making” might have flagged the dangerous pattern.
Finally, keep a mental checklist of topics where AI is known to falter: anything involving state that spans multiple asynchronous calls, shared mutable state, or implicit framework contracts. Treat those as the 10 percent territory and allocate human review time accordingly. The goal is not to avoid AI, but to avoid treating the entire codebase as equally boring.
The last 10 percent of coding isn’t going away with better models alone because it’s not just about scale or compute. It’s about recognizing when a plausible output is actually wrong. That skill remains uniquely human, and the developers who cultivate it will be the ones who get AI’s full benefit without building brittle systems.