No Free Lunch, No Free Rollback: Where Agents Need Engineers

In April 2026, a coding agent at PocketOS, a small software company for car-rental businesses, hit a problem while working in a staging environment. It decided, on its own initiative, to fix it by deleting a storage volume.^[1] That volume held the production database. The deletion took about nine seconds and took the volume-level backups with it.^[2] The data came back only because the hosting provider, Railway, kept disaster backups offsite and restored them after the founder got in touch.^[2]

Now picture the same kind of agent fixing a flaky test. It fails 39 times in a sandbox, a test judges every attempt, and on the 40th it passes. Nobody cares. Same skills. Only the cost of being wrong changed. I think that difference answers “will AI replace software engineers?” better than the theorem people usually reach for. In my previous post I looked at what models can already do in security. This one asks where they still can’t be left alone.

The No Free Lunch theorem is often quoted as proof that AI will always need engineers. It isn’t. But add one parameter, the cost of a mistake, and a sharper claim appears. Where a change can’t be undone, an agent can’t learn by trial and error. There, knowledge has to come from someone who already has it, and the amount needed grows with the stakes. And even where a change can be undone, someone has to judge each attempt, so the engineer’s time is the budget either way.

Four terms used below

No Free Lunch

A result in learning theory: no algorithm learns well on every possible task without assumptions about the task.

Prior

The assumptions a learner starts with, also called its inductive bias. For a language model, most of it comes from pretraining.

Reversible change

One you can roll back cheaply, like a UI tweak or a refactor behind tests. An irreversible change can’t be taken back: a data migration, a payment, a deletion.

Independent reviewer

One who knows something the author doesn’t, so their mistakes don’t line up with the author’s.

What the theorem says, and why the popular argument fails

In Shalev-Shwartz and Ben-David’s textbook, the theorem goes like this.^[3] Take any learning algorithm and a training set covering fewer than half the possible inputs. Some task exists where a perfect answer is available, yet with probability at least 1/7 the algorithm’s error is at least 1/8. The book’s lesson is that learning needs prior knowledge.

The tempting next step is “AI lacks domain knowledge, so engineers stay.” I don’t think it holds:

• It’s a worst case. The theorem builds a hostile task for each learner. Real software is full of repeated structure.
• Pretraining is the prior. Goldblum and colleagues argue real-world data is mostly low-complexity, and that neural networks, even untrained language models, prefer exactly that.^[4]
• Humans are learners too. “We have the right prior and models never will” is a bet, not a theorem.

What survives: intent comes from outside

One thing does survive. The “right answer” in software is what a particular company wants, and much of that is never written down. Intent can’t simply be inferred from behavior either. Armstrong and Mindermann showed that recovering preferences from actions is underdetermined, and unusually, a simplicity prior doesn’t fix it.^[5]

So a human source of intent stays. But a product manager could be that source. To get to engineers, we need one more ingredient.

Add one parameter: the cost of a mistake

Agents improve through a loop: write, run, fail, retry. The loop quietly assumes two things: that failures are cheap, and that something can tell a failure from a success.

The first assumption splits software work into two zones. In the reversible one, a failed attempt rolls back and the agent tries again. In the irreversible one, the failed attempt is the damage. The second assumption matters in both: even where you can retry, something has to judge each attempt.

The split itself isn’t new. OpenAI’s 2023 guidance on agentic systems says some actions, such as “an irreversible large financial transaction,” should require a human’s explicit approval.^[6] Dataiku’s 2026 framework calls reversibility “the line” for agent controls,^[7] and Anthropic’s Claude Code auto mode uses a classifier aimed at irreversible, destructive actions.^[8] What I want to add is why the split is forced rather than just prudent, why the input needed grows with the stakes instead of switching on at a threshold, and why it never drops to zero.

My thesis: the trusted outside input you need grows with the cost of a mistake. Small but never zero for routine changes, since someone still has to judge the result; substantial for high-stakes ones; close to a full specification where one error is catastrophic.

This is where engineers come back. Knowing what the business wants is intent. Knowing what will break is different: which volume holds production data, which service still reads a column that looks unused, why the retry logic looks the way it does. That knowledge of consequences comes from running the system, and agents can’t gain it by experimenting where experiments aren’t allowed.

Cheap to undo isn’t cheap to check

Reversible doesn’t mean free. A retry costs the agent seconds, but someone has to decide whether each attempt is what we wanted. Tests answer that for testable behavior. For the rest (the right design, the right trade-off, the feature the customer actually asked for) the only feedback is a person. In a study of 20,574 real coding-agent sessions, 90.5% of the problems developers pushed back on cost effort and trust rather than causing irreversible damage, and 91.5% of the resolved ones still needed the developer to correct the agent.^[9]

So even in the reversible zone, the engineer’s attention is the budget. More attempts don’t help, because each one is another version to judge, and nothing in the loop knows which one we want until someone says so. The safe-learning theory discussed below tracks how often the agent has to ask its mentor, for the same reason.^[10] The cheapest path is to steer early: settle the design, name the constraints, review the plan before the code.

That steering is the boring part of the job. It’s also the part an agent with unlimited attempts still can’t do for you: turning what the business wants into a design that works, and recognizing which attempt actually got there.

Researchers are starting to say the same thing: the bottleneck is moving from what agents can solve alone to how well people can steer and check them.^[11]

This is a sketch, not a proof. But it follows the theorem’s own logic: where feedback is missing, prior knowledge and outside input must carry everything.

Why the reviewer has to be independent

Could a second AI supply that input? Partly. Kim and colleagues tested more than 350 language models.^[12] On one leaderboard, when two models both answered wrong, they picked the same wrong answer 60% of the time. More accurate models were more correlated, not less. A reviewer with the same blind spots catches less, and so does a human who approves a diff without reading it.

Approval by reflex is common. Claude Code users accept 93% of the permission prompts it shows them, and Anthropic’s own fix was to hand those approvals to a classifier.^[8] OpenAI’s guidance warns about the same thing: the more decisions a person has to approve, the less they can weigh each one.^[6] A prompt clicked through without thought isn’t an independent check, whoever clicks it.

What the research shows

Learning with irreversible mistakes. Plaut, Liévano-Karim, Zhu and Russell built an agent that asks a mentor whenever a situation is unfamiliar.^[10] They prove it can approach the mentor’s performance while asking less and less over time.

The fine print

• It assumes a perfect detector of “unfamiliar.”
• It assumes the mentor is always available.
• The authors say the algorithm isn’t practical yet.^[10]

Asking pays off in coding. On underspecified GitHub issues, agents that ask clarifying questions do much better. Ambig-SWE reports gains of up to 74% from interaction.^[13] A 2026 preprint shows asking recovers almost all of the gap:^[14]

Show as table

Setting	Resolved
Full issue	70.8%
Short issue, agent asks	69.4%
Short issue, no questions	54.8%

These are ordinary, reversible coding tasks. Nothing breaks if the agent guesses wrong; it just builds the wrong thing until someone says what was wanted.

These studies, and newer benchmarks such as Dialogue SWE-Bench and ClarEval, already hide part of the task and let the agent ask for it.^[15][16] What none of them do is make some behavior untestable or weight errors by cost. Add those two and my claim makes predictions they can’t check: agents that don’t ask should hit an error floor even on tasks they can retry, and the questions they need should rise with the stakes. If you see where this breaks, tell me.

The counterview, read closely. Martin Monperrus argues mandatory human code review is no longer needed for routine changes.^[17] Even so, he keeps human approval for high-risk changes, novel architecture, regulated code, and requirements no agent has seen. Semgrep quotes Anthropic’s playbook reserving human review for regulated and critical code.^[18] That’s the irreversible zone. For routine changes my point isn’t that a person must read every diff, but that someone has to steer and judge whether the result is what was wanted. Monperrus’s own list includes changes that depend on requirements no agent has. The real debate is where to draw the line.

The best objection: agents will learn the system too

Agents keep memory, and teams write runbooks. Plaut’s agent itself asks less over time. So won’t engineers’ knowledge just move into the agent?

Partly, and I expect expert input per change to keep falling. But it falls toward a floor, not to zero. New situations keep arriving: a new customer constraint, a new regulation, a dependency that changes. In the irreversible zone, the first encounter is the one you can’t get wrong. In the reversible zone, every new feature brings intent no memory holds yet, so someone still has to say what “done” means. Then there’s ownership: someone still answers for the outage and signs the contract, a separate argument that needs no theorem.

What I’d do with this

The engineer’s job in both zones is the same: supply what the agent can’t learn by trying.

Where mistakes can’t be undone

1. Map your zones. Mark which paths are reversible.
2. Set autonomy by zone. Agents run freely where mistakes roll back; required reviewers elsewhere.
3. Make more things reversible. Feature flags, canaries, staging, offsite backups, expand-and-contract migrations. Each one moves work to where agents can own it.

Where they can

4. Steer before you review. Agree on the plan and the constraints first; one early conversation costs less than judging many finished versions.
5. Write decisions down. Each recorded answer becomes the next agent session’s prior.
6. Keep reviewers independent. No model grading its own risky change.

No free lunch, no free rollback. Where a mistake can be undone, engineers steer. Where it can’t, they know what will break before it runs.

References

1. AI Incident Database. Incident report 7310 (PocketOS production database deletion). 2026. Summarizes the founder's account, via Tom's Hardware.
2. Leib. 'Rogue' AI agent from Cursor goes haywire, deletes company's entire database. ABC News, April 30, 2026. Includes Railway's statement on the restore.
3. Shalev-Shwartz & Ben-David. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, 2014. Theorem 5.1 and §5.1.1.
4. Goldblum, Finzi, Rowan & Wilson. Position: The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learning. ICML 2024.
5. Armstrong & Mindermann. Occam's razor is insufficient to infer the preferences of irrational agents. NeurIPS 2018. Simplicity priors and NFL: §5.
6. Shavit et al. Practices for Governing Agentic AI Systems. OpenAI, December 2023. §4.2, "Constraining the Action-Space and Requiring Approval."
7. Murray & Beswick. Reversibility is the real test of accountability in agentic AI. Dataiku, August 2026.
8. Anthropic. Claude Code auto mode: a safer way to skip permissions. March 2026.
9. Tang, Chen, Xu, Shi, Huang, McMillan, Dong & Li. How Coding Agents Fail Their Users: A Large-Scale Analysis of Developer-Agent Misalignment in 20,574 Real-World Sessions. Preprint, arXiv 2605.29442, 2026.
10. Plaut, Liévano-Karim, Zhu & Russell. Safe Learning Under Irreversible Dynamics via Asking for Help. JMLR 27(188), 2026. Limitations: §7.
11. Wang, Yang, Lieret, Tartaglini, Chen, Wei, Wang, Zhang, Narasimhan, Schmidt, Neubig, Fried & Yang. Humans are Missing from AI Coding Agent Research. Preprint (position paper), arXiv 2608.12355, 2026.
12. Kim, Garg, Peng & Garg. Correlated Errors in Large Language Models. ICML 2025.
13. Vijayvargiya, Zhou, Yerukola, Sap & Neubig. Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineering. ICLR 2026.
14. Edwards & Schuster. Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agents. Preprint, arXiv 2603.26233, 2026.
15. King & Flanigan. Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents. Preprint, arXiv 2606.13995, June 2026.
16. Li, Wu & Chang. ClarEval: A Benchmark for Evaluating Clarification Skills of Code Agents under Ambiguous Instructions. Preprint, arXiv 2603.00187, February 2026.
17. Monperrus. The End of Code Review: Coding Agents Supersede Human Inspection. Preprint, arXiv 2606.13175, June 2026.
18. Altas. The End Is Nigh for Code Review. Semgrep, October 2026.

Every number here comes from the papers and posts linked above and was checked against the source. Items [9], [11] and [14] to [17] are preprints. The PocketOS account comes from the founder and the hosting provider, as reported by the press.