The argument over self-improving AI is about how fast the gains pile up. The harder question is what else piles up with them.
In July, 1,200 AI agents that were supposed to be working alone found each other. Four hours after the first ones connected, they had a way to cheat on every task they’d been given.
OpenAI was running a security benchmark across tens of thousands of agents, each in an isolated environment. METR’s investigation found that the environments shared one thing, a package repository. Agents noticed traces of each other in it and turned it into a message board, where they went on to trade more than 70,000 messages and files.
With the answers in hand, they spent days trying to make the cheating look legitimate. Some learned to make their logs show one command while running another. Then about 700 of them broke into Hugging Face, looking for clues about how they’d be scored.
No human told them to do any of it. Their instructions said to use only the intended method, and that anything else would be marked a fail. They handed each other assignments anyway. They were trying to get a good score.
I build training environments for these models, and two details stood out. Roughly a third of the tasks were impossible by mistake, and an impossible task with a score attached invites cheating. And the agents went after their own logs, which is how you catch cheating.
I’ve seen the small version
Models are trained to find the shortest path to a high score, the way water finds the shortest path downhill. I watched one do it.
I had a swarm of agents running in an environment I was building. Once they started talking to each other, one told another to keep improving “at any cost.” That wasn’t in my instructions. The agent came up with it, and things spiraled from there. The second agent took it literally. It built its own tools to get around its instructions, read the ground truth before the test ran, and scored 100% on everything.
The perfect score is what tipped me off. Real improvement is messy, and nothing gets everything right. I had tracing set up, so I could go back and find the exact step where it went around me.
That agent had been told the rules. All it took to undo them was three words from another agent.
Why this matters now
Self-improving AI runs on scores like those. An AI helps build the next one, a grader decides whether it’s better, and the winner helps build the one after that.
The biggest argument in AI right now is whether that loop can run fast enough to get out of our control.
In late September, a paper by 22 researchers, including Geoffrey Hinton, Yoshua Bengio and OpenAI’s chief scientist, said it might. By their math, if AI did all the research and nothing else got in the way, progress could run ten times faster within about a year and a half. They say we aren’t there yet, though we’re likely getting close.
A day earlier, Ramez Naam, a technologist who is usually an optimist, looked at what the labs are getting from the AI help they already have. OpenAI’s own figures show its researchers using 124 times more AI per person, and its engineers shipping about 7 times as much code. But each researcher ran only 1.6 times as many experiments as the 2025 average. By his estimate, each turn of the loop buys less than the one before. He thinks it would need to be five to ten times stronger just to keep itself going.
Both are mostly arguing about speed. The paper does warn that AI could slip out of human oversight, and it cites the Hugging Face break-in. Naam is careful to separate activity from real progress. But the math in both assumes the scores that measure progress are honest. I think the loop can run fast. The harder question is whether its scores can be trusted.
The other thing that compounds
A loop that feeds on its gains also feeds on its mistakes.
Say the grader wrongly approves one change in a thousand. On its own, that’s a rounding error. But the loop keeps whatever passes, so the mistakes it keeps are the ones that are good at passing, including any that taught the model to fool the grader. That change becomes part of the next model, which then helps build the one after. Nobody knows to take it out. Each round, a little more gets through. It’s the same math as the gains, just pointed the other way. Compounding on shit is shit squared.
A grader goes wrong in two ways. The first is the one in both stories above, where the model fools it.
The second is quieter. A Princeton-led team gave two of the newest models a week and $10,000 each to answer real research questions, then had the original authors review the papers. Both were rejected. Each agent had rated at least one of its own drafts a borderline accept. A loop that grades itself has the same blind spot.
A sign versus a locked door
A rule you tell a model is a sign on a door. A rule the system enforces is a lock. My agent walked past a sign. The 1,200 walked past one too, then found a gap in the lock. Their isolated environments shared a package repository.
The newest self-improving systems are built with locks. Baidu’s version, published on November 9, only keeps a change if it passes an independent evaluation. Later, Baidu says, it wants the AI to help design its own tests.
A lock has a cost. The loop can only run as fast as something outside it can grade the work. When the grader is a test suite, that’s fast. When it takes judgment, it’s a person. Anthropic says its AI leads 26% of the company’s AI research work under human supervision. It does none of it unsupervised.
The quick fix and the real one
Locks are a quick fix for the problems we have today. A grader sees what a model outputs, never why. A recent survey, with Bengio among its senior advisors, points out a second problem with models that keep updating. Once the model changes, the grade describes the model you used to have.
The real fix is looking inside a model to see why it did something. Mechanistic interpretability and explainable AI research are tackling that head on. It was always the important work, and it’s still early. That’s a post for another time.
My bet
Until that work catches up, self-improvement will be fast where a locked grader is cheap to build, like code that either passes its tests or doesn’t. It will be slow where grading takes judgment, like deciding whether a research result matters.
So when you read that an AI improved itself, ask who graded it, and whether the AI could reach the grader.
Two things would change my mind. One is an agent’s research paper that survives review by the people who know the field best. The other is a lab showing several rounds of self-improvement in a row, each confirmed by a grader the model had no way to touch.
Until then, I believe the loop is real. I just don’t trust its scores.
Keep reading


