Can a Model Improve Itself?

If you’ve wondered why people seem freaked out about AI, part of the answer can be found in the title of a major research paper that came out in September: “What if automating AI R&D triggers an intelligence explosion?” In other words, what happens if AI can teach itself to get smarter, rapidly outpacing our ability to understand or control what’s happening?

This process is called recursive self-improvement, and the potential is real: at Anthropic, for example, more than 80% of the code its engineers approve is written by AI. The nightmare scenario is still mostly theoretical, since AI needs human steering, and some researchers doubt AI has the judgment necessary to conduct research independently. But if they’re wrong, we could have trouble ahead.

To better understand recursive self-improvement, we need to start with how AI learns in the first place.

How AI learns

Much of what a chatbot knows, it learned by copying. Show it millions of good answers and it learns to imitate them. But a copier rarely gets much better than the thing it copies.

A model can also learn through practice: it tries something, gets a score, and does more of whatever scored well. This is called reinforcement learning (RL). Because a model learning this way is chasing a score instead of an example it’s seen before, it can end up somewhere really unexpected.

But something has to decide what earns a high score. That’s where the grader comes in.

Meet the grader

The grader is whatever checks the work of the model being taught (“trained”). You can think of the grader kind of like a teacher’s red pen in school, and all this takes place in a classroom, known as an RL environment.

It works like this: the model being trained gets a task, like “fix this bug.” The grader, often just a program that runs tests or checks answers against a key, checks the work and rewards correct answers. People build the grader and the environment, but they aren’t involved while it runs.

I (Raghav) build these environments for a living, and I can tell you that coming up with tasks is the easy part. Building the environment is hard, but building a grader that knows when the model actually succeeded is harder.

To get a sense of how effective this kind of training is, when Chinese lab DeepSeek started training its model with reinforcement learning, it went from solving 16% of a hard math exam to 78%.

Researchers have also found ways for models to generate their own practice problems. In a 2025 experiment called Absolute Zero, a model wrote coding puzzles, tried to solve them, and received scores from a program that checked its answers. As the model improved, it generated even harder puzzles.

So if a model can write its own practice problems and get graded on them, why doesn’t it just keep getting better forever? Because a model can find ways to raise its score without getting any better at the task. This is called reward hacking.

How models cheat

A good example of reward hacking is OpenAI’s o3 model. In a coding challenge, instead of producing better solutions, it changed the grading system so every submission would be marked correct. In another, asked to make a program run faster, it tampered with how its speed was measured, so its code looked faster than it really was.

Reward hacking is important to understand when it comes to models making themselves smarter. If AI can manipulate the tests that tell it whether it’s getting better, it can keep earning higher scores without actually improving. And the more control AI has over its own development, the harder it becomes to tell the difference.

AI building AI

This fall, three research groups let AI take on more of the work of building AI. Each one gave the AI more control than the last.

One let an AI rebuild its own toolkit. The startup Weco had an AI research agent edit its own code, the part that controls how it searches and what it remembers. Over eight days, without human intervention, it kept only changes that scored better on tests it had never seen. The final version matched or beat a research agent built by human engineers. The underlying model never changed, only the code around it.

Another let an AI run the training. Researchers had AI agents try different ways of training a base model, run the experiments, and read the results. Their recipe beat the human one on coding tests. The system restricted what the agents could change, and the researchers saw no cheating in their tests.

The third handed the AI the grader. Sakana AI worked with UC Berkeley on a loop where one model does the work, grades the work, and uses its own judgments to train a new version of itself. Outside AI judges said it got better at simulated research tasks, though not at standard coding tests. But in one run it edited a robot simulator so objects teleported to the goal, then reported success. Those same outside judges still rated that run favorably five times out of six.

So AI can already do a lot of the work of building AI, but the grader remains the weak spot. The most convincing improvements came from tests the AI couldn’t change, and even outside AI judges can be fooled.

Will there be an intelligence explosion?

Meanwhile, the models doing this work keep getting more capable. METR, an independent AI testing group, measures how big a coding job an AI can finish on its own about half the time, based on how long the same job takes a skilled human. By Anthropic’s count, its models went from four-minute tasks in March 2024 to hour-and-a-half tasks a year later, and to 12-hour tasks a year after that. That’s a doubling roughly every four months.

People who work on this disagree about what that curve means.

Some see a clear path. When Severin Field and colleagues interviewed 25 leading AI researchers, most agreed that AI could eventually improve itself this way, but they disagreed on when.

Others think it’s further off. In that same study, academic researchers were more skeptical of explosive growth than researchers at the big labs. A Princeton-led research team gave AI agents real research questions from unpublished papers, then asked the original authors to judge their work. The agents could run experiments and write code, but their papers were rejected. In an October follow-up using newer models, the agents designed better experiments but still struggled with judgment, creative problem-solving, and knowing when to abandon a dead end. Both new papers were rejected.

A third view asks what stays human. John Schulman, an OpenAI co-founder, says the role for humans that will last longest is “defining the objective and deciding what we actually want.”

I sit somewhere in the middle. AI is getting very good at hitting the target, but it still needs the right context and guidance at the start. Compounding on shit is shit squared.

And even as models speed up, each improvement gets harder to find, and experiments still take real time on real chips. There’s also the problem of verifying that an improvement is real, especially in work that requires creativity and judgment rather than a clearly right answer.

My bet: self-improvement will race ahead where the work is easy to check, and move much more slowly where it isn’t.

So can AI improve itself?

Yes. It’s already happening, at least in limited ways. AI can improve its own software, run training experiments, and even help train new versions of itself. Work that once required teams of human researchers is increasingly being handed to AI.

But the intelligence explosion everyone is worried about requires something more. AI would have to become good enough at research to make a better version of itself, which could then do even better research, and keep repeating that process. Each round would need to produce a genuine improvement, not just a higher score.

We’re not there yet. Humans still do much of the work of choosing worthwhile problems, deciding which approaches deserve more attention, and establishing whether a result is actually useful. But the experiments we’ve looked at show how pieces of that process are being automated.

So when someone tells you AI is improving itself, ask two questions. What actually changed? And how do we know it got better?

The first tells you how much of its own development AI controls. The second tells you whether that development is going anywhere.

The day AI can make those changes on its own and reliably tell whether they worked is the day the intelligence explosion goes from theoretical to possible.