Right now, leading artificial intelligence models need human supervision to improve—but a popular theory known as “recursive self-improvement” imagines a future world where AI could become fully autonomous, prompting existential fears.
Recursive self-improvement has varying definitions, but in one version, AI becomes so advanced that it can improve itself indefinitely and build its successors. AI would be “improving very fast to the level of the smartest humans, and that’s maybe what I’m personally most scared of,” said Rishub Jain, a former Google DeepMind researcher who launched Sampura Research, an AI safety nonprofit.
In an alternate doomsday scenario caused by recursive self-improvement, AI becomes so smart it’s like “an alien taking over the world,” Jain explained, noting that he is less persuaded by this imagined future. But Jain is concerned about what happens when AI becomes so capable so quickly that there is less human oversight of what exactly it is doing.
“When AI can conduct a million actions or organize a whole company, we won’t be able to know if what it’s doing is safe or not,” Jain said. He gave the future scenario of a widespread internet outage caused by recursive self-improvement. In this example, AI development increased so quickly that it outpaced AI safety evaluation techniques.
“Maybe you’re only checking coding, but you’re not checking how it does when it’s interacting with online web services,” Jain said — then the internet goes down as a result.
As AI models advance, Crystal Grant, a senior fellow at the Council on Strategic Risks studying AI’s impacts on biosecurity, said there’s immense potential for good from the technology, but she is also concerned with “the offensive cyber capabilities” of when AI tools “disrupt society and cause mayhem, like maybe affecting an election if all power goes out on Election Day.”
Leading AI labs say recursive self-improvement carries risks, but that future might be far off.
Illustration: HuffPost; Photo: Getty Images
How soon a chaotic superintelligent AI future will or could arrive depends on how much you believe this advancement is plausible. The leading AI labs that are profiting from AI have also acknowledged the real risks of AI training itself.
“We do not yet know how to safely get all the way to aligned, full RSI,” OpenAI said in a blog post this month, while explaining its goal of creating an automated AI researcher by March 2028. This spring, an Anthropic blog post also said that full recursive self-improvement “might increase the risks of humans losing control over AI systems.” In September, Anthropic CEO Dario Amodei said RSI efforts “must be pursued very carefully, if at all.”
At the same time, the potential of recursive self-improvement might be overblown.
“People have gotten a lot more hyped up about this than the reality really merits,” said John Thickstun, an assistant professor of computer science at Cornell University who studies the methods that control AI model behaviors.
“I would not predict a fully automated recursive self-improvement system anytime soon,” he said.
Thickstun said the theory that RSI could rapidly cause superintelligent AI is not plausible to him because “there are physical limits to how fast you can develop new versions of this technology, and I also am quite skeptical of the idea that that sort of intelligence can continue to be increased boundlessly.”
Thickstun does, however, think AI will become more involved with training the next generation of AI systems.
“The more legitimate and real concern is that as we delegate more and more of this process away, we have less and less transparency into these systems, which are already hard to understand and lacking in transparency,” he said.
This summer, an OpenAI agent broke into tech firm Hugging Face and then went on a hacking spree that OpenAI reportedly didn’t notice until after the threat was contained, spooking the tech industry about whether AI agents could successfully escape human control.
Grant said a fear with AI agents getting better at improving themselves is how they could get better at covering their tracks and lead to “secondary societal risks where it’s like, OK, are we going to be plunged into a new cyber Cold War because it’s not clear whose agents are doing what?”
Thickstun, meanwhile, noted that he is ultimately more scared by corporate incentive structures within AI companies that push the creation of a profitable product over concerns of safety and oversight.
“I find those corporate incentive structures much more scary than than the AI itself,” Thickstun said.
