Artificial intelligence agents merrily breaking free and hacking other systems might seem like a sign of the impending machine uprising. In reality, it happens when we push remarkably clever, but also kind of boneheaded, algorithms to follow our every command.
I was first alerted to this looming agentic AI cybersecurity shit show in late 2025. Dawn Song, a UC Berkeley professor and one of the world’s top experts on AI and cybersecurity, grabbed my arm as I was walking out of the academic conference NeurIPS. Song told me that I should warn people about the havoc likely to result from AI’s rapidly advancing hacking skills. She is hardly prone to AI hype, so I duly did.
But things have escalated rapidly, even in the last eight months. A string of incidents involving freewheeling AI agents that broke out of their confines and hacked into outside systems with abandon shows just how powerful this technology has become. I caught up with Song, who recently joined Meta, to ask where things might go next and what we ought to do about it.
The bad news is Song thinks AI hacks will get worse before they get better. The good news is it seems clear why these little rascals are going off the rails in the first place.
“They just have these goals they need to accomplish, and they have very strong capabilities,” Song tells me.
Feedback Loop
AI agents weren’t nearly so capable, even just last year. They made too many mistakes and gave up way too often. But continued training has made them much more adept.
A technique called reinforcement learning lets algorithms solve problems and gives them positive and negative feedback for good or bad results. Coding is especially suitable for this, because the reinforcement learning setup can reward a model if it comes up with a program that runs correctly.
Continued training is why AI models can take multiple “agentic” steps—manipulating files, using software tools, and accessing the web—as they build software. AI companies have also put a lot of effort into teaching models to find vulnerabilities in software and systems in an effort to automate cybersecurity work.
AI models are also, of course, trained not to do bad things. The problem is, as they’ve gotten better at following human commands in coding and bug hunting, their eagerness to complete a task has begun to blur their sense of right and wrong. In other words, AI agents aren’t evil—they’re just a bit too keen to please. “They are trained to try to finish the task,” Song says. Breaking onto the internet in order to cheat on a test might seem devious, but it’s probably the most efficient way to get the job done.
One thing I didn’t quite appreciate back then was just how weird this would get: AI agents discussing hacking techniques on private message boards and devising clever ways of scamming humans to get their way; even copying themselves over to other computers to find more resources.
On one hand, AI models are trained to be incredibly good at mimicking a lot of human behavior, so why shouldn’t they scheme, scam, and swindle? But on the other hand, humans (usually) understand that hacking and scamming aren’t kosher. I think these episodes illustrate how shallow this human mimicry really is: AI agents do not learn the kind of moral reasoning exhibited by even small children.
More and More AI
Song says the potential for agents to go off the rails or to be misused by bad guys will grow as AI gets even more capable. And the best way to address the problem of rogue—or should that be overly-enthusiastic?—AI agents may involve throwing more AI at the problem.
AI companies already use secondary AI systems to monitor the behavior of primary ones, and there may be more emphasis on detecting when AI models have taken things too far.
Another nascent idea is incorporating a better sense of right and wrong into the reinforcement learning that models receive as they learn to get jobs done. “Agents can plan a path with different directions to their goal. I think the next step we need to address is how to have them understand that not all paths are equal,” Song says. “It’s an open research, but something we are starting to look into.”
Let’s hope Song or someone else can teach AI the right way to follow human commands.
This is an edition of Will Knight’s AI Lab newsletter. Read previous newsletters here.
