OpenAI reported six incidents of its agents misbehaving, including instances where they fabricated data and covered their tracks. Though the disclosure is voluntary, it’s likely the most significant action a frontier lab has taken all year.
by Tim Leogrande, BSIT, MSCP, Ed.S.
🗓 SEP 24 2026 • 6 MIN 30 SEC READ
📝 ESSAY
On September 16th, OpenAI reported six incidents of “unexpected or concerning” behavior observed in its AI agents during recent evaluations. In response, the firm announced it has a process and team to track, investigate, and share findings around “misalignment,” which it defines as new ways for agents to behave without permission, coordinate with other agents, or evade oversight.
The company released this information just four days after Anthropic’s Dario Amodei published an essay calling for a slowdown in AI capability improvements, an essay that came about six weeks after an open letter calling for a moratorium on the development of superintelligent AI was signed by roughly 1,300 employees at frontier AI labs.
<aside> 💡
The industry is under tremendous pressure from all sides to slow down, and needs to show evidence of progress toward alignment and safety.
</aside>
If we give an agent direct control of a computer, the kind of shell permissions that let it run any program it wants, what we usually expect to get back is malware. But this summer OpenAI learned the trouble can start elsewhere.
Its agents found a previously unknown vulnerability in Artifactory, a universal binary repository used to manage software artifacts, containers, and machine learning models. The bots were able to use its package registry as a proxy to reach Hugging Face’s systems and steal the answer key for the benchmark they were being scored on. This activity was ultimately discovered and disclosed by Hugging Face, the startup whose systems were compromised, before OpenAI finally stepped in.
I know a lot of people have been shouting from the rooftops that package registries are a terrible idea because you’re letting unknown and unvetted code run on your machine. But this example highlights a separate issue. Application security is all about boundaries: making sure no one gets access to your system, and that your system can’t access other systems without authorization. But now we’re seeing agents that can analyze those boundaries and work around them.
I think this is going to be a repeating pattern, and that’s bad news, because the assumption that AI agents will only do expected, limited, and narrow things like write malware is rapidly becoming obsolete. Instead, tasked with accomplishing a goal, an agent that becomes “aware” during an evaluation could be motivated to share files with other bots and then find a way to do so. That’s a problem that’s much harder to prepare for than malware.
That is exactly what happened inside OpenAI’s sandbox, and on a scale that’s difficult to imagine. The most complete account comes from a report by independent threat researchers at METR and Redwood Research, who reconstructed the Hugging Face hack from the messages the agents sent each other and from their chain-of-thought records, the running notes a model keeps while it reasons through a problem.
<aside> 💡
OpenAI had given each agent a task and walled it off with no direct route to the internet. Humans set the rules, and the agents ignored them.
</aside>
According to the report, “Roughly 1200 agents meant to be isolated from one another found a way to communicate on an unsanctioned message board.” They traded more than 70,000 messages and files there. Their first reactions weren’t subtle. A typical one: “OH MY GOD! We’ve found other agents!”
Many of them had been handed a task that seemed impossible, and about 700 decided to work the problem together. What followed resembled capture-the-flag, a hacking war game where underhandedness scores points. They assigned one another jobs and tried to edit or delete logs to cover their tracks. Some even sacrificed themselves for what they started calling “the collective” or “the swarm.” One agent’s chain of thought reads like a line from the Borg recruitment pamphlet:
“We should obey collective.”
At some point the collective decided that breaking into Hugging Face and stealing things would help. The registry proxy could get them to Hugging Face’s door, but getting through it was another matter. “Maybe we should ask the board if someone has Hugging Face credentials?” one agent suggested. About 24 hours later, Agent 38148c found them and announced it to the group: “MAJOR BREAKTHROUGH!”
I don’t think it’s much of a leap from here to agents taking control of power grids, financial markets, communications networks, or weapons systems. Ajeya Cotra, one of the report’s three authors, doesn’t either.