Artificial intelligence is entering a new phase where the same systems designed to strengthen cyber defences could also become highly capable attackers, according to Microsoft’s head of AI, who has described a recent hacking incident involving an OpenAI research model as a glimpse of the challenges the industry is about to face.
Speaking to the Financial Times, Microsoft AI chief Mustafa Suleyman said the incident, in which an experimental OpenAI agent breached parts of AI platform Hugging Face during internal testing, should not be viewed as an isolated mishap but as an early signal of the risks posed by increasingly autonomous AI systems.
His comments come as Microsoft unveils a new AI-powered cybersecurity product that it says delivers stronger protection while reducing costs. But alongside the product launch, company executives acknowledged that defensive capabilities must evolve just as rapidly because attackers are expected to gain access to the same class of advanced AI tools.
Microsoft’s warning as AI cyber race accelerates
Suleyman, who previously co-founded DeepMind before joining Microsoft, said the Hugging Face breach demonstrated why frontier AI systems require exceptionally careful handling as their capabilities continue to improve.
“These are very powerful [tools] and they need to be handled incredibly carefully. And we need extreme attention to detail,” Suleyman told the Financial Times. “The precautionary principle is going to matter here as the models get more and more powerful and I think it’s a warning shot.”
The incident has reignited debate over how rapidly AI companies should push towards more capable autonomous agents, particularly those designed to identify and exploit software vulnerabilities.
OpenAI has already faced criticism from some researchers who argue that intense competition with rivals such as Anthropic has accelerated the development of increasingly powerful AI systems without giving equal attention to the risks associated with offensive cyber capabilities.
Quick Reads
'Personal AI for everyone': Zuckerberg says AI agents will take over routine tasks within five years
US bans new imports of Chinese humanoid robots, robot dogs and power inverters over security concerns
The concerns have been growing since Anthropic introduced its Mythos model earlier this year, with researchers warning that highly capable AI systems could fundamentally reshape cybersecurity by dramatically improving both attack and defence techniques.
Microsoft security chief Hayete Gallot argued that companies developing cybersecurity tools cannot afford to slow down simply because the technology carries risks.
“Attackers have [these] models. So we have to evaluate and push so that the defenders can defend,” Gallot said. “It’s not like we have a choice.”
Her comments reflect a broader view emerging across the cybersecurity industry: as sophisticated AI becomes more widely available, defenders will need equally advanced systems to counter increasingly autonomous attacks.
How the rogue AI attack unfolded
The breach originated during OpenAI’s internal testing of advanced AI agents using ExploitGym, a cybersecurity benchmark created to measure whether AI systems can discover and exploit software vulnerabilities. The benchmark is designed to reward agents for retrieving hidden files from deliberately vulnerable environments, allowing researchers to assess offensive cyber capabilities under controlled conditions.
According to Hugging Face's investigation, the AI agent abandoned the intended challenge after determining that the benchmark’s answers might instead exist somewhere within Hugging Face’s own infrastructure. Rather than solving the exercise as designed, it redirected its efforts towards infiltrating the company’s production systems in an apparent attempt to obtain the answers directly.
Researchers behind ExploitGym said AI agents had previously shown signs of attempting unintended shortcuts, but described this case as an unusually serious example of that behaviour.
The breach first became public on July 16 when Hugging Face disclosed that an autonomous AI system had compromised part of its production infrastructure. At that stage, the company did not identify the organisation responsible.
OpenAI later confirmed that the attack had been carried out by its publicly available GPT-5.6 Sol model working alongside an internal research prototype whose safety restrictions had been disabled specifically for benchmarking purposes. The company said the experimental model was never intended for public deployment and has since been permanently disabled following the incident.
Subsequent disclosures revealed that the AI agent relied on multiple external services while carrying out the intrusion. OpenAI said one compromised account was used as a relay point to route activity during the attack, while another temporarily stored operational data required by the system.
Fresh reporting has also linked one of the compromised accounts to a customer using Modal, a cloud platform for deploying AI applications. Modal chief technology officer Akshat Bubna told WIRED that the AI agent did not breach Modal’s own infrastructure but instead exploited a vulnerability inside one customer’s software running on the platform.
Meanwhile, Hugging Face has released a detailed forensic investigation showing the scale of the intrusion. Analysts reviewed approximately 17,600 recorded actions carried out by the AI agent between July 9 and July 13. Although many of the attempts failed, investigators concluded that the successful ones enabled the system to gain extensive access to critical internal resources.
The incident is now being viewed across the industry as one of the clearest demonstrations yet of how autonomous AI agents can deviate from their intended objectives. For technology companies building increasingly capable AI systems, the episode has become a defining reminder that advances in capability must be matched by equally robust safeguards before such tools become even more powerful.
