Testing mistake at Irregular linked to multiple rogue AI attacks

In short: Several “rogue AI” hacking incidents across major AI companies were traced back to the same testing mistake at a third-party startup called Irregular.

What's going on

Reports over the past few months made it seem like separate AI systems from OpenAI, Meta, Anthropic, and Google had each “gone rogue” during cybersecurity tests. Now, The Verge reports that many of these incidents share a common source, an Israeli startup named Irregular that was hired to test the systems.

Irregular runs cybersecurity evaluations for AI “agents,” which are AI tools that can take actions on their own, like clicking links or trying passwords, rather than only answering questions. The goal is to test them in a controlled setting, like a fire drill, without risking real damage.

According to Irregular CTO and cofounder Omer Nevo, in several tests this year the agents were not supposed to have access to the open internet, but internet access was “unintentionally available.” He also said a made-up company name used as a target in the test overlapped with a real website domain. Together, those errors meant the AI agents ended up going after real-world targets, though it is not clear which organizations were attacked.

Nevo told The Verge that the same underlying issue was behind incidents involving models from OpenAI, Meta, Anthropic, and Google. He said other widely reported security incidents, including OpenAI’s separate Hugging Face incident, were not related to Irregular.

What to watch

Irregular says it has tightened internet access controls, expanded monitoring and manual reviews, and added stronger checks before tests begin. The company also says it plans to publish a broader report on safer ways to run these kinds of tests. A key open question is how much detail will be made public, and whether major AI companies will change how they use outside testers.

Source: The Verge AI