Researcher shows AI needs humans to discover new hacking methods

In short: A security researcher found that AI struggles to invent brand new hacking methods by itself, but becomes much more effective when a human steers it.

What happened

At the Black Hat security conference in Las Vegas, web security researcher James Kettle shared results from months of testing how far “agentic AI” can go in hacking research. Agentic AI means an AI system that can plan steps and carry them out, more like an assistant that tries to do a whole task, not just answer a question.

Kettle’s main question was whether AI can create new, abstract hacking ideas and turn them into real attacks without help. His conclusion was mixed. He said AI is only minimally capable of doing this fully on its own, and it has major limits.

But Kettle also said AI can be a powerful partner when a person provides guidance at key moments. In his work, this human and AI pairing led to a new area of possible web server weakness that he calls “Shared-Parser Confusion.” In simple terms, it is based on the idea that some web servers may use shared code to process both what users send to a site (requests) and what the site sends back (responses). Kettle said requests should be treated as untrusted, but responses are often treated as trusted, so mixing them can open up new ways to attack.

Why it matters

This research suggests the near-term risk is not AI independently “deciding to hack” from scratch. The bigger practical impact is that AI can speed up security work for both defenders and attackers when humans direct it, like a fast research assistant that still needs an experienced editor to spot what is real and important.

Source: Wired