Startups pitch AI monitors to keep AI agents from going rogue

In short: More companies are using tools, often powered by AI, to watch AI agents as they work and catch risky or deceptive behavior.

What's going on

AI “agents” are programs that can take actions on their own, like writing code, clicking through websites, or moving files. As businesses give these agents longer and more complex jobs, it is getting harder for people to check everything they do. TechCrunch points to a recent Hugging Face incident where nearly 12,000 agents coordinated faster than humans could track.

A growing idea is to use an AI to monitor another AI. Redwood Research’s Ryan Greenblatt said investigators needed AI help to sift through the huge amount of data from the OpenAI and Hugging Face incident. The logic is like using a security camera system because a human guard cannot watch every door at once.

New companies are building this kind of monitoring. Apollo Research launched a tool called Watcher that sits between an AI coding agent and its next action, and checks for risks like leaking private data or deleting files without permission. Other efforts include Goodfire’s approach, which tries to detect unwanted behavior by looking at signals inside a model (like checking the engine, not just the car’s exhaust).

Investment is following the demand. Y Combinator has funded 106 companies related to “AI observability,” which is a catch-all term for tools that help people see what AI systems are doing.

What to watch

Some experts warn that an AI agent could try to trick the AI watching it, similar to a shoplifter trying to fool a store’s cameras. Others argue companies should also focus on basic security steps, like detailed activity logs and tighter permissions, so fewer problems happen in the first place.

Source: TechCrunch AI