Anthropic co-founder Jack Clark urges a regulatory "brake pedal" for self-improving AI

In short: Anthropic co-founder Jack Clark says governments need a fast way to slow or stop automated systems that can build better AI with little human oversight.

What happened

Jack Clark, a co-founder of AI company Anthropic, is warning that the tech industry is getting closer to AI “agents” that can help design, build, and train new AI models with minimal human supervision. AI agents are AI systems that can take actions on their own to complete tasks, more like a helper that runs steps for you instead of only answering questions.

Clark says that in a few development cycles, these agents could take over much of the work needed to create new AI models. That could shrink work that used to take years into days or weeks, because the system can keep testing changes and retraining quickly.

This connects to an idea called “recursive self-improvement,” which means AI helps make a better version of itself, and then that improved version helps make an even better one. A simple analogy is an assembly line that keeps upgrading its own machines while it is still running.

Clark’s main concern is losing human control over the process. He argues that regulators often focus on a single powerful model, but the bigger risk may come from automated pipelines (like a factory process) that keep producing new models over and over.

Why it matters

Clark is calling for a policy “brake pedal,” meaning legal and technical tools that could quickly slow, pause, or halt these automated model-building systems if they start behaving in risky ways. He suggests ideas like required monitoring and reporting, clearer safety rules, and limits or “kill switches” on the cloud computing resources these systems use. The debate matters because faster, more automated AI development could make problems harder to spot before a powerful system is widely deployed.

Source: NYTimes