Anthropic says it may be close to self-improving AI in IPO filing

In short: Anthropic says it may soon reach a point where using AI to build AI could sharply speed up progress, and it warns this could create safety risks.

What happened

Anthropic, the company behind the Claude chatbot, added new language to its IPO filing warning about “recursive self-improvement,” or RSI. RSI is a scenario where an AI system helps improve the next version of itself, and that makes future improvements come faster and faster (like a worker who builds a better tool, then uses that tool to build an even better one).

In the filing, Anthropic says it already uses AI to speed up its research and development. It also says this increased use could make its models improve faster than the company can build and check the matching safety and security protections.

Anthropic says it has, in the past, not noticed a big jump in its own AI progress until after it happened. It also says its internal estimates suggest it may “imminently” meet its own threshold for “dramatically accelerated progress attributable to AI.” At the same time, it says none of its current models have crossed that threshold yet.

Why it matters

If AI starts meaningfully improving the AI research process, progress could speed up quickly. That could be helpful, but it also raises a basic concern: safety checks might not keep pace. Anthropic says it expects humans to stay involved in building future “frontier” models (the most advanced systems) and it is considering updates to its safety policy to keep people in control.

Source: TechCrunch AI