AI labs warn advanced models can be misused while still building them

In short: Big AI companies are publicly warning that their most advanced models could be dangerous if misused, but they keep building them anyway.

What's going on

AI labs have been issuing stronger warnings about what their newest models could enable. They point to risks like large-scale hacking, false information that looks real, and even more severe threats such as help with biological harm. Some groups also warn about “loss of control,” which means an AI system does things its creators did not intend.

A recent example is Anthropic’s model called Mythos, also known as Claude Mythos Preview. Anthropic says the model is so good at finding software weaknesses that releasing it widely could cause “widespread disruption.” Think of it like giving a very fast lockpicker a map of every door in a city (the “doors” are apps and computer systems).

Anthropic says Mythos can do in minutes or hours what a skilled security researcher might do in a full day. The company and outside experts also say it can spot serious bugs in operating systems and web browsers, including flaws that may have been missed for decades. Anthropic says it is not releasing Mythos broadly for now, and is instead giving limited access to dozens of companies to help them find and fix weaknesses.

At the same time, the overall industry keeps pushing toward more capable models. The main reasons are intense competition, investor pressure, and the belief that you have to build powerful systems in order to understand how to secure them.

What to watch

A key question is whether governments will require stronger checks before these models are shared or sold. Watch for rules that require safety testing, limits on who can access the most capable models, and clearer accountability when AI tools are used for harm.

Source: NYTimes