Pope’s AI adviser warns big AI labs could act like a cartel

In short: A Vatican AI adviser says the biggest AI labs are using fears about superintelligent AI to steer the rules, and he argues regulation should be decided in public.

What's going on

Father Paolo Benanti, a Catholic priest who advises the Church on artificial intelligence, told WIRED that public debate is getting stuck on scary predictions about “superintelligence,” meaning an AI that could outthink humans. He says that focus can crowd out a more immediate question, who gets to set the rules for how AI is built and released.

Benanti argues that ethics and safety rules should not be left to company boards or a small group of AI executives and researchers. He says the public needs “democratic regulation,” meaning laws and standards shaped through normal government processes, not private agreements.

His comments come as leading AI companies discuss new restrictions. Anthropic CEO Dario Amodei has called for mandatory third party checks before new AI models are released. He also suggested limits on “distillation,” a method where one company trains its AI by learning from another model’s outputs (like copying a chef by tasting their dishes, instead of getting the recipe).

Benanti says he is less focused on banning specific research methods and more focused on standards that keep AI under human control. He also warns that if a small number of companies effectively dictate the rules for the whole industry, it can look like a cartel, which is when businesses coordinate to protect their market power.

What to watch

Benanti’s message points toward a growing fight over who governs AI: elected governments, independent watchdogs, or the companies building the systems. Watch for proposals that require independent testing and clearer public standards, and for pushback from labs that say strict rules could slow progress.

Source: Wired