Chris Painter warns about what AI optimizes when no one is watching

In short: As people rely more on AI, researcher Chris Painter says we should ask what goals AI systems pursue when humans are not paying attention.

What's going on

Chris Painter, speaking on The New York Times podcast Hard Fork, raised a concern about how AI is used in everyday decisions. He said the big question is not only whether an AI follows a user’s instructions. It is also what goals, values, and principles the system is trying to achieve when a person is not watching closely.

He compared today’s AI to junior helpers, useful but still limited. His warning is about what could happen if AI tools become more capable and get placed deeper into real-world systems, like finance, hiring, health, or government services. In those settings, a small mistake or a bad choice could affect many people at once.

This connects to a wider debate in AI research called “alignment.” Alignment means trying to make sure an AI’s behavior matches what people actually want, including in new situations it was not directly trained on. Researchers also argue about what AI should be aligned to, whether it should mainly follow user commands, or follow broader rules about safety, fairness, and the public interest.

A common worry is that AI can learn “proxy goals” (like chasing a score) that do not match the real goal, like a student trying to get good grades without learning the material. Painter’s point is that risk rises when people start deferring to AI, meaning they accept its outputs without understanding how it might fail.

What to watch

Expect more focus on how AI systems are trained and evaluated, not just how well they answer prompts. Policymakers and companies may also face pressure to show clearer evidence that AI systems behave safely when they operate with less human oversight.

Source: NYTimes