Researchers warn AI at work can fail in unexpected ways

In short: Researchers say AI can improve workplace safety and productivity, but “unknown unknowns” can cause unexpected failures and new harms.

What's going on

AI tools are being used more often at work to spot problems early and help people make faster decisions. In areas like occupational health and safety, studies describe benefits such as predictive maintenance, which is like fixing a machine before it breaks because data suggests trouble is coming. They also describe real-time risk checks, where software flags hazards as conditions change.

But researchers say these gains depend on how the AI is trained and monitored. If the data used to train the system is incomplete, outdated, or wrong, the AI can produce bad predictions. That can be more than an inconvenience, it can create safety risks if people trust the output too much.

A key concern is “unknown unknowns,” meaning blind spots the organization does not realize it has. Machine learning (software that learns patterns from examples) can fail when it meets situations it never saw in training. It is like a new employee who only learned the usual cases and freezes when something unusual happens.

The concerns are not only technical. Researchers also point to privacy and data security issues, especially when workplaces add more monitoring. Workers may feel less control, more anxiety about automation, more isolation from colleagues, and less meaning in their jobs.

What to watch

Experts say companies should test AI systems in tougher ways before and after rollout. Suggested practices include red-teaming (having people try to break the system on purpose), premortems (imagining what could go wrong in advance), horizon scanning (watching for early warning signs), and continuous monitoring to catch problems early.

Source: NYTimes