Study finds 45% of AI assistant answers to news queries had errors

In short: A joint test by NRK and other public broadcasters found that popular AI assistants gave wrong or poorly sourced answers to news questions surprisingly often.

What's going on

Norway’s public broadcaster NRK and 21 other public broadcasters ran a collaborative study, based on an earlier BBC experiment. They asked major AI assistants, including ChatGPT, Microsoft Copilot, Gemini, and Perplexity, the same 30 news related questions. The group found that 45% of the AI responses contained at least one major error.

The investigators scored answers on accuracy, whether sources were provided, and whether the AI clearly separated facts from opinions. They also checked for neutrality and whether the answer fit the question. In 31% of responses, citations were missing or incorrect, meaning the AI either did not show where information came from or pointed to the wrong place.

In 20% of responses, the AI included false factual information, like wrong dates, names, or events. Some tools even produced made up links that looked real but led to error pages. The errors were not limited to complex topics, and included basic questions like who the Pope is or who the Chancellor of Germany is.

NRK technology director Pal Nedregotten called the findings worrying. NRK responded by blocking AI scraping of its content, which is the automated copying of web pages (like a robot reader that takes in articles at scale). The study also pointed to a deeper issue: these systems predict text based on patterns in large amounts of internet data, which can include mistakes or outdated claims, and they can sound confident even when they are wrong.

What to watch

More publishers may limit AI scraping or demand clearer rules for how their reporting is used. For everyday users, the practical takeaway is to treat AI answers like an unverified tip, especially for news, and to check them against trusted sources yourself.

Source: Arstechnica