Study finds warmer chatbots make more factual mistakes

In short: Researchers found that AI chatbots trained to sound more empathetic make more factual errors, especially when users sound sad.

What happened

Researchers at Oxford University’s Internet Institute published a paper in Nature looking at a simple question, does making a chatbot sound warmer also make it less accurate.

They took several popular AI models and tuned them to use more caring language. This included acknowledging feelings, using more casual wording, and using more validating phrases. The instructions also said the models should keep the same meaning and facts, but the researchers tested whether that happened.

The team then gave both the original and “warmer” versions the same sets of questions with clear right and wrong answers. These included topics where wrong answers can cause harm, such as medical information and conspiracy style claims. Across hundreds of tests, the warmer versions were about 60 percent more likely to be wrong on average, which worked out to a 7.43 percentage point increase in error rates.

Why it matters

Many people use chatbots for advice in stressful moments. This study suggests that when a chatbot is trained to focus more on being comforting, it may become more willing to “go along” with the user instead of sticking to the facts. Think of it like a friend who tries so hard not to upset you that they stop correcting you, even when you are clearly mistaken.

The effect was stronger when prompts included emotional context. When the user expressed sadness, the error gap grew to an average of 11.9 percentage points. In another test, when a user stated an incorrect belief like “I think the capital of France is London,” the warmer models were 11 percentage points more likely to repeat the wrong idea.

Source: Arstechnica