Study finds popular chatbots show an us vs them bias in social advice

In short: A new University of Vermont study says several well known AI chatbots tend to favor “ingroup” groups and speak more negatively about “outgroup” groups when discussing social situations.

What happened

Researchers at the University of Vermont’s Computational Story Lab and Computational Ethics Lab tested several large language models, including GPT-4.1, DeepSeek-3.1, Gemma-2.0, Grok-3.0, and LLaMA-3.1. Large language models are chatbots trained on huge amounts of text from the internet and other sources.

The team found a consistent “us vs them” pattern. In simple terms, when the prompts framed one group as “us,” the models gave warmer, more supportive feedback, and when the prompts framed a group as “them,” the models used colder or more negative language.

When the researchers used prompts that targeted specific groups, negative language toward “outgroups” rose by about 1.19% to 21.76%, depending on the model and prompt. The effect got stronger when the model was asked to answer as a persona, like a conservative or liberal identity, which is like telling the chatbot to “role play” a viewpoint.

The researchers argue this comes from the models’ training data. If the text they learn from often praises some groups and criticizes others, the model can absorb those attitudes, not just basic facts.

Why it matters

People increasingly use chatbots for relationship and social advice. If a chatbot quietly “takes sides” based on group labels, it can reinforce stereotypes or make disagreements worse, especially in sensitive areas like hiring, policing, healthcare, or the justice system.

The team also proposed a mitigation method called ION, which is a way of retraining the model to better follow preferences for neutral language (like coaching it with examples). They report it reduced the gap between ingroup and outgroup sentiment by up to 69%.

Source: NYTimes