Researchers warn AI chatbots can degrade if trained on AI text

In short: Researchers say future chatbots could get worse if they are trained mainly on AI-generated text instead of fresh human writing.

What's going on

Researchers have been studying a risk called “model collapse.” It means an AI system can slowly lose quality when it is trained on text made by other AI systems. Think of it like making a photocopy of a photocopy again and again, where the details fade over time.

In experiments, when teams repeatedly trained new models using mostly synthetic data, meaning AI-generated data, the results became less varied and sometimes less accurate. Over multiple rounds, outputs could drift toward distorted patterns or even nonsense. Researchers describe this as a feedback loop, because AI text gets fed back into the next round of training, which can make the next model lean even more on the same AI-made patterns.

This matters for chatbots such as Anthropic’s Claude because today’s top systems were first trained on large amounts of human-written text. Human writing includes unusual phrasing, mistakes, and niche topics, and that helps chatbots handle the messy variety of real life. If the internet fills up with AI-written pages, future training collections could include more synthetic text and less authentic material.

What to watch

The evidence supports a risk for future training, not proof that earlier benefits from book-based training have already been erased or that collapse is guaranteed. Some research summaries suggest that mixing in even a small amount of real-world human data can reduce the collapse effect in certain setups. The key question is whether AI companies can keep getting enough high-quality human text as more online content is written by machines.

Source: NYTimes