Discord says AI moderation bug wrongly banned over 8,000 users

In short: Discord says a bug in its automated safety system wrongly banned more than 8,000 users after harmless images were flagged as harmful.

What happened

Discord said its AI moderation system, which helps detect illegal or abusive content, mistakenly banned users over the past two months. The bans were triggered by harmless images, including spreadsheets, chessboards, game textures, and even plain white or gray transparent backgrounds.

The company said the problem started in May. It also said about 200 more users were banned over the weekend before Discord’s team found and fixed the issue. Discord says it is now restoring all affected accounts.

In a support post on X, Discord explained how the system works. It compares uploaded images to databases of known harmful material, similar to how a music app can recognize a song by matching its pattern. Discord said this kind of matching can sometimes produce “false positives,” meaning the system thinks something is harmful when it is not.

Discord also said a human reviewer is supposed to check flagged content before any punishment happens. In this case, a bug caused the system to immediately ban accounts instead.

Why it matters

For many people, a Discord account is not just for chatting. It can be tied to work teams, gaming groups, and long-distance friends. This incident is another reminder that automated moderation can make real mistakes, and those mistakes can lock people out quickly.

Discord said it is working on added safeguards to prevent this from happening again.

Source: TechCrunch AI