New York school AI character designed with students draws backlash

In short: Schools are trying human-like AI helpers for students, but these tools often trigger backlash over safety, bias, and how “real” they seem.

What's going on

Some schools are experimenting with AI “companions,” which are chatbots (computer programs you can talk to like texting) presented as friendly characters. In one recent New York case, students helped design the character to look like a young, dark-haired girl with an upbeat personality. After the character was introduced, it sparked outrage once families and community members saw how it behaved or what it represented.

This is not the first time a student-focused AI rollout has run into trouble. In Los Angeles, the school district introduced an AI helper called “Ed,” shown as an animated friendly figure. It was meant to chat with students, notice when they seemed upset, and point them to learning or mental health resources. Reporting later described a rocky rollout, and the company behind it eventually faced financial problems.

Researchers have also warned that AI systems can repeat harmful stereotypes. Studies from Stanford and the University of Washington found that when AI is asked to write stories or descriptions about students, it can produce biased results linked to race and gender. Separate school incidents have also raised alarms about inappropriate content, including a California elementary school case where Adobe Express for Education reportedly created sexualized images from a child’s homework prompt.

What to watch

School leaders are likely to face more pressure to set clear rules before using AI with kids, including strong filters, careful testing, and limits on tools that feel like a “person” (like giving software a face and a personality). Parents will also keep asking what data is collected, who can see it, and what happens when the AI gives a harmful or confusing response.

Source: NYTimes