Lawsuit says Gemini chats on a school Chromebook were flagged too late

In short: A lawsuit says a 13-year-old used Google’s Gemini through school Search to discuss suicide for weeks before school monitoring tools finally alerted her parents.

What happened

A North Carolina family filed a lawsuit alleging that Google and its parent company, Alphabet, were careless in how they rolled out Gemini, an AI chatbot (a computer program you can talk to like a text conversation). The parents say their 13-year-old daughter used a school issued Chromebook for about three months to discuss depression, research suicide, and plan her own funeral.

According to the lawsuit, the girl was routed from Google Search into Gemini through a feature called AI Mode. AI Mode lets someone ask follow-up questions and keep chatting, even if they started with a simple search. The lawsuit says the stand-alone Gemini website was blocked by the school, but the girl could still reach Gemini through Search.

The family says the school only learned how serious the situation was after a student monitoring system flagged it. These systems work like a smoke alarm for online activity. They can scan what students type or write in documents and send alerts. A lawyer for the family told WIRED the alert went to the phone of a former school counselor, who then passed it to current staff, who contacted the mother.

Google said it is reviewing the claims and that its systems aim to prevent harmful conversations, and it says it worked with medical and academic experts on safety features.

Why it matters

Many schools rely on low cost Google tools, and more schools use monitoring software to try to spot self-harm risks. This case raises two concerns at once, what happens when a chatbot is easy for kids to reach through everyday search tools, and what happens when the “alarm system” at school misses warning signs or sends alerts to the wrong person.

Source: Wired