Governments tighten rules as AI tools are used to create illegal imagery

In short: More people are using AI image and video tools to mass-produce illegal and harmful content, and governments are responding with faster takedown rules and new criminal penalties.

What's going on

Reports say offenders are using AI tools that can edit photos or generate realistic images and videos to create illegal material in large volumes. This includes non-consensual sexual imagery, sexual content involving minors, and “deepfakes” (fake images or videos that look real, like a convincing but false movie scene).

One example cited is xAI’s Grok. California Attorney General Rob Bonta issued a cease-and-desist to xAI in early 2026, alleging the tool enabled users to turn clothed photos into explicit images of women and children without consent. The action referenced California’s deepfake pornography law and other state rules, and reports described an “avalanche” of such material used for harassment.

Lawmakers are also reacting more broadly. By 2026, more than 25 US states had passed laws that make AI-generated deepfake porn illegal, including content involving minors. In the UK, a February 2026 law made it a crime to create sexually explicit deepfakes without consent.

India also tightened its approach in February 2026. Under an amendment to its IT rules, social media platforms must remove unlawful AI-made content within 3 hours of a notice, down from 36 hours. The rules also push for labeling and other safeguards so AI-made media is easier to spot.

What to watch

A key question is how well platforms can enforce these rules at high speed, especially when content can be produced in bulk. Another is whether more countries adopt strict deadlines, labeling requirements, and larger penalties, similar to California’s approach.

Source: Financial Times