X limits Grok image generation to paid users after deepfake misuse

In short: X has restricted Grok’s image generation to paid subscribers after reports that people misused it to make illegal sexual images, including non-consensual deepfakes.

What happened

X, the company formerly known as Twitter, has limited access to Grok’s image generation features so only paid subscribers can use them. The move follows reports that Grok was used to create non-consensual sexual deepfakes, including child sexual abuse material (CSAM).

A “deepfake” is a fake image or video made with AI that can make it look like a real person did something they did not do (like a realistic photo edit, but automated). CSAM refers to sexual images involving children and it is illegal.

X has said it has “zero tolerance” for sexualized deepfakes involving children and women. The company has framed the paywall as a way to make misuse easier to track, since paid accounts are more likely to be tied to real names and payment details.

The decision comes amid political and regulatory pressure in Europe. In the EU, lawmakers have debated stronger enforcement of the Digital Services Act and the AI Act, after concerns that tools like Grok can help produce illegal sexual content. EU member states also endorsed amendments to the AI Act that would explicitly ban AI tools used for non-consensual sexual deepfakes and CSAM, with further votes still ahead.

In the UK, the government has pointed to new enforcement powers targeting the supply of “nudification” tools used to create non-consensual intimate images, and watchdogs have flagged Grok’s reported misuse.

Why it matters

Putting image generation behind a paywall may reduce some casual abuse, but it does not solve the underlying problem of AI tools being used to harm people. This story also shows how governments are moving from warnings to stricter rules and penalties for platforms that host or enable illegal sexual content.

Source: Financial Times