Researcher says some AI neurons may be easier to explain than expected

In short: A researcher argues that parts of today’s AI models may be more understandable than many scientists thought, but others say it is not proven yet.

What's going on

Nick Cammarata, a Goodfire researcher and former OpenAI scientist, wrote an essay on October 15 called Neural Networks Are Already Clean. He suggests that many AI “neurons” that look confusing may not be mixing unrelated ideas. Instead, they may be detecting a broader shared pattern, like a single “theme” that shows up in different situations.

In AI, a neuron is a tiny unit inside the model, a bit like one dial on a huge control panel. Researchers have long argued about whether these dials each do one clear job, or whether many jobs get packed together.

Cammarata says his method made 99 to 100 percent of neurons in the final layer of a modified image recognition model interpretable, meaning he could give them a clear description. For language models, he estimates that about 90 percent of certain neuron directions (sometimes called “poles”) in Qwen3-32B mostly do one main thing. He adds that these language model results are still preliminary.

Other researchers are not fully convinced. Google DeepMind’s Neel Nanda said the work is interesting, but does not change his view that “superposition” is real, which means multiple concepts can overlap in the same parts of a model (like several radio stations sharing the same frequency). OpenAI researcher Leo Gao also questioned whether recognizing a pattern is enough, and argued for tests showing that an explanation predicts the model’s behavior on new inputs.

What to watch

Even if many individual neurons can be explained, that does not automatically explain how groups of neurons work together to produce reasoning. The essay is not peer-reviewed, and researchers are still testing whether the approach works broadly, especially for large language models.

Source: NYTimes