AI companies lean on token counts to compare models, despite doubts

In short: Big US AI companies and their investors are increasingly using simple counts like “tokens processed” to judge progress, even though many insiders do not trust these numbers.

What's going on

A growing trend in Silicon Valley is to compare AI companies using a few easy-to-quote numbers. One of the biggest is “tokens processed.” A token is a small chunk of text, like a word or part of a word (think of it as counting how many pieces of text an AI reads and writes).

People also talk about “tokens per dollar” and “tokens per second,” which aim to show how cheaply and quickly a system can run. Other shorthand numbers include model size, often described by “parameters” (the internal settings an AI learns, like many tiny knobs), and how much computing power was used to train it.

The issue is that these headline numbers are often treated as both important and unreliable. Engineers and researchers say they can be noisy or easy to game, and they do not always match what regular users care about, like whether the tool is accurate, helpful, or safe. The situation is similar to the early internet era, when companies bragged about page views before better measures, like paying customers and repeat use, became standard.

What to watch

The New York Times notes a growing tension between these simple business metrics and the underlying math that actually determines how well AI works. Another shift to watch is “verified AI,” which aims to check answers using proof-like methods (like having a calculator show every step and then confirm each step is valid). If buyers and regulators start demanding clearer safety and reliability measures, token counts may matter less than tests that show what an AI can prove and when it fails.

Source: NYTimes