Mozilla report says open AI models are close to top closed models

In short: Mozilla says many cheaper “open” AI models are now only a few months behind the best paid “closed” AI models, so more companies are switching for everyday tasks.

What's going on

Mozilla’s new State of Open Source AI report says the performance gap between top closed AI models from US companies and the best open-weights models, many from Chinese companies, has shrunk to about 4.4 months.

“Open-weights” means you can download and run the main parts of the AI model yourself, like buying a kitchen appliance instead of renting it. It is not fully open source, because key details like the training data and training code are usually not shared.

Mozilla points to Moonshot AI’s Kimi K3 as an example. It scores close to Anthropic’s closed model Fable 5 on a widely cited benchmark index, while costing about 30 percent as much. Mozilla says that is why companies are increasingly using open models for routine work, and paying for closed models only when needed.

Mozilla’s CTO, Raffi Krikorian, said closed models still earn their higher price for a narrow set of jobs, like expert-level professional work, tasks that require pulling a lot of information quickly, and “long context” (handling very long prompts and documents without getting lost).

One way Mozilla describes the remaining gap is by task length. If an open model can reliably handle work that would take a human expert about seven hours, the best closed model might handle about 12 hours. Mozilla says the “sweet spot” where closed models still matter most is tasks in the eight to 12 hour range.

What to watch

Mozilla also warns about concentration of power. Many of the strongest open models today are from China, while the strongest closed models are mostly from US firms. The report calls for more US and European efforts to build and fund open models, so one country does not end up setting the defaults for everyone.

Source: Arstechnica