Trillium Labs launches to publish high risk AI research openly

In short: Trillium Labs is a new nonprofit that says it will run and openly publish details of higher risk AI research, including work on self-improving systems and AI agents.

What happened

Trillium Labs launched today, founded by AI researchers Nathan Lambert and Tom Zick. The group says it will publish the details of its experiments so outside scientists can study them and try to reproduce the results.

Many leading AI companies keep their most advanced systems behind closed doors. People can usually only use them through an app or an API (a controlled doorway that lets you send requests to the model, but does not show you how it was made). Lambert argues that this secrecy makes it harder for independent experts to spot problems and suggest fixes.

Trillium Labs plans to work on areas that some researchers consider risky. One is recursive self-improvement, which is the idea of using AI to help design better AI, again and again (like a tool that upgrades itself). Another area is “agents,” meaning AI systems that can take actions toward a goal, not just answer questions (more like a task-doer than a chatbot).

Zick says the lab will initially focus on “post-training,” meaning fine-tuning a large model after it is built. The group will also study reinforcement learning, a training method that uses rewards and penalties (like training a pet with treats and time-outs). This method can make models more capable, but it can also lead to unexpected behavior.

The nonprofit has raised an undisclosed amount from Schmidt Sciences, Halcyon Futures, and others. The founders say they aim to raise $40 million to $100 million total and plan to spend $30 million on training over the next 18 months.

Why it matters

As AI systems get stronger, they can be used to find software weaknesses and help break into systems. Trillium Labs is betting that showing its work in public will make it easier for more researchers to catch risks early, but critics of openness worry that publishing details could also make misuse easier.

Source: Wired