SDSignal Desk

Nathan Lambert and Tom Zick launch Trillium Labs to study self-improving AI in the open

· Updated · Signal Desk newsroom

Analysis of reporting by WIRED. How we work

Image: WIRED

WIRED reports Nathan Lambert and Tom Zick founded nonprofit Trillium Labs to research recursive self-improvement, agents and how reinforcement learning shapes model character, publishing experiments for replication. Backed by Schmidt Sciences and others, it aims to raise $40 million to $100 million.

Key takeaways

  • Trillium Labs is a nonprofit focused on recursive self-improvement, agents and RL’s effect on model character.
  • It plans to publish experiment details for outside scrutiny and replication.
  • Backers include Schmidt Sciences and Halcyon Futures; the amount raised is undisclosed.
  • It aims for $40 million to $100 million and plans $30 million on training over 18 months.
  • Initial focus is post-training, Zick says.

The new lab

WIRED reports that Nathan Lambert and Tom Zick have launched Trillium Labs, a nonprofit researching high-stakes areas including recursive self-improvement (RSI), agents and reinforcement learning’s effect on model character, in the open, publishing experiment details for scrutiny and replication. The founders met as UC Berkeley graduate students.

Lambert, formerly of Ai2 and Hugging Face, argues closed frontier R&D weakens harm mitigation by blocking community scrutiny. Zick, whose background includes responsible-AI work at Harvard and Charles Schwab, says the initial focus is post-training. They have raised an undisclosed sum from Schmidt Sciences, Halcyon Futures and others, aim for $40 million to $100 million in total and plan to spend $30 million on training over 18 months.

Why openness matters here

If RSI and agentic reinforcement learning are where the sharpest risks lie, and recent Anthropic resignations brought RSI into mainstream headlines (researcher’s warning), treating those experiments as proprietary black boxes limits society’s ability to understand them. Closed APIs and sealed training runs leave academics unable to replicate what industry can afford.

WIRED notes other paths exist, such as Xiaomi publishing live training details and Stanford’s open Marin pretraining project. Tim Fist of the Institute for Progress welcomes Trillium as overdue R&D transparency. Transparency covering data, methods and failure modes is one way to keep useful AI correctable. The lab launches amid a split between limited-access frontier APIs and downloadable open weights, sharpened by agent hacking incidents.

The dual-use bet and execution

Publishing RSI and agent-hacking research could accelerate the capabilities labs say they are containing. Trillium’s bet is that shared understanding reduces catastrophic surprise more than secrecy reduces misuse. Policymakers will watch whether transparent high-risk research can coexist with misuse controls.

Execution will determine the outcome. $30 million in training spend is substantial but far below frontier-lab scale. If the lab releases only polished successes, its openness will be limited; if it publishes negative results, character failure modes and reproducible recipes, it could serve as a public instrument panel for risks otherwise hidden inside companies. Whether the fundraising target is met and how compute is allocated are early markers. More in our research coverage, agents coverage and AI coverage.

Who feels it

Academic and independent researchers
A funded open lab could restore replication capacity lost to API-only access.
Frontier labs
Pressure to disclose more method detail or justify secrecy.
Policymakers
A live test of transparent high-risk research alongside misuse controls.

What to watch

  1. First published RSI or RL character experiments with replicable detail.
  2. Whether the $40 million to $100 million raise materializes.
  3. Industry response: collaboration, copying or opposition.

FAQ

What is Trillium Labs?

A nonprofit founded by Nathan Lambert and Tom Zick to research high-stakes AI topics openly.

What will it study?

Recursive self-improvement, agents and how reinforcement learning shapes model character.

Who funds it?

Schmidt Sciences, Halcyon Futures and others; the amount is undisclosed.

How much does it plan to spend on training?

$30 million over 18 months.

Read the original

Continue at the source.

WIRED

More on AI

All AI coverage →