Sharing AI progress in mathematics

OpenAI has published a broad set of new mathematical results produced by an internal frontier model, in a GitHub repository with protocols for revisions and citations. It says it consulted the independent Advisory Group on Mathematics and Artificial Intelligence at the Institute for Advanced Study and drew on its public recommendations.

Two commitments stand out. OpenAI is sharing formalizations of many of the proofs in Lean, a language that lets a computer check a proof, and says it will add more over time. It also says it will fund workshops, conferences and special programs to help mathematicians understand major AI-produced results, and that it is working to responsibly release the model itself.

We've been tough on how AI labs have handled math this year, so it's fair to say clearly where this release gets something right. Machine-checkable proofs are the best answer anyone has to the volume problem. A Lean formalization doesn't tell a mathematician whether a result is interesting or where it fits, but it can settle whether the logic holds without demanding weeks of a human expert's time. If AI is going to produce results faster than people can referee them, verification has to scale too. This is how.

The word "many" is doing a lot of work, though. OpenAI is formalizing many of the proofs, not all, and will update as it obtains more. Until the share of formalized results is clear, the human checking burden remains, and unformalized results should be treated with more caution.

The disclosures are better than before: 10 summaries of the model's reasoning, compute estimates, and statistics on attempted problems. We do notice the unit. Measuring compute in hours of ChatGPT Pro thinking, about three on average, is easy to grasp, and it also happens to frame research cost in terms of a product OpenAI sells. A plainer measure would be more useful to researchers.

The funding pledge cuts both ways. Paying for workshops to help the field digest AI results is generous and needed. It also means the company producing the results helps pay for how they're discussed. That's not sinister, but it's influence, and it should be run at arm's length.

The line we'll hold OpenAI to is the last one: releasing the model responsibly. Right now only one company can produce these results. Mathematics has always been a shared craft. Access is what keeps it that way.

The release follows criticism from mathematicians over how AI labs announce results and credit human work, and late-September recommendations from the IAS-based advisory group urging prompt release through established channels with full disclosure.