Sakana AI’s LLM Peer Review System Catches 73% of Core-Claim Errors

Sakana AI has published a peer-reviewed paper in TMLR, titled Beyond Imitation, that changes the scoreboard for AI-assisted peer review. Instead of asking whether an AI reviewer writes reviews that look like human ones, the team planted contradictions in real papers and checked whether the reviewer noticed. Their own system, called Multi-Layered Review, caught 73.43% of errors that hit a paper's core claim when it ran four reviews, compared with 14.81% for the best competing system they tested.

That matters because science is drowning in submissions and reviewers are stretched thin. A tool that reliably flags a broken central claim, at roughly 47 cents a review on off-the-shelf models, is the kind of AI help that could actually raise quality rather than just speed things up.

We like this paper mostly for the question it asks. A lot of AI review tools have been judged on imitation, on how closely they echo what a human reviewer wrote. That rewards a model for sounding like a referee, not for doing the job. Planting a known mistake and seeing if the system finds it is a cleaner, more honest test, and we would like to see it become the default.

The design is also worth noting. Multi-Layered Review uses three agents built on Claude Sonnet 4 and Claude Haiku 3.5: one digests the appendix, an optional one searches prior literature, and a main reviewer reads the paper in three passes, outline first, then detail, then a merged verdict. The paper's own ablation suggests both halves matter. Swapping the model in a simpler baseline more than doubled core-claim detection, and the three-pass structure added roughly another 25 points on a single review. In other words, how a system reads is as important as which model is doing the reading.

Now the humbling part. On 211 real papers that were later withdrawn from arXiv, the system matched the actual problem exactly only 16.11% of the time. Planted contradictions are a useful lab test, but real errors are messier, quieter and often spread across a paper. The authors also report that hidden prompt injection still sways every AI reviewer they tested, including theirs. That is not a footnote. If AI reviewers become part of how conferences triage papers, someone will hide instructions in a PDF to flatter the machine.

There is a subtler gap too. The system's scores lined up reasonably with human scores on ICLR 2025 submissions, but it weighs different things, leaning on validity and experiments while humans give more weight to clarity and novelty. The authors call that a complementary perspective, and we agree that is the right framing. The useful future here is an AI that checks the math and the logic while people judge whether the work is worth publishing.

Where this goes if it scales is the real question. Used as a second reader, a tool like this could catch errors before they reach print. Used as a gatekeeper, it could quietly reward papers written for the machine and punish ones that are unconventional. We are for the first version and wary of the second.

The benchmark covers 1,164 planted contradictions across 257 openly licensed papers from ACL, AISTATS, CVPR and ICML 2025, plus NeurIPS 2024. Errors were graded by how close they sat to a paper's main claim, and an o3-based judge scored the reviews. The authors say the judge may undercount real catches, so the reported numbers could be conservative.