SDSignal Desk

METR’s new time-horizon chart finds the limit of AI progress: judgment

· Updated · Signal Desk newsroom

Analysis of reporting by MIT Technology Review. How we work

Image: MIT Technology Review

The updated chart still shows exponential gains on checkable work, but adds open-ended tasks where models fall short of people. The gap is real, the measurement is shaky, and both facts matter for claims about automated AI research.

Key takeaways

  • METR’s updated time-horizon plot adds longer tasks and open-ended, judgment-heavy tasks to its original suite.
  • The original plot showed the length of tasks frontier models could complete doubling roughly every four months; verifiable-task progress shows no sign of slowing.
  • No AI-designed evaluation was judged useful enough for METR’s research, and models fell short of humans at scoring philosophical arguments.
  • Newer models such as Opus 5.5 now reach “borderline above the bar” on evaluation design, up from clearly below a year ago.
  • Human completion times are now mostly estimates, and one person rated most of the philosophy task, which limits confidence.

Why this chart carries so much weight

METR’s time-horizon plot became famous for a simple-looking line with a complicated meaning. The AI evaluation nonprofit gives models tasks that humans have also completed, scores each task by how long the humans took, and rates models by the length of task they can finish. The headline finding was that the length of tasks frontier models could complete doubled roughly every four months. It was widely shared and, as MIT Technology Review notes, widely misread, often as a forecast of when AI replaces whole jobs.

The new version matters more because of timing. Earlier this month a heavily redacted METR report suggested AI might be doubling the pace of AI progress at Anthropic, and fed fears about recursive self-improvement. Any update to the field’s most cited capability chart now gets read as evidence for or against machines building their successors. That makes it especially important to be precise about what the update shows.

What changed: the fuzzy tasks arrive

The update adds tasks that take humans far longer, and, crucially, tasks without an automatic answer key. That distinction is the whole story. Reinforcement learning works best where success can be checked by machine, which is why models have become strong at coding and math. Choosing a research direction or judging an argument offers no such clean signal, so it is harder both to train and to measure.

On those new tasks, models fall short. METR asked models to design evaluations it could use in its own research; none of the AI-designed evaluations were good enough for staff to consider useful, even though top models now outperform almost every human at software engineering. On a task scoring philosophical arguments, models also trailed people, though METR staff disagree on how relevant that task is. The encouraging note is the trend: a year ago every model sat clearly below the bar on evaluation design, and some newer models such as Opus 5.5 now reach borderline above it. That lines up with what OpenAI described this week in its own report on a jagged research frontier: strong gains on checkable work, weaker results on the messier tasks real employees assign.

The honest caveat is the measurement itself

Here is where caution is warranted. In the original plot, METR timed real people doing each task. In the new report, most completion times are estimates, because paying experts to spend hundreds of hours on a single task is impractical. For the philosophy task, one person rated most of the arguments, so models are effectively scored on agreement with a single individual. METR includes only a handful of these open-ended tasks, and its own staff say trends here are unlikely to ever be as clean as for well-specified work.

We read this as a welcome step and an incomplete one. It is right to stop treating a coding-heavy benchmark as a proxy for general research ability, and Sayash Kapoor’s view that research taste may be the real bottleneck deserves attention. But the people warning that skills learned on verifiable tasks will spill over into judgment are not refuted either; the data is too thin to settle it. The responsible takeaway is narrower than either camp wants: verifiable capability is still compounding fast, judgment is improving from a low base, and nobody yet has a reliable way to measure how quickly that gap closes.

Who should pay attention

AI labs
Claims about automating research now face a public yardstick that separates checkable work from judgment.
Policymakers
Timelines built on the old doubling line overstate certainty about open-ended capability.
Enterprises
Agents remain far stronger on well-specified tasks than on decisions that need taste or judgment.
Evaluation researchers
Better methods for scoring hard-to-verify work are now a priority.

What to watch

  1. Whether METR adds more open-ended tasks and more human raters in future updates.
  2. How quickly models move from borderline to clearly above the bar on evaluation design.
  3. Further disclosures on AI-accelerated research at Anthropic and other labs.
  4. Independent replications that time humans directly instead of estimating.

FAQ

What is METR’s time-horizon plot?

A chart rating AI models by the length of tasks they can complete, measured by how long the same tasks take humans.

Is AI progress slowing?

Not on verifiable tasks like coding, according to the report. Models still lag humans on open-ended, judgment-heavy tasks, though they are improving.

Why are the new results less certain?

Most human completion times are estimates, there are few open-ended tasks, and one task relied largely on a single human rater.

Read the original

Continue at the source.

MIT Technology Review

More on AI

All AI coverage →