SDSignal Desk

OpenAI hands ChatGPT’s slowdowns to Codex agents, and keeps a human on the merge button

· Updated · Signal Desk newsroom

Analysis of reporting by OpenAI. How we work

Image: OpenAI

OpenAI’s Performance Factory uses Codex agents to investigate regressions, propose fixes and verify recovery. The candid part is the error rate: 70% of root causes graded accurate, and fixes harder still.

Key takeaways

  • The Regression Workflow opened more than 1,700 investigations and root-caused over 600 in just over three months, about 17 a day.
  • A manual review graded 70% of 100 root-cause suggestions accurate; 30% needed refinement or didn’t establish causality.
  • A separate adversarial Codex session tries to disprove each diagnosis and each proposed fix.
  • Engineers still review and merge every fix; OpenAI says it is exploring letting agents ship low-risk changes.
  • Agents run under a scoped service identity with access to code, metrics, logs, traces and profiles.

From copilot to factory

OpenAI’s post describes a progression many engineering teams will recognize. Performance work started with engineers reading traces and profiles by hand, backed by more than 30 internal tools. Then Codex became a partner that could answer why something got slower. Then the team packaged its method into a reusable skill, perf-forensics. Then engineers found themselves pasting alerts into Codex and pasting answers back into Slack, acting as a human relay. The Performance Factory removes the relay.

The system has four parts: a task tracker with workflows defined in configuration files and Markdown prompts, a dispatch loop that assigns ready tasks to agents, an internal API that runs Codex in the cloud inside OpenAI’s network, and subscriptions that wake a task when a pull request gets a review comment or a deployment finishes. Work persists outside any one agent’s context, so agents and engineers can hand off cleanly.

Why it matters now

OpenAI says the pace of agent-assisted development has made traditional performance engineering unsustainable. That is a telling admission: code is being written faster than humans can keep it fast. ChatGPT’s latency is part of the product, which OpenAI has emphasized with releases like a faster ChatGPT Luna and Nvidia-backed ultrafast GPT-6 Astra. Regressions are inevitable when shipping that quickly; the question is how fast they get caught and fixed.

The numbers worth trusting

In just over three months, the Regression Workflow tracked more than 1,700 investigations and root-caused over 600, about 17 automated investigations a day. The figure we find most credible is the one that flatters OpenAI least. In a manual review of 100 root-cause suggestions, 70% were graded accurate and 30% needed more work or didn’t establish causality. Early agents stopped at the first plausible explanation, so OpenAI added an adversarial Codex reviewer that tries to disprove each diagnosis. In one case, that reviewer reopened an investigation after noticing that a file-locating service had recovered while image requests kept failing.

OpenAI is equally frank that fixing is harder than finding. Agent patches were sometimes more complex than needed or introduced new problems, and some fixes, like adding a cache that risks stale data, need an engineer who understands the trade-off. So every pull request still goes through human review, and after deployment an agent checks whether metrics returned to baseline, reopening or splitting the task if not.

The direction of travel

Two plans in the post deserve attention. OpenAI wants agents to find slowdowns proactively from profiles, heap dumps and traces before alerts fire, and it is exploring whether agentic review tools can identify low-risk changes that ship without a human in the loop. That second step is where operational risk rises. A 70% diagnostic accuracy rate is a fine starting point for a system that proposes; it is not a bar we would accept for one that deploys. The evaluation work OpenAI describes, backtesting against past incidents with spoiler-free inputs and grading handoff summaries separately, is the right foundation. Readers should hold the company to publishing how that bar is set.

Who it affects

Platform and SRE teams
A concrete blueprint for agent workflows: explicit states, adversarial review and human sign-off.
ChatGPT users
Faster detection and repair of slowdowns, if the workflow performs as described.
Performance engineers
Less time on triage and evidence gathering; more on trade-offs and approvals.
Codex customers
A showcase for multi-agent pipelines that OpenAI may turn into product features.

What to watch

  1. Whether OpenAI starts letting agents ship low-risk fixes without human review, and what threshold it uses.
  2. Updated accuracy figures for root causes and fixes as the adversarial review matures.
  3. Proactive detection running across services without alerts.
  4. Productization of the factory pattern for outside Codex users.

FAQ

What is the Performance Factory?

An internal OpenAI system that assigns Codex agents to investigate performance regressions, propose fixes, and verify recovery, with engineers reviewing changes.

How accurate are the agents?

In a manual review of 100 root-cause suggestions, OpenAI graded 70% accurate and 30% as needing refinement or not establishing causality.

Do agents deploy fixes on their own?

Not yet. Engineers review and merge fixes; OpenAI says it is exploring autonomous shipping for low-risk changes.

How much work has it done?

More than 1,700 investigations and over 600 root-caused cases in just over three months, according to OpenAI.

Read the original

Continue at the source.

OpenAI

Companies: OpenAI

More on OpenAI

All OpenAI coverage →