SDSignal Desk

Stop counting tokens, start counting what ships

Nov 2, 2026, 5:10 AM · MIT Technology Review

Image: MIT Technology Review

The metric is right: judge AI coding spend by merged work, not tokens. Just remember this case comes from a vendor that sells the on-premises fix.

Why it matters

Agentic coding is expensive in a way chat never was. An agent plans, writes, tests and retries, and the piece puts a single task at 10 to 50 times the tokens of a basic prompt. It points to Uber reportedly exhausting its 2026 AI budget by April after expanding Claude Code use, GitHub Copilot moving to metered credits, and a Gartner prediction that AI coding token costs will pass average developer salaries by 2028.

The common reaction is to ration: caps, seat limits, approvals. The argument here is that rationing measures the wrong thing, and that cost per merged pull request, tracked alongside rework and defects, tells you whether the spending is producing software.

One important disclosure: this is sponsored content produced by HPE, not MIT Technology Review's editorial staff.

From the desk

We think the core idea is good and that engineering leaders should use it whatever infrastructure they buy. Token counts measure activity. Merged pull requests, checked against rework and defects, measure output. A spending cap can't tell a runaway retry loop from a hard bug finally getting fixed, and a per-output metric at least tries to.

The routing advice also holds up: use frontier models for difficult planning and novel reasoning, and send routine generation, tests and refactors to cheaper open-weight models. The piece is careful here, noting that benchmark results don't prove equivalence on a given codebase and that teams should test candidates against their own code. That's the right caution.

Now the caveat. The conclusion, that owning inference capacity is the answer, comes from a company that sells private AI infrastructure. The numbers offered as evidence, token costs down 30 to 50% and power users consuming roughly 450 times more than the lightest, are HPE's own internal results. They may be accurate, but they aren't independent, and lower token costs are not the same as fully loaded cost down once power, operations and depreciation are included. To its credit, the piece says ownership isn't free and that low or unpredictable usage favors APIs.

The harm we'd worry about if this thinking scales is a different one. Cost per merged PR works until teams start optimizing for it. Smaller, more numerous PRs, or merges that pass review but carry hidden debt, can make the number look good while the codebase gets worse. The safeguard is in the piece itself: track rework and defects. We'd make those the headline numbers, not a footnote.

Context

Pricing pressure on AI coding tools has been building as agent features drive far heavier usage than autocomplete or chat. Vendors on every side, including API providers, seat-based tools and hardware sellers, are now competing over how that cost should be measured.

Who feels it

Engineering leaders
Set a baseline for cost per active developer and cost per merged PR now, before changing tools or infrastructure.
Developers
Measuring output rather than usage could ease blunt caps, but it may also bring more scrutiny of rework and defect rates.
Security teams
Running routine generation on owned capacity keeps proprietary code inside the company, which may open up sensitive repositories.

What to watch

  1. Independent, non-vendor data on cost per merged PR across API and owned setups.
  2. More AI coding tools moving from flat pricing to metered credits.
  3. Open-weight coding models measured on real company repositories, not just public benchmarks.
  4. Whether teams report gaming of PR-based metrics.

Read the original

Continue at the source.

MIT Technology Review