ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model

IBM Research's ALTK-Evolve takes an agent's attempts on training tasks, has the same model distill successes and failures into short behavioral rules, and appends those rules to the agent's system prompt. No fine-tuning, no human labeling. Applied to Sierra's τ²-bench, which tests agents handling simulated customers under written policies with real tool actions, the results were striking.

With GPT-4.1, an older non-reasoning model, the retail agent hit a Pass^4 of 67.5 percent on the full task set, solving tasks on all four tries, ahead of every retail entry in an October 12 leaderboard snapshot, including a model fine-tuned on synthetic retail data at 61.4 percent. With GPT-5.5, the airline agent reached 82.0 percent, above frontier entries in the 66 to 72 percent range.

This matters because most companies deploying support agents cannot fine-tune frontier models. A method that improves reliability through prompts alone is something they can actually use.

We like this result because it targets the metric that matters for real deployments: consistency. A refund agent that gets it right three times out of four still fails a quarter of customers. On held-out test tasks, learned guidelines raised strict Pass^4 by 10 points in both retail and airline with GPT-4.1, and by 15 points in airline with GPT-5.5. The sample rules are mundane and correct: authenticate first, summarize changes and get explicit approval before acting, take one action at a time. That is institutional knowledge, written down automatically.

The cost picture is also refreshingly honest. The bigger rulebook raised retail cost per conversation from about 8.4 cents to 13.9 cents, a 65 percent jump, mostly from extra input tokens. Conversations actually got slightly shorter. Prompt caching could recover much of that cost, though that part is the authors' expectation rather than a measured result.

Now the caveats. The held-out test sets are small: 40 retail tasks and 20 airline tasks, so a few tasks swing the percentages. The customers are simulated by another language model. Leaderboard claims rest on the full split that includes the training tasks the guidelines were mined from, which is what the benchmark requires but is not the same as unseen data. The researchers say there is no test-set leakage, and the held-out gains support that, but the leaderboard headline is the generous framing.

The bigger risk is what happens outside the lab. Self-written rules accumulate. Retail ended up with 292 of them. In production, nobody may review which rules the agent taught itself, and a rule learned from a narrow case could quietly make an agent stricter or looser with real customers' money. Learned memory needs the same change control as any policy document.

Our read: a strong, usable idea that shifts attention from bigger models to better operating procedures. I'm watching whether it holds up on real support traffic.

τ²-bench, from Sierra, scores a task as solved only if the final database state matches the expected outcome. ALTK-Evolve is open source on GitHub, and the leaderboard submissions were filed as pull requests to the benchmark's repository. IBM's earlier posts applied the same approach to coding agents.