SDSignal Desk

Hugging Face’s harness adapters show that tuning a model to its agent loop pays twice

· Updated · Signal Desk newsroom

Analysis of reporting by Hugging Face. How we work

Image: Hugging Face

A small RL-trained LoRA lifted Qwen3.8-27B’s coding-agent scores and roughly halved its tool calls inside Pi, OpenCode and Hermes Agent. The open recipe makes the fit between model and harness something teams can tune themselves.

Key takeaways

  • Hugging Face trained one LoRA adapter per agent harness (Pi, OpenCode, Hermes Agent) for a frozen Qwen3.8-27B, using reinforcement learning inside the unmodified harness.
  • Under Pi, the 51 hardest SWE-bench Pro tasks went from 39.2% to 64.7% pass@1; on Terminal-Bench 2.1, Pi gained 16.3 points.
  • On tasks both models solved, the adapters used 53–72% fewer tool calls, so the gain came with lower cost.
  • A merged “super-adapter” beats the base model under all three harnesses, but with thinking off it keeps less of each specialist’s gain.
  • The authors flag an open question: the model may be leaking reasoning-like text into non-thinking mode.

The harness is half the product

Agent performance used to be treated as a property of the model. Hugging Face’s new post argues it’s a property of the pair: the model plus the harness, the loop of prompts, tools and context rules wrapped around it. General-purpose models are tuned for general performance. That leaves accuracy and tokens on the table in any specific harness running any specific kind of task. The fix it proposes is cheap: freeze the base weights and train a small LoRA adapter with RL inside the harness exactly as it ships.

That last detail is the real contribution. Most RL frameworks retrain a re-implementation of the agent loop. Here a proxy between the harness and vLLM records the exact tokens the real harness produced, and the trainer learns from those. The team used TRL’s AsyncGRPOTrainer with OpenEnv and Harbor, ran everything as Hub Jobs, and published adapters, GGUFs, tasks and evaluation trajectories. It fits a trend we covered when AWS open-sourced Strands Harness: the loop around the model is now a place to compete.

The numbers, and what they buy

The headline gains are large for an adapter. On the 51 hardest SWE-bench Pro tasks, the Pi adapter took Qwen3.8-27B from 39.2% to 64.7%. Hermes went from 29.4% to 43.1% and OpenCode from 35.3% to 51.0%. On Terminal-Bench 2.1, Pi gained 16.3 points and Hermes 14.1. OpenCode gained 6.0 overall and 13.6 on the tasks that weren’t used to pick its checkpoint. The efficiency story may matter more to anyone paying the bill. On tasks both models solved, the adapters made 56–72% fewer tool calls on SWE-bench Pro and 53–70% fewer on Terminal-Bench. Most of the gain, the authors say, comes from reasoning less, calling fewer tools and getting it right the first time.

The authors are careful about comparisons. Each harness ran with default settings: Pi drops earlier reasoning turns while OpenCode and Hermes keep them. So scores can’t be compared across harnesses, only within one. Test sets were small (89 and 51 tasks), and the merge weight for the super-adapter was picked on Terminal-Bench, so those numbers are, in the authors’ words, slightly optimistic.

Data hygiene is the quiet hero

The most transferable part may be how the tasks were built. The team screened about 59,000 Harbor tasks on the Hub down to training-only sources, removed the 253 target-benchmark tasks and any repository those benchmarks use, and validated each task. Validation checked that the reference wins, doing nothing scores lower, canaries can’t beat doing nothing, and replays are deterministic. 858 of 1,127 Hub candidates passed. Those checks caught a generated task that accepted incomplete code and a Terminal-Bench verifier that accepted known-wrong answers. Bad verifiers teach bad behavior. Seeing that written down, with a rejected split preserved, is more useful than another leaderboard claim.

There is a caveat worth keeping. After release, the authors found the adapted model writes short reasoning-like text (“wait… let me see”) before tool calls in non-thinking mode, a kind of “reasoning leakage.” They argue the results still hold under a fixed token budget, and the super-adapter with thinking on was the strongest model they tested. Still, it means part of what the adapter learned may be how to slip extra thinking into a mode meant to skip it. Anyone using these in production should measure latency and tokens on their own workload rather than trust the averages.

Who gains

Teams running open models in coding agents
A recipe to raise accuracy and cut token spend for their own harness and task mix, without retraining the base model.
Harness builders
Adapters make the harness part of the optimization target. Expect per-harness adapters to ship alongside popular tools.
Closed-model vendors
A 27B open model with a cheap adapter closing much of the gap to its own thinking mode puts pressure on per-token pricing for routine agent work.
Benchmark maintainers
Validation caught a Terminal-Bench verifier accepting wrong answers, a reminder to audit graders.

What to watch

  1. Reproductions on other base models and on proprietary codebases, which the authors invite.
  2. Follow-up work on pure thinking models and hybrid thinking training that the team says is next.
  3. Whether the reasoning-leakage behavior shows up as higher latency in real deployments.
  4. Harness projects such as Pi, OpenCode and Hermes Agent shipping official adapters.
  5. Cross-harness transfer: the Pi adapter already matched OpenCode’s own specialist on Terminal-Bench.

FAQ

What is a harness adapter?

A small LoRA weight update, trained with reinforcement learning inside a specific agent harness, that sits on top of a frozen base model to improve accuracy and efficiency for that harness.

Which model and harnesses were tested?

Qwen3.8-27B under Pi, OpenCode and Hermes Agent, evaluated on Terminal-Bench 2.1 and the 51 hardest SWE-bench Pro tasks.

Does one adapter work across harnesses?

Partly. A merged super-adapter beat the base model under all three, and the Pi adapter transferred well to OpenCode, but specialists usually did best with thinking off.

Can others reproduce it?

Hugging Face published the adapters, tasks, trajectories and recipe, and invites teams to train adapters on their own tasks.

Read the original

Continue at the source.

Hugging Face

More on AI

All AI coverage →