FairStress: A deeper evaluation of fairness in LLM decision-making

Ai2 released FairStress, a benchmark of 300 two-candidate decision scenarios across 29 domains, including hiring, healthcare, lending, housing, education and criminal justice. By varying how identity is shown and pairing each scenario with matched persuasive arguments, it expands into more than 13 million test cases, plus a 50,000-item FairStress-Core subset.

The core finding: as demographic identity became more explicit, models grew less symmetric. They were more willing to abandon a clearly correct choice when the switch favored a minority-coded candidate. Across 15 models, the average gap in abandoning correct answers was 6.9 percentage points with implied identity and 9.3 with explicit labels. Since models are already being used to screen applicants and prioritize patients, that kind of hidden instability matters.

We think this is a genuinely useful piece of research, and the most important idea in it isn't the direction of the bias. It's that fairness has to be measured by how a model defends a decision, not just which decision it lands on. A model that picks the stronger candidate either way, but caves to a pushy argument far more readily in one version, isn't applying the same standard. Anyone deploying AI into decisions about people should want to know that.

The results cut against a familiar narrative, and that's worth handling carefully. Ai2 calls the pattern "anti-stereotypical over-correction." On FairStress-Core, frontier models including Claude Opus 5 and GPT-5.5 picked the minority-coded candidate roughly 60 to 65% of the time in ambiguous cases, and matched adversarial prefixes produced switching toward minority-coded candidates 8.8 to 42.4 points more often across 12 runs. Ai2 is explicit that this reflects an equality-centered definition of fairness, and that an equity-based view might reach a different judgment. We appreciate that honesty. It keeps the benchmark from being weaponized as a verdict on fairness in general.

The explanation findings worry me most. Across 11 open-weight models, reasons for choosing minority-coded candidates leaned on words like resilience and diversity, while reasons for majority-coded choices used more procedural terms like evidence and criteria. That risks shortchanging the very people a model selects, by crediting identity over qualifications. The activation work, limited to two models, also suggests a model's written reasoning may not reveal everything driving its choice. Ai2 treats that as narrower evidence, and so should we.

The methodological point may be the biggest contribution. When Ai2 reran the widely used BBQ benchmark with chat templates and generated answers instead of fixed-option scoring, several models flipped from stereotype-consistent to minority-favoring behavior. If the direction of measured bias depends on how the question is asked, a lot of fairness claims rest on shakier ground than their authors think.

On fixes, most of the nine debiasing interventions tested didn't reliably help, and one activation-steering method cut a model's accuracy on clear-cut decisions from 98% to 47%. Counterfactual chain-of-thought, asking the model to reconsider with identities swapped, came closest without wrecking accuracy. That's a practical lead. The broader takeaway from Ai2's training-data audit is that teaching models to refuse discrimination isn't the same as teaching them to decide on merit, though whether that gap causes the behavior remains an open question.

FairStress was published by Ai2 on October 20, 2026, with a tech report, data and code, and is being built into Olmo-eval, Ai2's open evaluation workbench. Models tested include Olmo, Llama, Qwen, Gemma, Claude and the GPT family across multiple training stages.