Zyphra AI Introduces Zyphra Hierarchical Attention (ZHA): A Path to True Subquadratic Attention

Sparse attention is supposed to make very long context affordable, but most methods still have a component that scans every past token to decide what to read. Zyphra Hierarchical Attention replaces that flat scan with a tree search, so the cost of finding relevant tokens is meant to grow much more slowly than the context.

The practical hook is that it can be added to an existing frozen model. On Qwen3.6-35B-A3B, Zyphra trained only 81,920 parameters using about 4 billion tokens, and reports results close to dense attention on reasoning, coding and long-context retrieval benchmarks up to 256K tokens.

The accuracy results are the encouraging part. On the RULER-13 retrieval test at 256K context, Zyphra's wide tree scored 96.2 against dense attention's 96.0, while a MiniMax-style sparse indexer it trained with ten times the token budget fell to 77.1. On LongBench v2 the deep tree beat dense outright. If those hold up, a cheap retrofit that keeps quality on long documents is genuinely useful.

Now the asterisks. Every number here is self-reported. No optimized kernels were released, and the speed figures come from a modeled roofline for a single layer on an H100, not from measurements. The headline gains of hundreds of times at a billion tokens are projections at a context length almost no one runs today. At 256K, the modeled improvement is closer to about two times. That's still worth having, but it's a different story.

There's also a weak spot the authors do report. On Nemotron-3.5-Lightning, a model without positional encodings, the deep tree dropped 25.3 points below dense on RULER. Models under a billion parameters struggled to convert at all. So this isn't a universal plug-in yet.

Our read: a promising method with honest caveats in the paper and a big gap between paper and production. I'm watching for real kernels and wall-clock numbers from someone other than Zyphra.

Earlier fine-grained sparse methods like DeepSeek Sparse Attention make the final attention sparse but still score every cached token. According to work cited in the paper, that indexing step can take 60 to 90 percent of layer time near a million tokens. ZHA keeps the full KV cache in memory; the tree only decides which entries each query reads, so it saves compute rather than memory.