ReplaySSM: Rethinking State Storage for Faster Hybrid-Model Decoding

Hybrid language models that mix state-space layers with a few attention layers—Nemotron 3, Qwen3.5 through Qwen3.8, Kimi-K3, GLM-5.3-Flash—were supposed to decode faster because fixed-size SSM state replaces a growing KV cache. At serving batch sizes, though, the SSM layers themselves become the latency bottleneck up to about 100K tokens of context. Speculative decoding and aggressive state quantization, two tools that speed up Transformers, also get messy on SSMs.

ReplaySSM, joint work from NVIDIA and the Dao AI Lab at Princeton, reframes SSM state as a checkpoint plus a buffer of recent inputs. That change separates output computation from state updates. On Blackwell Ultra B300 GPUs at batch size 256, NVIDIA reports SSM decoding kernels running 1.46× to 1.80× faster, with speculative serving throughput 1.85× to 2.10× higher on 120B-class models. The technique is already in TensorRT-LLM, vLLM, FlashInfer, and production systems like GLM-5.3-Flash and Kimi-K3.

We think this is one of those infrastructure papers that quietly decides who can afford long-context hybrid serving. The insight is almost boring in hindsight: stop rewriting the full state every token; cache the inputs and flush when the buffer fills. That roughly halves the dominant memory traffic and unlocks cleaner rollback for speculative drafts—append draft inputs, move a pointer, no per-token state snapshots.

The second payoff is quantization without the usual error spiral. Conventional SSM state feeds its own rounded values back into the next step, so low precision compounds. ReplaySSM keeps a high-precision checkpoint for recurrence and a low-precision copy for reads. NVIDIA’s numbers say three of four evaluated models stay within about two percentage points of FP32 on GPQA-Diamond when NVFP4 reads sit on a higher-precision recurrence. That is the kind of result that makes 4-bit serving feel serious instead of experimental.

Useful AI wins here if hybrid stacks get cheaper to run at batch. The downside to name is concentration: the wins show up hardest at large batch on top GPUs, and the first production citations are already inside big Chinese open-model serving stacks plus NVIDIA’s own tooling. Smaller shops still benefit when the kernels land in vLLM and TensorRT-LLM, but the early lead accrues to whoever ships hybrids at scale first.

I’m watching whether window size L=16 remains the default sweet spot across new SSM variants, and whether speculative decoding becomes table stakes for Mamba-2 and Gated DeltaNet fleets the way it did for Transformers.

Hybrid models interleave many SSM layers (Mamba-2, Gated DeltaNet, Kimi Delta Attention) with fewer attention layers. Earlier ReplaySSM work from the Dao lab framed the idea; this post and the paper “ReplaySSM: Cache SSM Inputs, Not State” expand the evaluation across Nemotron 3 and Qwen3.5 families from 4B to 120B-class MoE models.