Co-Designing AI Models Using Speculative Decoding for Faster LLM Inference

Speculative decoding is how you make a large model feel faster without changing its answers. A small drafter proposes several tokens; the large target verifies them in one pass and keeps only what it would have said anyway. NVIDIA's latest technical post turns that idea into five guidelines for choosing draft length and draft architecture across the Pareto frontier.

The post is part three of a co-design series. Part one covered hardware-friendly model design. Part two covered attention group size, head dimension, and long context. This one is about the decode loop — the part users feel as lag.

If you run agents, coding assistants, or anything that streams token by token, this is the difference between a model that is theoretically fast and a stack that is actually fast.

The useful claim is that there is no universal draft length D. Verification gets cheaper as you push GEMMs into the compute-bound region, which is why longer drafts help when experts are sparse and KV pressure is high. Attention is the other constraint. Decode attention's arithmetic intensity scales with query heads per KV head; speculation multiplies that. NVIDIA's guideline when attention dominates: set D so that (D+1) times group size is about 128 — the point where current attention kernels saturate. Past that, runtime steps in 128-token tiles and you pay for a half-empty last tile.

At the far right of the curve, batch size collapses and fixed kernel overhead dominates. Then you only increase D while acceptance length still pays for draft cost. A one-layer MTP head run serially and a five-layer DFlash head that emits D tokens in one pass have very different overheads at minimum latency. Higher acceptance length is not the same as higher speedup.

The mechanism table is the product decision. External draft models can win acceptance and still lose on GPUs because you are serving a second LLM. MTP rides in the base checkpoint. EAGLE-3, DFlash, and DSpark are post-trained heads. Suffix and n-gram methods cost almost nothing and only work when the token stream repeats itself. NVIDIA recommends SPEED-Bench for acceptance measurement and TensorRT-LLM for draft overhead. Fine-tunes break learned drafters; MTP has to be kept in the training loop, attached heads need a follow-up, external drafts need distillation.

Signal Desk's read: labs that ship a default drafter without publishing D, acceptance length, and the workload split are selling a slogan. Serving teams should treat this post as a checklist, not a benchmark score.

Acceptance length AL is the number of tokens actually produced per target iteration. It ranges from 1 to D+1 because the target can always emit one ground-truth token plus accepted drafts. SPEED-Bench is NVIDIA's speculative-decoding suite across coding, summarization, and sequence-length splits. Training examples for EAGLE-3, DFlash, and DSpark are in NVIDIA/Model-Optimizer, including a Nemotron 3.5 Lightning path that fine-tunes DSpark then quantizes to FP8 or NVFP4.