3 Ways to Boost MoE Inference on NVIDIA DGX Spark

Running large mixture-of-experts models on a desktop-class machine is one of the more useful directions in AI right now. It keeps data local, cuts cloud bills and puts capable models in the hands of small teams. This walkthrough takes Inclusion AI's Ling-3.0-flash, a 124-billion-parameter model with 5.1 billion active per token, and shows how to make it decode faster on DGX Spark.

The practical payoff: on the full GSM8K test set, three stacked optimizations raised decode throughput 42.5% and cut mean latency 24.5%, with no statistically detectable accuracy change.

We like this post because it does something vendor blogs rarely do: it tells you where its own big number stops being true. The synthetic run shows a 185.1% throughput gain once speculative decoding is added. On real GSM8K questions, with normal sampling and shorter answers, the gain is 42.5%. NVIDIA spells out why: synthetic output is repetitive, so draft tokens get accepted more often, 3.90 per round versus 2.68 on the real workload. That candor deserves credit.

The diagnosis is the lesson. At batch size one, decoding was memory-bound; reading weights dominated. So the fixes target weight traffic. A runtime-compiled kernel library called Humming specializes the expert computation to the model's actual shapes, an FP8 output layer halves the read volume of a huge vocabulary projection, and the model's built-in NEXTN layer drafts multiple tokens so fewer decode steps are needed. Two of these make each step cheaper; one makes there be fewer steps. That's why they stack.

Except when they don't. The most important paragraph is the warning that combining FP8 output with speculative decoding can silently fall back. The draft reads FP8 weights as BF16, acceptance collapses to one token, and the result is slower than doing nothing. No error, just worse performance. That's the downside of an optimization culture built on flags and environment variables: the stack gets fragile, and a quiet misconfiguration can cost more than the tuning ever gained.

I'm watching whether this kind of tuning becomes packaged and default, or stays artisanal. If local inference is going to matter beyond enthusiasts, the speedups need to arrive validated at startup, not discovered by reading a blog post.

Ling-3.0-flash has 42 layers mixing linear-attention and latent-attention designs, routes each token to eight of 512 experts plus a shared expert, and offers a native 256K-token context window expandable to 1M. The baseline used FlashInfer MXFP4 kernels and an FP32 output head, following the official cookbook.