Dense vs. MoE Models: Active Parameters, Throughput, and When to Choose Each
Sep 15, 2026, 10:00 AM · NVIDIA Developer
NVIDIA’s Sophia Abbassi explains MoE versus dense tradeoffs with Nemotron 3.5 Lightning—30B total, 3B active—where memory stays high but compute per token drops.
Why it matters
An NVIDIA Developer Blog walks through Mixture-of-Experts versus dense models: MoE loads all experts into VRAM but activates a subset per token, decoupling memory cost from compute cost. Nemotron 3.5 Lightning activates 3B of 30B parameters per token; at equal total size, MoE like Lightning can beat dense peers such as Gemma 4 31B on token throughput, though latency gaps narrow at high concurrency.
Fine-tuning MoEs risks router imbalance; quantization hits router and recurrent-projection layers differently.
Parameter count without active-parameter literacy misleads buyers.
From the desk
We’re glad NVIDIA published the decision tree, vendor model and all.
The durable lesson: total parameters ≠ active compute. MoE can deliver more throughput per watt of math when sparsity is real, but you still pay memory for experts you aren’t firing. Concurrency closes some latency advantages. Fine-tune and quantize with eyes open.
Useful AI deployment means picking the architecture that matches your memory budget and traffic shape—not the biggest billboard number. The harm of cargo-cult MoE: teams adopt sparse models, botch routers, and get worse quality than a smaller dense model would have given.
I’m watching independent throughput/quality comparisons of Lightning versus dense 30B-class models outside NVIDIA’s harness.
Context
Nemotron 3.5 Lightning is NVIDIA’s MoE offering discussed in the post; Gemma 4 31B is used as a dense comparison point. Weights are pointed to Hugging Face and build.nvidia.com.
Who feels it
- Inference engineers
- Size hardware from active params and expert memory, not marketing totals alone.
- Fine-tuning teams
- Monitor router load balance; MoE training failures look different from dense ones.
- Procurement
- Ask vendors for active-parameter and concurrency curves, not only “30B.”
What to watch
- Third-party benchmarks of Lightning vs dense peers at matched quality.
- Router-imbalance incidents in production fine-tunes.
- Quantization recipes that preserve MoE routing quality.