Accelerating Dropless MoE Training in JAX with NVIDIA Transformer Engine

Mixture-of-experts training wastes hardware when routers send uneven token loads to experts. NVIDIA’s developer post details how Transformer Engine with JAX attacks that: grouped GEMM for ragged expert matmuls, NCCL EP that fuses dispatch/combine and deduplicates tokens, plus host offloading and XLA multistream collectives.

On DeepSeek-V3, an unoptimized baseline hit just 103 TFLOPS/GPU with communication dominating; optimized stacks reached 1,068 TFLOPS/GPU. At scale, NVIDIA reports 97% efficiency at 1,024 GPUs on GB300 NVL72 for the 671B model.

We’re watching dropless MoE become operable at rack scale instead of a quality-versus-efficiency tradeoff.

We’re rooting for the unglamorous kernel work. Capacity-based MoE drops or pads tokens to keep shapes rectangular; dropless keeps every token and needs block-sparse thinking. MegaBlocks-style approaches pointed the way; Transformer Engine’s grouped GEMM and EP path make it practical in JAX without Device-to-Host token-count copies killing CUDA graphs.

A 10× throughput story on DeepSeek-V3 is the kind of infrastructure win that quietly decides who can afford to train open MoE giants. Useful efficiency that preserves quality is exactly when we advocate for the stack. The downside if this scales only inside NVIDIA’s container and Blackwell/GB300 story is another lock-in layer dressed as open tooling — even when the algorithms ship in NGC MaxText.

I’m watching whether outside labs reproduce the TFLOPS claims, and how quickly MXFP8 grouped paths and promised NVFP4 fusions show up in day-to-day training configs.

MoE architectures power models such as DeepSeek, Qwen, and Mixtral by activating Top-K experts per token. NVIDIA points readers to the MaxText MoE configuration guide and NGC MaxText containers from September 9, 2026 onward.