GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model

Meta says its Generative Ads Recommendation Model, used across Facebook and Instagram, now trains on several thousand current-generation GPUs with trillions of sparse embedding parameters and billions of dense parameters. In 12 months, training FLOPs grew fourfold while end-to-end model-FLOPs utilization doubled to 20–25%.

The gains came from custom attention kernels for variable-length histories, mixed ultra-low-precision training and topology-aware parallelism. Meta says padding jagged sequences could waste up to 50% of compute. Jagged Flash Attention avoids that overhead, while BlockAttention reduces long-history attention cost from quadratic to linear.

Meta reports more than 30% end-to-end throughput improvement from optimized Generalized Dot-Product Attention kernels and describes using MXFP8 where ad objectives tolerate it while retaining BF16 in sensitive layers.

The engineering lesson is that foundation model does not imply one infrastructure recipe. Recommendation workloads combine irregular sequences, sparse embeddings, asymmetric interactions and numerically sensitive objectives. Generic language-model kernels leave expensive hardware idle.

Signal Desk's read: Meta’s most important result is organizational co-design, not one clever kernel. Doubling utilization required coordinating math, precision, memory, network topology and load balancing. That advantage is hard to buy off the shelf and favors companies with huge workloads and deep systems teams.

This is a vendor-authored Meta Engineering account of Meta’s own production system. All utilization and speed claims are vendor-reported and measured on internal hardware and workloads; the source offers no independent reproduction or direct business-outcome measure.

The work also reframes model scale: a fourfold rise in computation is defensible only if efficiency improves with it. Even at 20–25% utilization, much theoretical GPU capacity stays unrealized, preserving strong incentives for another round of systems optimization.

Conventional LLM training generally assumes dense, relatively uniform token sequences. Ad recommendations combine user histories of widely varying length with location, creative and engagement signals, producing very different compute, memory and communication patterns.

The result is a warning against evaluating infrastructure with synthetic LLM workloads alone. For recommendation systems, the shape of data and the business objective are inseparable from the hardware outcome, so benchmark portability has to be demonstrated rather than assumed.