How to Size GPUs for AI Inference and TCO Without Overspending

The framework starts by sorting workloads into four buckets — chatbots and copilots, agents doing deep research, content generation, and translation — because each has a different token shape. Chatbots run long input and short output. Agents can exceed 128,000 cached input tokens. Content generation flips to short input and long output. Those profiles, not GPU spec sheets, drive memory and compute.

From there the inputs are concrete: model choice, daily active users, concurrency, input and output string lengths, cache hit rate, latency targets including time to first token and 99th-percentile, requests per user per day, and contract length. Nvidia's capacity advice is core-and-flex: reserved or on-prem GPUs for steady-state traffic, public cloud spot or on-demand for bursts.

Coming from the company that sells the GPUs, the most notable line is that bigger is not always better. Nvidia explicitly points readers at smaller mainstream models and fine-tuned alternatives, then spends the back half of the post on shrinking what you run. That is not altruism. It is a company that would rather sell you more inference workloads on right-sized hardware than watch a stalled TCO review kill the project.

The numbers it does put forward are optimization numbers, not sizing numbers. FP8 post-training quantization with Model Optimizer cut Llama-3.1-8B weight memory from 16.06GB to 9.08GB, a 43.5 percent reduction with no retraining. A pruning-plus-distillation run on Qwen3-8B produced roughly 6B-parameter students; in that run width pruning reached a lower final validation loss than depth pruning, 3.21 versus 3.60, while depth converged faster. Nvidia flags its own dataset as small and the figures as illustrative, which is the correct caveat and the one most slide decks will drop.

Signal Desk's read: the practical value here is the argument that concurrency, not user count, is what breaks a deployment. High concurrency strains memory and latency more than raw DAUs, and cache hit rate is the lever most teams never measure — repeated prefill is invisible on a dashboard until it is 40 percent of the bill. Any team quoting cost per token without those two numbers is guessing.

The weak spot is the sample scenarios. They are useful shape, but the memory recommendations are generic enough that a finance team could not defend them in a procurement review. Treat them as the starting question, not the answer.

Quantization, pruning, and distillation are ordered here by engineering effort, and Nvidia recommends FP8 as the starting point, escalating to quantization-aware training only when post-training loss exceeds tolerance. That ordering is the useful part; it turns a vague cost-cutting mandate into a sequence.