Google Cloud Ships llm-d 1.0 With Cache-Aware Routing for TPU and GPU Fleets

The llm-d project has shipped version 1.0, its first stable release. It is an open-source, Kubernetes-native layer that turns a fleet of vLLM or SGLang servers into one coordinated pool, routing each request to the replica that already holds the relevant cached context. It runs on NVIDIA, AMD and Intel GPUs as well as Google TPUs and was released at the PyTorch Conference North America.

Why it matters: agent workloads send long, repeated context step after step. Simple round-robin load balancing throws away cached work and wastes accelerators. The 1.0 release focuses on production concerns like fair scheduling between tenants, zero-downtime upgrades and session-aware routing for agents.

We are bullish on this kind of work. A lot of AI cost lives not in model weights but in how efficiently inference is served, and open infrastructure that cuts waste lowers the price of useful AI for everyone, not only for the biggest labs. The governance helps: the project was founded by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA, sits in the CNCF Sandbox, is Apache 2.0 licensed, and counts more than 500 contributors from over 25 organizations.

The performance claims deserve the usual discount. Every figure is self-reported by Google and adopters. The headline is more than 85% lower median time to first token and 2.4x throughput on an Ironwood TPU coding workload. Google also says prefix cache hit rate on its Gemini platform doubled from 35% to 70%. An IBM demo served 3,000 concurrent coding agents at a fraction of API pricing. These are plausible directions, but they are vendor benchmarks on favorable workloads, and the honest weak spot is in the same release: reinforcement learning time-slicing only lifts accelerator duty cycle to above 70%, leaving roughly 30% idle.

The strategic read is that Google is making TPUs a first-class citizen in the open serving stack, alongside its native PyTorch backend, so that choosing TPUs no longer means choosing a closed toolchain. It is also notable that this is cooperative with NVIDIA's Dynamo rather than a head-on rival; CoreWeave runs both, and NVIDIA contributes components.

The downside to watch is that cheaper inference at scale means more agents running more often. Efficiency gains tend to get spent on volume, not saved. That is fine for builders, but it is worth remembering when anyone claims better serving will ease power demand.

Prefix-aware routing, KV cache offloading to host memory or storage, and splitting prefill from decode were already in earlier llm-d versions. Version 1.0 is framed as making them reliable for day-to-day production. On Google Cloud, GKE Inference Gateway runs the llm-d router as a managed service.