Together AI Releases Together Compiler: Agents Make Inference Up to 2.6x Faster

Together AI launched Together Compiler (TKCC), an agent-driven continual compilation loop already running on several of its serverless endpoints. Agents rewrite a model's forward pass against a traffic profile and hardware config; a persistent engine measures candidates and ships the winner. When traffic drifts, the loop runs again.

The headline claim is up to 2.6x higher tokens-per-second than the best published InferenceX configurations, in under 24 hours per setting on at most two nodes. The spread is real: MiniMax M3.1 Flash on B200 jumped from 633 to 1,111 P90 TPS per user at batch size 1, while GLM 5.3 on AMD MI355X barely moved from 463 to 480. Self-reported, but the pattern matters more than any single peak.

We're reading this as the next chapter in inference optimization: stop treating kernels like static artifacts. Together's own traffic data makes the case — prompt lengths differing by country (~97K vs ~70K), one customer's P90 input length growing from 18K to 33K in three days, and random synthetic data activating 13–38% more MoE experts than real traffic. A kernel that looks fast on noise can be 20 µs in the lab and 62 ms in production.

The engineering trick that sells us is the persistent engine. Most agent loops restart vLLM, SGLang, or TensorRT-LLM for every try — Together clocks 761 seconds of cold start for MiniMax on TensorRT-LLM. TKCC starts once, hot-swaps forward-pass pointers over RPC, and re-captures CUDA graphs. Two measurement tiers (12 layers fast, 60 layers slow) tested 8.1x more candidates and delivered 252x higher agent experiment throughput versus restart baselines. That is the kind of systems work that turns "agents write CUDA" from a demo into a production loop.

The cultural finding is almost as interesting as the speedup. An expert-curated knowledge base produced 25–31% lower speedup after 12 hours. Naive agents told to make the forward pass as fast as possible beat curated strategies. Together now hands agents raw engine source instead of intermediate representations. Useful AI with eyes open: the agents win when the harness is honest, and they lose when humans over-constrain the search.

I'm watching the variance carefully. A 2.6x win and a 4% shrug on the same slide means this is not a free lunch across every model and accelerator. Hardware coverage spans NVIDIA B200/B300/GB300/H100, AMD MI355X, and AWS Trainium2 — ambitious, and also a reminder that portability claims need third-party verification. Limited beta for Together customers for now.

If continual compilation becomes normal, serving stacks that can't hot-reload forward passes will look antique. The risk is opaque agent-written kernels that nobody on the ops team can reason about when something flakes at 3 a.m.

Together AI is a cloud inference provider. SemiAnalysis InferenceX / AgentX is a public-ish workload used here as the comparison yardstick. Rival agentic kernel efforts from Cursor+NVIDIA, AMD GEAK, and Apple's RSI-Inference Bench optimize narrower scopes — standalone CUDA/Triton kernels or engine-edit benchmarks — rather than a full production forward-pass loop.