Transformers now runs llama.cpp quants

GGUF made local LLMs practical through llama.cpp, Ollama, and LM Studio. Hugging Face is bridging that world to transformers: load a Hub GGUF with from_pretrained, generate with the usual API, and optionally serve an OpenAI-compatible endpoint via transformers serve.

Initial focus is Apple Silicon, starting with Qwen3.5 architectures, reusing ggml kernels through the kernels library so performance can approach llama.cpp without abandoning PyTorch.

This is infrastructure glue that matters more than another leaderboard spike. Developers who live in transformers get quantized laptop workflows without a second mental model. Inspect activations, hack the forward pass, run existing eval suites on the same GGUF file Ollama users download — then dequantize for fine-tuning if needed.

Benchmarks on an M2 Max 32 GB MacBook Pro show transformers close to llama.cpp token generation across small dense, larger dense, and MoE GGUF checkpoints, with the caveat that the transformers measurement includes prefill while llama-bench’s tg128 figure is decode-oriented. Good enough to be useful; not a claim that transformers replaces llama.cpp as the dedicated local engine. Hugging Face still recommends llama.cpp when pure local efficiency is the priority.

Limitations are stated clearly: packed inference path is MPS-only for now; architecture coverage starts with Qwen3.5 dense/MoE (and compatible Qwen3.8); padding/batching still need work. That’s honest scoping. The longer game — ggml kernels accelerating models llama.cpp may never implement, including other modalities — is the strategic tell.

Useful AI is local AI that practitioners can actually instrument. The harm path is people assuming GGUF-in-transformers equals full llama.cpp parity on every device and shipping broken laptop demos. Read the fallback: without a compatible quantization kernel, weights dequantize and memory balloons.

I’m watching architecture coverage expanding beyond Qwen3.5, CUDA/Vulkan packed paths, and whether Jan/Pi-style clients standardize on transformers serve as a second local backend.

Hugging Face blog by marcsun13 with Arthur Zucker and Lysandre, published September 22, 2026. Performance numbers measured on the authors’ M2 Max setup as described in the post.