How to Run NVIDIA Dory 4B with llama.cpp: Loop by Loop Guide

Dory 1.0 4B is a small reasoning model from NVIDIA’s Nemotron Labs. Its middle layers form a shared block that can be applied once, twice, three, or four times for every token. Each extra loop spends more computation per token without adding a single parameter, so one checkpoint can run at four different compute budgets with no retraining.

A hands-on walkthrough builds the llama.cpp branch that NVIDIA’s model card pins, downloads the 3.7 GB GGUF, and reads the architecture from the file to predict what each loop should cost in parameters and cache memory. It then checks those predictions against a running server, measures decode speed at every loop count, tests whether the cheaper last-loop cache changes the model’s predictions, and runs a paired GSM8K evaluation looking at accuracy, answer length, and time per solved question as loops increase.

That is a genuinely useful idea: trade speed against depth without downloading four checkpoints. It only works if the loop count is set on purpose.

We're watching test-time compute become a server flag, which is the interesting part of this walkthrough. Dory support is not in released llama.cpp. The walkthrough builds llama-server from a specific NVIDIA fork — github.com/brianchmielNV/llama.cpp at commit 98a74c97ea09ac72f6ceda1559808fc0a2f447ce — because ggml-org pull request #30306 is still a draft and no released binary can load the model. The GGUF comes from nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-GGUF at revision df53755cb8741843af8e019e088819fbef17e29b.

The header is more revealing than the marketing. Dory’s 72 layer entries split into an input group of 16, a shared recurrent group of 40, and an output group of 16, alternating attention and MLP, with one full-attention layer for every three sliding-window layers. The recurrent block accounts for 1.84 of the model’s 3.98 billion parameters. The built-in loop count is 4; the card recommends 2. A server started without an override silently runs the most expensive setting. Loop count and cache mode are read when the model loads, so each setting needs its own process.

From those group sizes the notebook predicts the bill before anything runs: four loops apply about 2.51 times the parameters of one loop and read about 2.28 times the weight bytes. For a 4,096-token context, the per_loop cache is predicted to grow from 306 MiB at one loop to 816 MiB at four, while last_loop stays at 306 MiB. The authors start a server for each of seven settings — one to four loops in both cache modes — and record measured KV memory next to those predictions. They also check the claim that last_loop keeps the KV cache at the one-loop size for every loop count, compare the top 20 next-token candidates on 50 GSM8K questions used as raw prompts, and run the same seeded sample under five settings with reasoning on and the card’s sampling settings.

I'm not going to pretend a thirty-question paired GSM8K run settles the quality curve. The tutorial itself says thirty paired questions cannot settle small accuracy differences, reports accuracy with Wilson intervals, and uses exact McNemar tests on contrasts. The published write-up does not state those measured scores, and the authors say the practical lessons hold whatever the measurements show. That is the honest frame. Claims the notebook cannot reach — the card’s broader benchmark table, a 256K context, and NVFP4 pre-training — are left untested rather than waved through.

Useful AI here is real: one 3.7 GB file that runs as four different models, from a single pass through its 20-block recurrent core to four, with costs the GGUF header alone can predict. The downside is operational, not hypothetical. It is easy to silently run four loops when the card wanted two. It is easy to depend on a pinned draft fork that has not merged. If this pattern scales — recurrent blocks as a load-time compute dial — every local reasoning model will need an explicit budget flag and a pinned runtime, or people will benchmark the expensive setting by accident and call it the model.

The card’s download commands call the file dory-q5_k_m.gguf, but the repository names it NVIDIA-Nemotron-Labs-Dory1.0-4B-Q5_K_M.gguf, so copying the card verbatim fails. The walkthrough pins the real filename, size, and checksum together before any server starts.