Tarfa 2: a 320-billion-parameter model on a machine with less RAM than the model, faster than llama.cpp

A different kind of hard

The first Tarfa post was about streaming: a model too big for one GPU, fed expert by expert from NVMe. GLM-5.3-Flash is hard in other ways.

• The architecture. 45 layers. 34 of them use KDA, a gated linear attention with a recurrent state instead of a KV cache. The other 11 use MLA, DeepSeek-style latent attention with a sparse key indexer. Every layer mixes four parallel residual streams through a small Sinkhorn-normalized matrix (hyper-connections). Apart from the first three dense layers, the feed-forward layers are a mixture of 288 experts, 8 routed per token plus one shared expert. About 17 to 18 billion of the 320 billion parameters are active per token.
• The weights. The only copy on disk is Unsloth’s UD-Q4_K_XL GGUF: experts in Q4_K, Q5_K and Q6_K, nearly everything else in Q8_0, with small norm, router and other small tensors in F32. Z.ai does publish a BF16 checkpoint, but I had only the GGUF on disk, so there was no BF16 copy to be bit-identical to.
• The machine. Two Xeon Gold 5122 (8 cores in total, two NUMA nodes), 183 GiB of RAM, an RTX 4060 Ti 16 GB and a Quadro RTX 6000 24 GB. The 186 GiB file does not fit in RAM, let alone VRAM. Both GPUs together hold about 1,760 of the model’s 12,096 routed experts; every other expert a token needs is computed on the 8 CPU cores, straight from the page cache.

So “exact” changes meaning. Tarfa 2 uses the same quantized weights llama.cpp reads, dequantizes them exactly, and treats llama.cpp itself as the reference.

Day one: rebuild the model from its reference

The pieces are in papers, but nothing public that I found pins down this combination (its epsilons, clamps and tensor layouts) at the level an engine needs, so the specification came from llama.cpp’s source. I patched llama.cpp’s llama-eval-callback to dump full tensors, wrote every component as a numpy reference, and compared each one against those dumps:

For the component rows, the remaining error (about 1e-4) is consistent with quantization and floating-point order. The last row was hiding a real bug: the next token matched, but the logits agreed only to a cosine of 0.958. Along the way, two details each cost real time: the KDA output norm uses an epsilon of 1e-5, not 1e-6 (the scan output is so small that the wrong epsilon gave a 3× magnitude error), and the experts use a clamped SwiGLU, with the gate capped at 10 and the up projection clipped to ±10.

The bug position zero cannot see

The first longer decodes produced a loop: “Paris. Paris is the capital of France.”, repeated forever. The KDA test had run at position 0, where the recurrent state is still zero. A later probe at positions 1 to 4 did show drift (cosine down to 0.902), but I misread it as harmless rounding differences between llama.cpp’s step-by-step and chunked code paths, and moved on to speed work with the loop still there.

KDA decays its state with a separate gate per channel, and that decay has to act on the key axis. I had transcribed llama.cpp’s non-fused autoregressive code path, which reshapes the gate onto the value axis. For older gated-delta models with one scalar gate per head the axis does not matter; for KDA it does. llama.cpp’s real decode does not use that path (it calls a fused ggml_gated_delta_net op), so llama.cpp’s own output was fine. Mine was not. One line fixed it: block-output cosine at positions 1 to 4 rose from as low as 0.902 to 0.99996 or better, and the decode became coherent.

From 0.01 to 13 tokens per second

The first end-to-end engine, a PyTorch trunk with every expert dequantized in numpy, decoded at 0.01 tokens per second: about 90 seconds per token. Then came the fast path: Triton kernels, CUDA graphs, and a C kernel for the CPU experts. Each row is the speed after that change, from my run logs and notes:

The rows up to 11.1 use the same 5-token test prompt at different lengths, from 2 to 128 tokens (the 2.18 run was sampled, not greedy). The last two are single 128-token greedy runs on the 17-token story prompt I used for llama.cpp below (all runs are in the comparison); on that prompt the engine did 10.95 just before the AVX-512 change.

The comparison, under the same conditions

llama.cpp ran with the flags of the systemd unit I normally serve this model with (-ngl 99 -ncmoe 40 -sm layer -ts 41,4 -c 16384 -np 1 -t 16 -tb 16 -b 512 -ub 256 --numa distribute --flash-attn on --fit off --jinja), outside that unit’s 176 GiB memory cap. Both engines read the same GGUF, got the same raw 17-token prompt with no chat template, “Write a short story about a lighthouse keeper who finds a message in a bottle.”, and generated 128 greedy tokens. These are the flags I serve with, and tuning did not beat them: with 8 threads instead of 16 llama.cpp was slower (best 6.87), and automatic GPU placement (--fit on), a smaller context and loading without mmap (which does not fit in this machine’s RAM) were slower too.

The model is bigger than RAM, so each engine’s speed depends on what the page cache holds and where it sits. For the numbers below I dropped the whole file from the page cache before each engine started, so each one loaded it its own way: llama.cpp twice, with five requests per load, then Tarfa twice in each mode, and I compare best with best. llama.cpp’s first request after a load runs while it is still reading experts (1.13 tok/s); Tarfa’s first start after the cache is emptied can still take page faults during decode, and its second start reuses the pages the first one placed. Before decoding, Tarfa reads exactly its CPU-side experts (about 148 GiB) into the page cache; with CUDA graph capture that takes about 4 minutes per start (about 6 with MTP, which captures a second set of graphs), against about 2.5 minutes for llama.cpp to load from an empty cache and 30 to 40 seconds when the file is already cached. Neither engine’s number includes its startup.

Best against best, Tarfa 2 is 1.4 times faster, 1.5 times with MTP.

Correction, 9 November. The first version of this post said 1.6 and 1.7 times. In that comparison llama.cpp ran on a page cache Tarfa had filled and split across the two NUMA nodes for its own kernel, which slowed llama.cpp down: it peaked at 7.88 tok/s there, against 9.09 when it loads the model itself. The re-run also turned up a Tarfa bug: on an empty cache it could not move the expert pages to the right node afterwards, because both nodes were nearly full. It now reads each half of an expert on the node that computes it. Every run of both sessions, and every llama.cpp tuning attempt, is in the receipts.

Where the time goes

At 12.5 tokens per second a token takes 80 ms, and 47 ms of that is the CPU experts: about 190 of the 336 expert calls a token makes miss the GPUs and run on 8 cores. The GPU-resident experts run inside the CUDA graphs at the same time as the CPU work, so they add no time. The rest is the GPU trunk and the host glue around one routing decision per layer.

That made the CPU kernel the thing to beat. I started with llama.cpp’s own AVX2 k-quant dot products and measured them: for one token, memory latency limits them more than arithmetic does. Rewriting the Q4_K and Q5_K dots with AVX-512 changed nothing on its own. Prefetching 2 to 4 KB ahead inside the dot product helped, running across the end of one row into the next. Rows of an expert are contiguous in memory, but the hardware prefetcher stops at every 4 KB page. Time per expert went from 0.28 to 0.22 ms, with the same integer math as ggml. Prefetching whole rows ahead from the outer loop instead was three times slower.

Speculative decoding

GLM-5.3 ships with a multi-token-prediction (MTP) head: one extra block that guesses the token after next. llama.cpp loads the file but ignores that block for this architecture, so there was no reference implementation to check the wiring against. I tested the plausible layouts and let the acceptance rate decide: one ordering of its two inputs gets 78% of its guesses right, the other gets 0%.

The head guesses well: 77 to 88% of drafts were accepted across runs, so each verification step produces about 1.8 tokens. The speedup is small: 6% best against best (12.94 plain, 13.68 with MTP), and 4 to 14% between plain and MTP runs started back to back. The reason is the bottleneck above. Two consecutive tokens route to mostly different experts, so verifying two tokens touches about 1.7 to 1.8 times the CPU experts of one. CPU time scales with distinct experts, so only the GPU trunk and the host glue are shared, and the draft itself costs 6 to 10 ms per step.

As in Tarfa 1, where speculation moved to the labeled fast mode, it is a separate mode here. Verifying two tokens in one batch rounds differently from decoding one at a time: 0.4 to 1% of the largest logit at the first position, and later positions drift further once a close expert-routing call flips. That can flip a near-tie in the output. I checked that a verified token’s logits are bit-identical whatever draft sits next to it, and that after a rejected draft the restored state stays within that same rounding. It is deterministic and causal, but it does not round like the plain engine, so it is not the default.

What did not work

• Adaptive GPU expert set. Swapping experts between GPU and CPU as the conversation shifts raised the GPU hit rate from 38% to as much as 53%, as simulated, and gave no speedup. With RAM smaller than the model, every evicted expert comes back from NVMe, and the page-cache churn slowed the CPU side by as much as the extra hits saved.
• Streaming CPU experts to the GPU over PCIe. The Quadro can pull 12 GB/s from page-locked host memory. Sending it some of each layer’s CPU work made MTP decoding clearly slower (13.0–13.6 down to 12.0–12.3 tokens per second), and the one plain streaming run (11.8) was no faster than the plain runs without it (11.9–12.6). The CPU phase got longer: the copies and their bookkeeping cost more than the experts they took off the CPU.
• Hyperthreads for the CPU experts. Faster in a standalone benchmark, slower inside the engine.

What “matches llama.cpp” means

On the reference prompt, “The capital of France is”, the final build picks the same first token as llama.cpp, and its top five candidates are the same five tokens in a different order. On that prompt the two engines part ways at the third token, where llama.cpp leans to “In” and Tarfa to “Paris”.

On the story prompt they part at the fourth token. In 18 of its 19 runs on 7 November and in all 10 of the re-run, llama.cpp continued “The message is dated 50 years in the future”. In one run it opened exactly as five of my six Tarfa runs do, word for word as far as I logged it: “The message is from a shipwrecked sailor from 100 years ago”. That token is evidently a near-tie, and rounding decides it inside one engine as well as between two. Tarfa’s own runs split at later near-ties: all six of mine chose “from”, but one of the three plain runs parted from the other two at the 63rd token, and one of the three MTP runs at the 25th. The set of experts that ends up on the GPUs varies slightly from run to run, and an expert computed on a GPU rounds differently from the same expert on the CPU.

Limits

• Contexts up to 2,048 tokens. Below that, the MLA layers’ sparse indexer selects every key, so skipping it is exact; above it, the indexer is not implemented yet, and the published engine refuses longer contexts.
• One sequence at a time, greedy decoding, no serving API yet.
• Tuned for this one machine: the GPU expert set, thread pinning and NUMA layout all assume it.
• The GLM-5.3 engine is published as an experiment, on its own branch of the Tarfa repo (tarfa/engine/glm5), with every benchmark run in its receipts.

Why it matters to me

The first post argued that the ceiling on consumer hardware is engineering, not silicon. This one is a second data point: eight old CPU cores and two mismatched GPUs, with less RAM than the model file, run a 320-billion-parameter model faster than llama.cpp with my production flags on the same machine. Most of the gain came from measuring where the time went before changing anything.