llama.cpp and ollama run local models on everything from a laptop GPU to an H100, which makes their memory accounting and quantized kernels worth auditing closely. A wrong number in either place spills computation onto the CPU that did not need to move, or runs a card out of memory, for real users. Four investigations went into that stack. One reached a submitted patch. The other three are code-grounded diagnoses, measured but not yet submitted, and reported at that confidence.
Every one followed the same loop: size the real bottleneck from a byte budget rather than a guess, target the single biggest term, try the smallest lever that moves it, and verify the result against the production default with a correctness gate wired into the measurement itself. The last clause is not boilerplate. The most instructive of the four is a win we retracted ourselves, because we ran the correctness test on the same matrix shapes the speed benchmark had been timing, and it disagreed.
The one that shipped: ollama’s MoE memory estimate
ollama decides how much of a model to place on the GPU by estimating its memory footprint, then splitting layers between GPU and CPU to fit. Underestimate, and it either offloads work to the CPU it did not need to, or picks a split that runs the GPU out of memory. For mixture-of-experts models the estimate was off by a fifth of the real allocation.
Finding the two wrong terms was a subtraction, not a hunt. llama-server reports what it actually allocates, the estimator reports what it predicts, and both decompose into the same short list of terms. Lining the two breakdowns up, the weight terms already agreed, so the entire 20.6% shortfall had to sit in the two that did not. The KV cache was sized from a single averaged head dimension, and the compute buffer was counted as exactly zero.
The averaged head dimension is the sharper bug. Attention memory depends on the real per-head key and
value lengths and on how many key/value heads a layer actually has, and both of those vary by
architecture in ways one averaged number cannot represent. DeepSeek-V2’s multi-head latent attention
(MLA) sets an explicit key_length = 192 and value_length = 128; folding the two into one dimension
under-sizes the asymmetric cache by about 20%. Grouped-query attention (GQA) shares each key/value head
across many query heads, so a single averaged head hides that ratio and mis-counts the cache by a
factor of two. The compute buffer, the scratch space llama.cpp reserves for the largest intermediate
tensors in one graph step, was not modeled at all. It was hardcoded to zero, which drops a real
multi-hundred-MiB allocation straight out of the budget.
Each wrong term admits a cheap patch and a correct one, and the two point in different directions. The KV cache could be scaled back into line with a per-architecture correction factor, a multiplier fit to MLA and GQA. That repairs the two models in front of us and silently mis-sizes the third, because the averaged head dimension is not an approximation of the right value, it is a different value that has already discarded the per-layer key and value lengths and KV-head counts the model’s own metadata carries. The fix is not a better average, it is to stop averaging. That is what per-layer and expert-aware means here, and it matters most for mixture-of-experts models precisely because their layers are heterogeneous enough that no single number stands in for all of them. The compute buffer posed the same choice. Replaying the graph would recover the true peak but couple the estimator to llama.cpp’s internal scheduling and break whenever that changed, so the rewrite reserves a conservative floor instead. A floor cannot regress below the zero it replaces, and it parks the residual error in one lower-bounded term rather than spreading it through the cache.
The rewrite is per-layer and expert-aware. It reads each layer’s real key and value lengths and its real KV-head count on both architectures, and it models the compute buffer as a conservative floor instead of zero.
The breakdown shows where the two terms moved:
| Estimate | KV MiB | Compute MiB | Total MiB | vs. actual |
|---|---|---|---|---|
| Actual (measured) | 8640 | 2376 | 19392 | ref |
| Old estimator | 6912 | 0 | 15405 | −20.6% |
| New (PR #17201) | 8640 | 1103 | 18235 | −6.0% |
KV is now exact and matches the measured 8,640 MiB. The compute buffer is a floor (1,103 of the true 2,376 MiB) rather than zero, which is where the remaining 6.0% gap sits. On a GQA model (qwen3:30b-a3b) the same code moves the estimate from −1.6% to +3.4%, erring slightly high rather than dangerously low, which is the direction ollama prefers.
The dangerous term is now exact. The residual on the MLA model is entirely the compute-buffer floor, which under-reserves rather than over-promises, and on the GQA model the net estimate flips to +3.4% over. This is the only item here that reached a validated, submitted patch.
Why one averaged head dimension mis-sizes the KV cache
The KV cache stores one key and one value vector per key/value head per token, so its size scales as
2 × layers × kv_heads × head_dim × context × bytes. Collapsing head_dim and kv_heads into one
averaged number breaks on exactly the two architectures at issue. MLA sets an explicit
key_length = 192 and value_length = 128; averaging the two under-sizes the asymmetric cache by about
20%. GQA makes kv_heads a small fraction of the query-head count, and a single averaged head erases
that ratio, mis-counting the cache by a factor of two. The rewrite reads each layer’s real key and value
lengths and its real KV-head count, which is why the KV column lands exactly on the measured 8,640 MiB.
What the compute buffer holds, and why zero is the worst default
Past weights and KV cache, llama.cpp reserves a compute buffer for the largest intermediate tensors one graph step produces: attention scores, the router’s expert activations, MLP intermediates. It grows with context length and batch, and on this model the real allocation is 2,376 MiB. Counting it as zero deletes that entire term from the budget, so the estimate runs shortest exactly when the context is longest, which is the case most likely to OOM. A floor (1,103 MiB here) does not capture the peak, but it replaces a guaranteed zero with a real lower bound, and it confines the remaining error to one conservative term instead of the KV cache.
The kernel speedup that was computing garbage
llama.cpp’s quantized-matmul kernel (MMQ) ships one hand-picked tile configuration and applies it to
every GPU it supports. A single tuple, nthreads=256, occupancy=1, I=128, K_vram=256, stream_k=true,
covers an A100, a 3090, a 4090 and an H100, across a 1.5× spread in SM count and memory bandwidth. One
constant covering that much silicon is the textbook place to look for tuning headroom, so the search
swept the knobs that do not change the math.
One setting stood out. Raising the VRAM tile to K_vram=512 measured 3.43× faster on Q4_K, 138.9
against 40.5 TFLOP/s, and reproduced as 1.60× end-to-end in llama-bench. Two independent harnesses
agreed, which is usually enough to write a number down.
The size of the jump is what kept it from being written down. K_vram controls how the kernel stages
weights through VRAM, not how much arithmetic it runs, and a matmul’s FLOP count is fixed by its shape.
A kernel already close to its roofline cannot drop to a third of its runtime from a staging change
alone, because there is no third of the work to shed unless the work itself shrank. So 3.43× was not a
plausible scheduling win, which left one explanation to rule out before publishing anything, that the
config was quietly doing less math. The test for that already existed.
Run on the same shapes the speed benchmark had timed, test-backend-ops test -o MUL_MAT failed 25 of
its 78 Q4_K and Q6_K cases with NMSE near 1.0, an output essentially uncorrelated with the true product.
The failing cases were the large-n shapes, precisely the prefill-regime matmuls the speed benchmark had
measured. The config was fast because, on those shapes, it was not computing the full product.
| Tile config | Q4_K throughput | vs. shipped default | MUL_MAT correctness |
|---|---|---|---|
| Shipped default · K_vram=256 | 40.5 TFLOP/s | 1.00× | 78 / 78 pass |
| Swept "winner" · K_vram=512 | 138.9 TFLOP/s | 3.43× micro · 1.60× bench | 53 / 78 · 25 FAIL (NMSE≈1.0) |
The 25 failing cases are the large-n Q4_K and Q6_K shapes the throughput benchmark timed; the config is fast there because it skips work. Both the 3.43× microbenchmark and the 1.60× llama-bench figure are retracted. The structural finding stands: one tuple serves four GPU generations across a 1.5× hardware spread, a real per-GPU tuning opportunity whose magnitude is unproven until the sweep reruns with the correctness gate inside the search.
What survives is structural, not a speedup. One hand-picked tuple really does serve four GPU generations across a 1.5× spread in cores and bandwidth, so a correct per-GPU tuning win is plausible. Its magnitude is now labeled unproven, pending a rerun with the correctness gate inside the search loop.
What NMSE ≈ 1.0 actually means
NMSE normalizes mean-squared error by the variance of the true output: NMSE = MSE(out, ref) / var(ref). Zero is a perfect match; 1.0 means the kernel’s output carries no more information about the
correct product than a constant would. The 25 failing cases were not slightly off. Their results were
statistically unrelated to the true matmul, which is what “fast because it skipped the work” looks like
in a correctness metric.
Why the failure hid until the full shape sweep ran
A speed benchmark and a quick correctness smoke test sample different matrix shapes. The K_vram=512
config was correct on the small-n shapes a casual check hits and wrong only on the large-n prefill
shapes. The performance benchmark lived entirely in that large-n regime, because that is where the
tiling change was meant to help, so it timed exactly the shapes the config got wrong. The result looked
fast and plausible right up until the full 78-shape sweep pushed the large-n cases through the
correctness test.
The k-quant gap: measure the incumbent first
The next gap was llama.cpp’s k-quant matmul, an INT8 tensor-core kernel that fuses dequantization into the GEMM. The kernel’s own upstream notes state that this INT8 path is suboptimal against an FP16 cuBLAS GEMM once the batch is large enough to be compute-bound (roughly batch ≥ 512), which is the premise the investigation started from: at large batch, replace it with a Marlin-style w4a16 GEMM.
Measuring the incumbent first changed the picture. On a small model (Qwen2.5-1.5B, one contended RTX 3090) the INT8 k-quant path sat on the favorable side of that crossover, 1.34× faster than F16 in prefill and 1.74× in decode. Which kernel wins is set by shape and scale, not by a fixed ranking.
The large-model, large-batch regime where cuBLAS is expected to pull ahead is the one measurement these pods could not run, so the opportunity is established in principle and turns on a single data point in the regime that motivated it. The Marlin-style w4a16 GEMM is specified against that crossover and becomes worth writing the moment a large-model prefill actually reaches it.
The k-quant / cuBLAS crossover
The INT8 k-quant kernel moves 4-bit weights and runs the matmul on INT8 tensor cores, folding dequantization inline. At small batch the step is memory-bound, so moving a quarter of the weight bytes wins, and here that is 1.74× in decode. At large batch the step becomes compute-bound, the dequant overhead stops hiding behind memory traffic, and FP16 cuBLAS’s higher sustained tensor-core throughput takes over, which is the upstream-noted crossover near batch 512. On a 1.5B model at these batch sizes the crossover has not been reached, so even prefill still favors the INT8 path at 1.34×. On a larger model the same prefill measurement is expected to invert, and that measurement was never taken here.
The MLA gap: one gated line
The last gap was DeepSeek’s multi-head latent attention. The assumption going in was that MLA had no tensor-core attention path and that llama.cpp therefore materialized a large scores buffer (about 2.4 GB) on every MLA step. Checking current HEAD showed the assumption was already half wrong.
MLA runs in two forms, and only one of them still falls back:
| MLA form | Head dims K / V | Group ratio | Tensor-core path |
|---|---|---|---|
| Absorbed | 576 / 512 | gqa = n_head | yes, MMA kernel |
| Uncompressed | 192 / 128 | gqa = 1 | no, falls back |
Support sweep: 32 of 176 configurations reach the MMA kernel, 144 fall back. The uncompressed form is excluded by one line, `fattn.cu:412`; the assumed ~2.4 GB materialized-scores buffer only applies on that fallback path. Restoring double-buffering there projects 1.15–1.4× (unmeasured). The absorbed path already runs on tensor cores, so half of the assumed gap was closed upstream before the work began.
The residual work is narrow. It is one gated shape, not a missing kernel, so the lever is to restore double-buffering on the fallback path rather than to write a tensor-core MLA kernel from scratch.
Absorbed vs. uncompressed MLA, and what fattn.cu:412 gates
MLA can run two ways. The absorbed form folds the latent up-projection into the attention math, so the
kernel sees compressed key/value dimensions of 576/512 with a group ratio equal to the head count,
which matches the shapes the tensor-core MMA flash-attention kernel already supports. The uncompressed
form keeps the explicit 192/128 head dimensions at a group ratio of 1, and that shape is excluded by a
single gate at fattn.cu:412, falling back to the non-tensor-core path. A 176-configuration sweep
confirms the split: 32 reach the MMA kernel, 144 fall back. Restoring double-buffering on the fallback
path projects 1.15–1.4×, unmeasured.
What shipped, and where this goes
The ollama memory estimate shipped, moving the MoE footprint from 20.6% under the real allocation to
6.0% under with the dangerous KV term now exact
(PR #17201). The MMQ sweep returned the most reusable
result in the set, a config that measured 3.43× faster and failed 25 of 78 correctness cases, and that
failure is what moved the correctness gate inside the tuning loop for every measurement after it. The
k-quant and MLA gaps both came out of the investigation smaller than they went in, each with one named
next measurement: for k-quant, a large-model prefill at batch ≥ 512 to find where the INT8 path yields
to cuBLAS; for MLA, restoring double-buffering on the fattn.cu:412 fallback and measuring the
projected 1.15–1.4× against HEAD. Both are a single run to settle, not a project.
The through-line is the loop, not any single figure. Size the bottleneck from a byte budget, localize the term that is actually wrong, pull the smallest lever that moves it, and gate the result on correctness measured on the same shapes the speed run timed. That loop shipped one patch, caught a 3.43× mirage before it could become a headline, and reduced two open gaps to one confirming run apiece. Pointed at the next stack, it runs the same way.