systemsJul 15, 2026 · 19 min
Fixing ollama's MoE memory estimate — and a 3× kernel speedup that wasn't
Four investigations into the memory accounting and quantized kernels behind ollama and llama.cpp. One shipped: ollama under-counted MoE GPU memory by 20.6%, and a per-layer rewrite (PR #17201) cuts that to 6.0%. One got retracted: a tile config that measured 3.43× faster was computing wrong answers on 25 of 78 correctness cases. Two more gaps turned out already half-closed upstream.
mixture of expertscudamemory