Writing
How we make AI efficient.
Each post is one result, whether a model built from scratch or a kernel made faster, measured against a real production baseline.
16.8× faster video generation: distilling Wan 2.2 from 40 steps to 4
A public, no-training LoRA cuts Wan 2.2 A14B from 809 s to 48 s per clip on one A100 (16.8×), held across 13 scenes. The full path to that number: the evaluation-count budget that set the target, the distilled output that came back washed-out, the color grade that half-fixed it, and the checkpoint swap that fixed it for free.
2.34× faster LLM decoding: FP8 weights and native speculative decoding on Gemma-4
The full path to 2.34× single-stream decoding of Gemma-4-26B on one A100: a batch-1 byte budget that named the only two levers, FP8 weights, a one-line vLLM patch that switched the model's own speculative head back on, a tuned kernel that did nothing, and the benchmark that first reported the win as a loss. Exact same output.
Fixing block-sparse INT8 attention for Wan 2.2 — and a 1.78× two-expert cache
The published SOTA sparse-attention kernel gives ~1× on Wan 2.2 out of the box, because its block selector marks most video-latent blocks 'unpredictable.' Swapping only the selector recovers 1.3–1.76×. Plus a step cache tuned to the model's two experts, and a free sparsity signal already in the kernel's registers.
Fixing ollama's MoE memory estimate — and a 3× kernel speedup that wasn't
Four investigations into the memory accounting and quantized kernels behind ollama and llama.cpp. One shipped: ollama under-counted MoE GPU memory by 20.6%, and a per-layer rewrite (PR #17201) cuts that to 6.0%. One got retracted: a tile config that measured 3.43× faster was computing wrong answers on 25 of 78 correctness cases. Two more gaps turned out already half-closed upstream.
Custom on-device voice commands from zero recordings, in one 8.58 MB bundle
Freeze a shared speech encoder, train only a tiny classifier head on synthetic speech, and a custom voice-command model fits in 220 steps and ships as one 8.58 MB file that runs with no PyTorch and zero network calls, built for Qualcomm edge silicon.
6.1× smaller, 5.77× faster, same FID: compressing a coupled-oscillator image model
The full path to a 6.1× smaller and 5.77× faster Un-0 CIFAR-10 model at unchanged clean-FID: the byte budget that named two levers, a Monarch coupling that cuts 16.8M params to 0.52M, a tensor-core path worth 4.82×, the quality gate that rejected a bigger model, and the custom-kernel work that mostly failed. clean-FID held at 8.98.
Half the tensor cores of a consumer GPU sit idle on dense physics solves. Here's the one line that takes them back.
Dense solves, GP regression, and PDE Green's functions all reduce to one matmul that runs at half rate on a consumer RTX 3080. One line of PyTorch recovers 1.84–1.91× at ~1e-3 error. The reasoning that got there, including where it breaks.
One cuBLAS flag makes GPU retrieval scoring 1.59× faster — and nothing for sparse search
The full path to 1.59× on a consumer RTX 3080: every retrieval and recsys pattern collapses to one dense matmul that cuBLAS runs at half rate, one flag flips it, and recall@10 holds at 0.987 against the fp32 ranking. Two numbers reported, 1.59× on the matmul and 1.25× end to end, plus the sparse follow-on that was measured and refuted on purpose.