Topic
quantization
4 posts tagged quantization.
2.34× faster LLM decoding: FP8 weights and native speculative decoding on Gemma-4
The full path to 2.34× single-stream decoding of Gemma-4-26B on one A100: a batch-1 byte budget that named the only two levers, FP8 weights, a one-line vLLM patch that switched the model's own speculative head back on, a tuned kernel that did nothing, and the benchmark that first reported the win as a loss. Exact same output.
Fixing block-sparse INT8 attention for Wan 2.2 — and a 1.78× two-expert cache
The published SOTA sparse-attention kernel gives ~1× on Wan 2.2 out of the box, because its block selector marks most video-latent blocks 'unpredictable.' Swapping only the selector recovers 1.3–1.76×. Plus a step cache tuned to the model's two experts, and a free sparsity signal already in the kernel's registers.
Custom on-device voice commands from zero recordings.
How we trained a fully-functional voice AI model in <1min for Qualcomm edge silicon (and you can too!).
6.1× smaller, 5.77× faster, same FID: compressing a coupled-oscillator image model
The full path to a 6.1× smaller and 5.77× faster Un-0 CIFAR-10 model at unchanged clean-FID: the byte budget that named two levers, a Monarch coupling that cuts 16.8M params to 0.52M, a tensor-core path worth 4.82×, the quality gate that rejected a bigger model, and the custom-kernel work that mostly failed. clean-FID held at 8.98.