Topic
quantization
3 posts tagged quantization.
inference
2.34× faster LLM decoding: FP8 weights and native speculative decoding on Gemma-4
Optimizing single-stream inference for Gemma-4-26B on an A100 GPU using FP8 quantization and native Multi-Token-Prediction speculative decoding.
llmspeculative decodingquantizationmixture of experts
2.34×faster Gemma-4 decoding, same output127 tok/s ▸ 297 tok/s
video
Optimizing block-sparse INT8 attention and multi-expert step caching for Wan 2.2
Analyzing hardware execution limits and block selector heuristics in Wan 2.2 video generation. Demonstrates 1.30–1.76× speedups in block-sparse INT8 attention using a mean-pool selector and 1.78× speedups via per-expert diffusion step caching.
video generationattentionsparsityquantization
1.78×two-expert step cacheno cache ▸ per-expert
edge
Custom on-device voice commands from zero recordings
Designing a zero-shot synthetic data pipeline and INT8 quantized speech encoder to train custom on-device voice command classifiers for Qualcomm edge NPUs in under 60 seconds.
speechon-devicequantizationsynthetic data
1.24 msper inference, Snapdragon 8 Elite (fp32 profile)