inferenceJul 21, 2026 · 17 min
2.34× faster LLM decoding: FP8 weights and native speculative decoding on Gemma-4
The full path to 2.34× single-stream decoding of Gemma-4-26B on one A100: a batch-1 byte budget that named the only two levers, FP8 weights, a one-line vLLM patch that switched the model's own speculative head back on, a tuned kernel that did nothing, and the benchmark that first reported the win as a loss. Exact same output.
llmspeculative decodingquantization
2.34×faster Gemma-4 decoding, same output127 tok/s ▸ 297 tok/s
