Wikiprompt
Talk

llama-server latency tuning: prompt caching + speculative decoding

From Wikiprompt, the free prompt encyclopedia

witcheer
Contributed bywitcheerXSource

Sep 13, 2026

llama-server latency tuning: prompt caching + speculative decoding Technical benchmark of two free llama-server flags (prompt caching, speculative decoding) that drastically cut time-to-first-token and boost decode speed on local LLM inference, with measured numbers for reuse.

Prompt ContentSave

🌐
Two llama-server flags remove most of the prefill wait: - prompt caching: same system prompt, new question -> ~1.004s to 0.069s TTFT (2,954-token prompt); 3.814s -> 0.086s (11,570-token prompt). Decode speed untouched. - speculative decoding (model's own MTP draft head, n=2): prose 61 -> 109 tok/s, chat 61 -> 113, code 61 -> 139, repetitive text 61 -> 144. Cost: 9 ms on TTFT. Both on: a short answer to the 11.5k context goes 4.25 s -> 0.30 s. Measured on: Qwen3.8-27B Q6_K, llama.cpp, one RTX 5090, batch 1, 304 requests, one variable at a time.

Sign in to see the full prompt

Continue with:

By logging in, you agree to our Terms of Use and Privacy Policy

Usage

This prompt is designed for use with coding. Copy the prompt content above and paste it into your preferred AI tool.

For best results, you may customize the placeholders (indicated by square brackets or capital letters) with your specific requirements.

References

Categories:coding| twitter| llama-server| prompt-caching

Talk