A served model’s short-term memory of the conversation, filled by reading the whole prompt at once. Filling it accounts for the pause before the first word appears, the time to first token. Chapter 14.
From The AI Compute Race. Read the book
Where the book uses it
- Chapter 14 — The Convergence
One campus, Meta’s Hyperion, run through four possible futures to show how science, power, money and permission meet in the price of an accelerator-hour.
Sources that use it
- DeepSeek-AI, “DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression,” technical report, posted to the model’s Hugging Face repository 10 September 2026 (the laboratory’s own report on its own model, and the four-point per-token cache series that circulates with it, 389,120 to 48,068 to 3,514 to 890, is read off Figure 1(b) and appears nowhere in the paper’s text; the report carries no date on its face, the repository metadata giving createdAt 2026-09-10 02:17 UTC and the PDF’s own file metadata 2026-09-10 05:49 UTC; Abstract: these designs “reduce its global KV cache footprint (always in HBM) to 890 bytes per token, roughly 1/4 of the corresponding footprint of DeepSeek-V4-Flash”; Figure 1(b) caption: “approximately 4-fold and 437-fold reductions in per-token global KV cache size relative to DeepSeek-V4-Flash and DeepSeek-V1, respectively”; the report prints no price and no dollar sign, and makes no hardware, export-control or national claim; its architecture overview: “Overall, DeepSeek-V4.1-Flash has 552B backbone parameters and 196B Engram parameters, activating 8B parameters per token during prefill and 16B during decode”; section 2.4.2, of Engram: “During inference, deterministic addressing enables embeddings to be prefetched from host memory via background RDMA transfers, with prefetching for the first module overlapping computation in the first Transformer block”). In the Sources
Related terms
From the glossary of The AI Compute Race.
How to cite this page
Dryburgh, Lee S. “KV cache.” In The AI Compute Race. DRYBURGH. https://dryburgh.com/vol1/glossary/kv-cache (accessed 2026-11-01).
BibTeX
@misc{dryburgh-kv-cache-2026,
author = {Dryburgh, Lee S.},
title = {{KV cache}},
howpublished = {In \emph{The AI Compute Race}, DRYBURGH},
year = {2026},
url = {https://dryburgh.com/vol1/glossary/kv-cache},
note = {Accessed 2026-11-01}
}