Systems Engineering · Machine Learning

KV Cache

Reference entry · last updated September 11, 2026

KV cache (Key-Value cache) is a memory buffer in autoregressive transformer inference that stores intermediate Key and Value activation vectors across attention layers for past tokens.[1] By retaining these projections in accelerator high-bandwidth memory (HBM), the model evaluates attention for each newly emitted token without recomputing linear projections for preceding context.[2]

1. First Principles: Computational Trade-off

In standard transformer self-attention, the scaled dot-product attention for query tokens \(Q \in \mathbb{R}^{N_q \times d}\), key tokens \(K \in \mathbb{R}^{N_k \times d}\), and value tokens \(V \in \mathbb{R}^{N_k \times d}\) is defined by Vaswani et al.:[3]

During the prefill phase, all \(L\) prompt tokens are available at once. The attention mechanism calculates keys and values for all tokens simultaneously via dense matrix-matrix multiplication (GEMM), operating with high arithmetic intensity on GPU tensor cores.[2]

During autoregressive decoding, the model generates output tokens sequentially. Emitting step \(i+1\) requires calculating attention between the current token query \(q_i\) and the representations of all preceding tokens \(t_1, \dots, t_i\). Two operational choices exist:

• Naïve Recomputation: Recompute representations for all \(i\) preceding tokens through the full model stack at every step. While projection and feed-forward operations accumulate \(O(N^2)\) FLOPs across \(N\) generated tokens, recomputing uncached quadratic self-attention over the expanding prefix scales as \(\sum_{i=1}^N O(i^2) = O(N^3)\) total attention operations, imposing severe redundant compute overhead.
• KV Caching: Compute key and value vectors \(k_i = t_i W_K\) and \(v_i = t_i W_V\) only for the newly emitted token. Concatenate these vectors to cached tensors stored in high-bandwidth memory: \[K_{\le i} = [K_{< i}; k_i], \quad V_{\le i} = [V_{< i}; v_i]\] The model computes attention using vector-matrix multiplications (GEMV). Per-token projection compute stays \(O(1)\), and accumulated self-attention operations over \(N\) tokens scale as \(\sum_{i=1}^N O(i) = O(N^2)\).

This efficiency introduces a fundamental systems trade-off: the KV cache eliminates redundant mathematical operations by trading compute for memory storage and memory bandwidth.[1] Because every decoding step loads the full accumulated KV cache from high-bandwidth memory (HBM) into on-chip cache (SRAM) for a single token query vector, autoregressive decoding shifts from being compute-bound to memory-bandwidth bound.[2]

2. Memory Footprint Formulation

The total bytes required to store the KV cache for a transformer model depend on batch size, sequence length, network depth, and attention dimensions.[2]

For standard Multi-Head Attention (MHA), the total memory footprint \(M_{\text{KV}}\) in bytes is:

where:

• \(2\) represents the two distinct stored matrices (Key and Value).
• \(B\) is the serving batch size (number of concurrent sequences).
• \(L\) is the sequence length (prompt tokens plus generated tokens).
• \(n_{\text{layers}}\) is the total count of transformer decoder layers.
• \(h_{\text{kv}}\) is the number of key-value attention heads per layer.
• \(d_{\text{head}}\) is the dimension per head (often \(d_{\text{model}} / h_{\text{q}}\)).
• \(P\) is the numerical precision in bytes per parameter (e.g., 2 bytes for FP16 and BF16; 1 byte for FP8; 0.5 bytes for INT4).

For a 70-billion-parameter model with \(n_{\text{layers}} = 80\), \(h_{\text{kv}} = 64\), \(d_{\text{head}} = 128\), and 16-bit precision (\(P = 2\)):

At a context length of 128,000 tokens for a single user (\(B = 1\)), the KV cache footprint requires:

This single-sequence footprint exceeds the physical 80 GB capacity of an NVIDIA H100 or A100 GPU before accounting for the model parameter weights themselves.[1] Consequently, KV cache memory footprint is the primary bottleneck limiting maximum context length, batch size, and serving concurrency in production AI systems.

3. Architectural Variants: MHA, MQA, GQA, and MLA

To reduce KV cache storage requirements and memory bandwidth pressure during decoding, neural network architectures modify attention head topologies:[4]

Introduced by Shazeer in 2019, Multi-Query Attention collapses all Key and Value heads into a single head per layer.[5] While drastically slashing memory bandwidth demands, MQA can lead to capacity loss and training instability in large models.

Grouped-Query Attention, formulated by Ainslie et al. in 2023, strikes an empirical middle ground.[4] By assigning query heads into \(g\) groups that each share a single KV head (e.g., 8 KV heads serving 64 query heads), GQA recovers nearly all representational expressiveness of MHA while delivering an \(8\times\) reduction in KV cache storage and transfer overhead.

Multi-Head Latent Attention, introduced in DeepSeek-V2, compresses keys and values into a shared low-rank latent vector prior to caching.[6] During generation, only the compressed latent vector and decoupled positional encodings need to reside in the cache, reducing the effective per-token memory footprint to a fraction of standard GQA architectures.

4. Memory Management and PagedAttention

Conventional deep learning frameworks allocate memory for the KV cache as contiguous physical memory blocks sized to the maximum theoretical sequence length (\(L_{\text{max}}\)). This introduces severe memory fragmentation:[1]

• Internal Fragmentation: Memory reserved for future tokens remains unallocated while requests run shorter sequences.
• External Fragmentation: Dynamically changing sequence lengths create variable-sized gaps in memory allocation pools.
• Reservation Waste: Serving systems over-allocate slots in advance to guarantee generation does not crash due to out-of-memory errors.

In 2023, Kwon et al. introduced PagedAttention in the vLLM serving system, adapting operating system virtual memory concepts to transformer inference.[1]

PagedAttention partitions the KV cache of each sequence into fixed-size physical blocks (e.g., 16 or 32 tokens per block). A centralized block table translates logical token positions to non-contiguous physical pages in GPU VRAM. When a new token is generated, the engine writes to the current block; when the block fills, the system allocates another page from a shared global pool. This design eliminates external fragmentation and reduces internal memory waste to under 4%, multiplying effective serving throughput and concurrency by factors of 2 to 4.[1]

PagedAttention also enables copy-on-write memory sharing across multiple sequences. Parallel sampling, beam search, and multi-turn agent conversations share physical memory blocks for identical token prefixes, laying the foundation for modern prompt caching runtimes such as RadixAttention.[7]

5. Eviction and Compression Strategies

When contexts expand to millions of tokens, physical GPU memory limits require dropping or compressing older activations:[8]

• StreamingLLM (Attention Sinks): Xiao et al. observed that autoregressive transformers allocate an outsized proportion of attention weight to the initial 1 to 4 prompt tokens regardless of their semantic content.[8] By preserving these initial "attention sink" tokens alongside a rolling window of recent local tokens, language models maintain perplexity stability across infinite token streams without retraining.
• H2O (Heavy Hitter Oracle): Zhang et al. demonstrated that attention matrices exhibit cumulative sparsity: a small subset of tokens (heavy hitters) account for the vast majority of attention scores.[9] Evicting low-scoring tokens while preserving heavy hitters reduces cache size by up to 80% with minimal degradation in generation accuracy.
• Quantized KV Caches: Quantizing Key and Value tensors from FP16 (16-bit) to FP8 (8-bit) or INT4 (4-bit) reduces memory footprint by 50% to 75%. Modern inference engines apply per-channel or per-token scale factors to mitigate quantization error in outlier activations.