← Reference · Nestor G Pestelos Jr · Print this page

Systems Engineering · Machine Learning

Chunked Prefill

Reference entry · last updated September 8, 2026

Chunked prefill is an inference scheduling optimization for large language models that partitions long prompt prefill requests into smaller token chunks across multiple execution iterations. By interleaving chunked prompt evaluation with active token decoding, chunked prefill prevents long input prompts from blocking decode steps, bounding tail inter-token latency without sacrificing GPU compute saturation.

1. First Principles: The Prefill-Decode Interference Problem

Serving systems for autoregressive language models handle two fundamentally asymmetric computational phases [1]:

• Prefill: Ingests the entire user prompt of length \(N_{\text{prompt}}\). Computes all Key-Value (KV) activations simultaneously using dense matrix multiplications. Operates at high arithmetic intensity and compute saturation, taking tens to hundreds of milliseconds.
• Decode: Emits one token at a time autoregressively. Each step loads model weights from memory for a single token per stream, operating at low arithmetic intensity and memory-bandwidth saturation. Each step requires 15 to 30 milliseconds.

In standard continuous batching systems, the scheduler prioritizes whole requests. When an incoming request carries a long prompt (such as a 16k or 32k token document), the GPU spends hundreds of milliseconds exclusively computing the prefill. During this window, all ongoing decode streams are paused. This head-of-line blocking creates extreme tail latency spikes in Inter-Token Latency (ITL), degrading user streaming experiences.

2. Mechanism and Token Budget Allocation

2.1 Budgeting and Slicing

Chunked prefill resolves phase contention by setting a maximum token budget \(T_{\text{budget}}\) per iteration step (for instance, 512, 1,024, or 2,048 tokens) [1, 2]. When a request arrives with prompt length \(N_{\text{prompt}} > C_{\text{chunk}}\), the scheduler slices the prompt into consecutive chunks:

In each forward pass, the model processes only one chunk \(C_j\) for that request, caching its resulting KV activations into memory before yielding to the next scheduling iteration.

2.2 Piggybacking and Mixed Batching

Rather than executing prefill chunks and decode steps in separate batches, chunked prefill co-schedules them in a single mixed batch. The iteration batch satisfies:

This technique (termed "piggybacking" in Sarathi) creates a complementary hardware synergy: the memory-bandwidth-bound decode tokens piggyback on the compute-heavy prefill chunk forward pass [1]. Tensor cores remain saturated with chunk GEMM (General Matrix Multiply) operations while weights are streamed to advance decode streams, maximizing aggregate hardware efficiency.

3. Performance Trade-offs: ITL Stability vs. TTFT

Chunked prefill introduces a deliberate engineering trade-off between tail inter-token latency and time to first token:

• ITL Tail Stabilization: Because iteration duration is strictly bounded by \(T_{\text{budget}}\), decode streams advance smoothly. High-percentile ITL (P99) drops dramatically, eliminating interactive pauses.
• TTFT Inflation: Spreading prompt evaluation across \(m\) scheduling iterations introduces minor overheads and scheduling intervals, slightly increasing overall Time to First Token (TTFT) for the incoming long prompt compared to unchunked dedicated execution.
• Attention Computation Overheads: Evaluating self-attention across multiple chunks requires attending back to previously cached chunks. While attention mechanisms like FlashAttention support chunked caching, total attention memory transfers increase moderately compared to single-pass full-context attention.

4. Systems Architecture and Disaggregation

Chunked prefill is standard in modern inference engines including vLLM and TensorRT-LLM. In larger multi-node production clusters, chunked prefill coexists with physical disaggregation (such as Splitwise and DistServe) [3, 4]. While disaggregated serving assigns separate GPU pools to prefill and decode workloads, chunked prefill remains essential within homogeneous clusters and within prefill workers handling variable context lengths.