Continuous Batching and KV Caching: Maximizing LLM Throughput

alt

You bought the expensive GPUs. You fine-tuned the model. You deployed it to production. And then you looked at your dashboard and saw that your GPU utilization was hovering around 15%. It feels like a waste of money, right? You are paying for thousands of cores to sit idle while waiting for a single user to finish typing their next token.

This is the hidden cost of naive Large Language Model (LLM) serving. The bottleneck isn't usually raw compute power; it's memory bandwidth and scheduling inefficiency. If you want to double or triple your throughput without buying more hardware, you need to understand two specific techniques: KV Caching and Continuous Batching. These aren't just buzzwords from research papers; they are the engine under the hood of modern serving systems like vLLM and TensorRT-LLM.

The Memory Problem in Transformer Inference

To fix the throughput issue, you first have to understand why standard inference is so slow. Transformers rely on an attention mechanism that looks back at all previous tokens to generate the next one. Without optimization, generating the 100th token requires recomputing the attention scores for the previous 99 tokens every single time. This computational complexity grows quadratically with sequence length, which kills performance.

KV Caching solves this by storing the key (K) and value (V) vectors for past tokens in memory. Instead of recalculating them, the model retrieves these cached values during the decode phase. This changes the computational cost per token from O(n²) to O(n). But there is a trade-off: memory usage grows linearly with the sequence length.

For a typical large model, say one with 32 layers and 40 heads, each cached token consumes significant VRAM. If you have 100 concurrent users, each generating a 2,000-token response, that cache fills up fast. When the cache is full, new requests stall, and your throughput crashes. This is where traditional static batching fails-it reserves memory for the maximum possible length, wasting space when users write short prompts.

Why Static Batching Leaves Money on the Table

Before continuous batching existed, we used static batching. The logic was simple: wait until you have a batch of N requests, process them together, and wait for all of them to finish before starting the next batch.

Imagine Batch A contains one request asking for a haiku (10 tokens) and another asking for a novel summary (2,000 tokens). With static batching, the GPU processes both in parallel. But once the haiku finishes after 10 steps, that slot sits idle for the remaining 1,990 steps while the long request continues. Your GPU is doing work for only one user out of two for 99% of that time. As output lengths vary-which they always do in real-world chat applications-static batching becomes incredibly inefficient.

How Continuous Batching Works

Continuous Batching, also known as iteration-level batching, changes the unit of scheduling from the request level to the token level. It doesn't wait for a batch to complete. Instead, at every single generation step, the scheduler checks who has finished and admits new requests immediately.

Here is the mechanical flow:

  • Admission: New requests enter the queue and start processing immediately if memory allows.
  • Execution: The GPU processes the current batch of active tokens.
  • Eviction: As soon as a request generates its end-of-sequence token, it is removed from the active batch.
  • Replacement: A waiting request slots into that freed position instantly for the next step.

This keeps the GPU saturated. There are no idle slots waiting for the longest request in the batch to finish. Benchmarks from systems like vLLM show this approach can improve throughput by 10x to 20x compared to static batching. That’s not a marginal gain; that’s the difference between needing ten servers and needing one.

Visual metaphor of organized memory blocks versus fragmented storage

PagedAttention: Solving the Memory Fragmentation Issue

Continuous batching creates a new problem: dynamic memory allocation. If requests come and go unpredictably, how do you manage the KV cache without fragmenting memory? Enter PagedAttention. Inspired by virtual memory paging in operating systems, this technique splits the KV cache into fixed-size blocks (pages).

Instead of reserving a contiguous block of memory for a whole sequence, PagedAttention allocates pages on demand. If a request needs more space, it grabs another page from the free pool. If a request finishes, its pages return to the pool. This eliminates internal fragmentation because you don't reserve space for unused tokens. It also enables non-contiguous storage, meaning physical memory addresses don't need to be sequential, which vastly improves memory utilization rates.

Chunked Prefill: Handling Long Prompts

While continuous batching optimizes the decoding phase (generating output), the prefill phase (processing input) can still cause latency spikes. If a user submits a massive document prompt, the prefill computation can hog the GPU, delaying other users' decode steps.

Modern implementations use Chunked Prefill to break long prompts into smaller chunks. These chunks are interleaved with the decode steps of other ongoing requests. This ensures that a single long prompt doesn't starve the rest of the system. The scheduler balances the heavy compute load of prefilling against the memory-bound nature of decoding, keeping latency predictable even under mixed workloads.

Comparison of LLM Serving Strategies
Feature Static Batching Continuous Batching + PagedAttention
Scheduling Granularity Request Level Token / Iteration Level
Memory Management Pre-allocated contiguous blocks Dynamic paged blocks
Idle Time High (waits for longest request) Negligible (slots filled instantly)
Throughput Gain Baseline 10x - 20x improvement
Complexity Low High (requires custom kernel/scheduler)
High-speed token processing line with instant slot replacement

Implementation Tools and Real-World Impact

You don't need to write these kernels from scratch. The industry has standardized on a few open-source frameworks that implement these optimizations natively.

vLLM is currently the gold standard for open-source inference. It introduced PagedAttention to the mainstream and combines it with continuous batching out of the box. Teams using vLLM often report being able to serve 2-3x more traffic on the same hardware compared to older Hugging Face pipelines.

TensorRT-LLM from NVIDIA offers similar benefits but focuses heavily on kernel fusion and low-level CUDA optimizations. It’s often faster than vLLM for specific models but can be harder to configure.

Other emerging tools like SGLang focus on structured generation and prefix sharing, leveraging the same underlying batching principles but optimizing for complex agent workflows where multiple calls share common context.

When Not to Use Continuous Batching

Is it always the right choice? Mostly, yes. But there are edge cases. If you are running a single-user local application on a laptop, the overhead of the scheduler might outweigh the benefits. Continuous batching shines when you have concurrency-multiple requests arriving asynchronously.

Also, keep an eye on Time To First Token (TTFT). While continuous batching maximizes overall throughput (tokens per second), aggressive batching can sometimes increase TTFT if the queue gets too deep. You need to tune the max batch size and memory limits based on your specific latency requirements. For interactive chat apps, prioritize lower TTFT by limiting batch sizes. For offline data processing, maximize batch sizes to crush costs.

Key Takeaways

  • KV Caching prevents redundant computation but eats VRAM.
  • Continuous Batching schedules at the token level, eliminating idle GPU cycles caused by variable-length outputs.
  • PagedAttention manages memory dynamically to prevent fragmentation, allowing higher concurrency.
  • Tools like vLLM make these optimizations accessible without writing custom CUDA code.
  • The primary goal is maximizing GPU Utilization; if your GPU is idle, you are losing money.

What is the difference between static and continuous batching?

Static batching waits for a group of requests to finish before starting new ones, leading to idle GPU time if requests have different lengths. Continuous batching adds and removes requests at every generation step, keeping the GPU fully utilized regardless of individual request duration.

Does KV caching affect accuracy?

No, KV caching does not change the mathematical output of the model. It simply stores intermediate results to avoid recomputation. The generated text remains identical to what would be produced without caching, assuming floating-point precision remains consistent.

Which framework implements PagedAttention best?

vLLM is widely recognized for popularizing and robustly implementing PagedAttention. However, newer versions of TensorRT-LLM and other specialized engines also support similar memory management techniques, often with additional optimizations for specific hardware architectures.

Can I use continuous batching for single-request scenarios?

Yes, but the benefit is minimal. Continuous batching is designed for multi-tenant environments with concurrent requests. For a single stream, the scheduler overhead might slightly exceed the gains, though modern implementations handle this gracefully with negligible penalty.

How much memory does KV caching require?

It depends on the model architecture. Generally, it scales linearly with sequence length, number of layers, and head dimension. For large models, a single long conversation can consume several gigabytes of VRAM, which is why efficient memory management like PagedAttention is critical.