The key-value cache stores attention state for tokens already processed. It speeds decoding but grows with context and concurrent sequences.
What you will be able to do
2-hour session
- Explain what the key-value cache stores, why it removes redundant attention work during decoding, and why it grows with both context length and concurrency.
- Apply the sizing estimate 2 x layers x KV heads x head dimension x cached tokens x bytes per element to a stated architecture, and scale it by the number of active sequences.
- Predict the memory consequence of doubling the active context or doubling concurrency under that linear estimate, and name where the estimate stops being linear.
- Compare bounded context, deliberate batching, GQA/MQA, paged attention, verified prefix reuse and lower cache precision as distinct levers with distinct costs.
- Distinguish a provider prompt cache from the run-time KV cache of a single request, and state the eligibility and persistence rules each one is governed by.
- Select the service metrics that expose cache pressure: time to first token, inter-token latency, output throughput, cache occupancy, eviction and recompute rate, p50/p95 latency.
- Design a load test whose context-length distribution and concurrency level match the production traffic it is meant to predict.
Sizing model
A common decoder-only estimate is 2 × layers × KV heads × head dimension × cached tokens × bytes per element. Multiply by active sequences when caches are independent. Architectures and runtimes vary.
Optimization levers
Bound context, batch requests deliberately, use grouped-query or multi-query attention when the model supports it, page cache blocks, reuse verified shared prefixes, and evaluate lower cache precision separately from weight quantization.
Measure the service
Track time to first token, inter-token latency, output throughput, cache occupancy, eviction/recompute rate, and p50/p95 latency at realistic context distributions and concurrency.
Open the interactive — KV cache
Quick check
What happens to KV memory when active context doubles?
Show answer
Under the simplified linear estimate, it approximately doubles.
Is provider prompt caching identical to a request’s runtime KV cache?
Show answer
No. Provider prefix reuse has provider-specific persistence and eligibility rules.
Practise this
Work the three labs, then check yourself against the 10-question quiz.