Learning objectives
- Explain what the key-value cache stores, why it removes redundant attention work during decoding, and why it grows with both context length and concurrency.
- Apply the sizing estimate 2 x layers x KV heads x head dimension x cached tokens x bytes per element to a stated architecture, and scale it by the number of active sequences.
- Predict the memory consequence of doubling the active context or doubling concurrency under that linear estimate, and name where the estimate stops being linear.
- Compare bounded context, deliberate batching, GQA/MQA, paged attention, verified prefix reuse and lower cache precision as distinct levers with distinct costs.
- Distinguish a provider prompt cache from the run-time KV cache of a single request, and state the eligibility and persistence rules each one is governed by.
- Select the service metrics that expose cache pressure: time to first token, inter-token latency, output throughput, cache occupancy, eviction and recompute rate, p50/p95 latency.
- Design a load test whose context-length distribution and concurrency level match the production traffic it is meant to predict.
Materials
- One device per participant with a spreadsheet or a Python REPL — the sizing work is arithmetic and must be done by hand at least once.
- Printed configuration cards for three open-weight models, each listing layer count, attention heads, KV heads and head dimension.
- The documentation page of one serving runtime that implements paged attention, so the block-allocation vocabulary is read from the source rather than paraphrased.
- One published provider prompt-caching page, to contrast its persistence and eligibility rules with run-time caching.
- A whiteboard for the per-token, per-sequence, per-deployment memory diagram.
- Offline fallback: pre-captured screenshots of a serving dashboard showing TTFT, inter-token latency and cache occupancy under two concurrency levels, plus the printed configuration cards, so both labs run without network access.
Minute-by-minute plan
| Time | Duration | Segment |
|---|---|---|
| 0:00 | 10 min | Framing: why the second token is cheaper than the first, and what that costs in memory. |
| 0:10 | 20 min | Concept 1: what the KV cache holds. Whiteboard per-token, per-sequence, per-deployment. |
| 0:30 | 25 min | Lab A: apply the sizing formula to three architectures and build the context-by-concurrency table. |
| 0:55 | 5 min | Break. |
| 1:00 | 10 min | Debrief Lab A: where the linear estimate holds and where it quietly breaks. |
| 1:10 | 20 min | Concept 2: the levers — bounded context, batching, GQA/MQA, paged attention, prefix reuse, cache precision. |
| 1:30 | 15 min | Lab B: choose and sequence levers for a deployment that no longer fits in memory. |
| 1:45 | 10 min | Concept 3: measuring the service. TTFT, ITL, occupancy, eviction and recompute, p50/p95. |
| 1:55 | 5 min | Close: provider prompt cache is not run-time KV cache. Open questions. |
Key messages to land
- The KV cache buys decoding speed with memory. Every token you keep in context is state you must hold for as long as the sequence is alive.
- Under the simplified linear estimate, doubling the active context roughly doubles KV memory, and so does doubling the number of concurrent sequences.
- The formula 2 x layers x KV heads x head dimension x cached tokens x bytes per element is an estimate for a plain decoder. Architectures and runtimes deviate from it, so treat the result as an order of magnitude and verify against the runtime.
- KV heads, not attention heads, drive the cache size. That is why GQA and MQA cut cache memory while leaving parameter count nearly unchanged.
- Cache precision and weight precision are separate decisions with separate quality effects. Evaluate a lower-precision cache on its own, with its own evaluation set.
- A provider prompt cache is not the run-time KV cache of your request. Prefix reuse on the provider side has its own eligibility conditions, its own persistence window and its own billing.
Common pitfalls
Participants use the total attention-head count instead of the KV-head count and overstate memory by the GQA group factor.
Put both numbers side by side on every printed configuration card and require the KV-head figure to be circled before any arithmetic starts.
The formula is treated as an exact accounting of GPU memory, and learners are surprised when the runtime reports more.
State early that weights, activations, fragmentation and runtime overhead sit outside the formula. Ask for a range, never a single exact number.
Someone concludes that quantizing the cache is free because weight quantization worked well on their model.
Separate the two explicitly on the whiteboard and require a distinct evaluation. Long-context and multi-turn behaviour is where cache precision degrades first.
Prefix reuse is proposed for prompts that only look identical — same template, different tenant or different system instruction.
Force a byte-level comparison of two real prompts during Lab B. Reuse across trust boundaries is a correctness and privacy question, not an optimization detail.
Provider prompt caching is conflated with the run-time KV cache, and cost claims drift into fiction.
Read the provider page aloud during the close, and list the eligibility rules it actually states rather than the ones people assume.
The benchmark uses one fixed prompt length and one concurrency level, then the deployment behaves differently in production.
Require a context-length distribution and at least two concurrency levels in the Lab B measurement plan before accepting it as done.