Lab A — Sizing a KV cache from the architecture sheet
Duration: 25 minutes · Format: pairs · Sensitive data: none
Apply the estimate 2 x layers x KV heads x head dimension x cached tokens x bytes per element to one architecture, then extend it across contexts and concurrency levels. Work in bytes, convert at the end, and write the assumed architecture next to every figure. Assume for this lab a decoder with 80 layers, 8 KV heads, head dimension 128 and a cache stored in 16-bit elements (2 bytes).
| # | What to compute | Your result |
|---|---|---|
| 1 | Bytes cached per token for the stated architecture | |
| 2 | Cache for one sequence holding 4096 tokens | |
| 3 | Cache for one sequence holding 8192 tokens | |
| 4 | Cache for 8 concurrent sequences at 8192 tokens each | |
| 5 | Same as row 4 with an 8-bit cache instead of 16-bit | |
| 6 | Same as row 4 if the model used 64 KV heads instead of 8 |
Worked solution
Per token the estimate gives 2 x 80 x 8 x 128 x 2 bytes = 327 680 bytes, roughly 320 KiB per token; at 4096 tokens that is about 1.3 GB (1.25 GiB) for a single sequence, and about 2.7 GB (2.5 GiB) at 8192 tokens. Eight concurrent sequences at 8192 tokens each land near 21 GB (20 GiB), which is already a large fraction of a single accelerator before weights and activations are counted. Halving the element size to 8 bits halves the figure to roughly 10 GB, and moving from 8 KV heads to 64 multiplies it by eight to roughly 170 GB — that eightfold gap is the entire argument for grouped-query attention. Report these as orders of magnitude: the formula ignores weights, activations, allocator fragmentation and runtime overhead, so the measured footprint will be higher, and pairs should end the lab with a range and a stated architecture rather than a single number.
Lab B — Choosing levers for a deployment that no longer fits
Duration: 15 minutes · Format: groups of three · Sensitive data: none
A service serves a 70B-class model with GQA on a single 80 GB accelerator. Weights in 16-bit consume roughly 140 GB, so the deployment already spans two devices; the team wants 16 concurrent sequences at 8192 tokens, which Lab A prices near 43 GB of KV cache. Decide what to change, in what order, and what each move costs.
- Write the current KV budget and the target budget explicitly, using the Lab A figures and stating the architecture you assume.
- Rank these levers by expected memory saving: bound the retained context, cap concurrency, quantize the cache, adopt paged attention, reuse a verified shared prefix.
- For each lever, write the cost you accept: lost context, reduced throughput, quality risk, engineering effort, or a trust-boundary constraint.
- Choose the two you would ship first and justify why the others wait, in one sentence each.
- Write the measurement plan that would confirm the change worked: which metrics, at which context-length distribution, at which two concurrency levels.
Worked solution
The two moves that ship first are almost always bounding retained context and adopting paged attention, because both cut memory without touching model quality: bounding the context attacks the linear token term directly, and block allocation recovers the memory that per-sequence worst-case reservation wastes. Capping concurrency works arithmetically but pays for it in throughput, so it belongs as a load-shedding guard rather than a design choice. Cache quantization is the largest single lever after those — roughly halving the footprint from 16-bit to 8-bit — but it must wait for its own evaluation on long-context and multi-turn cases, because its failure mode is not the one weight quantization exhibits. Verified prefix reuse helps only when a substantial prefix is genuinely byte-identical and shared inside one trust boundary, which rules it out for per-tenant system prompts. An acceptable measurement plan names TTFT, inter-token latency, output throughput, cache occupancy and eviction/recompute rate, reported at p50 and p95, over a realistic context-length distribution at two concurrency levels rather than a single fixed prompt.
Lab C — Reading a serving dashboard under cache pressure
Duration: 15 minutes · Format: individual · Sensitive data: none
Using the dashboard screenshots (or a live run if one is available), diagnose a deployment whose latency degrades as concurrency rises. Fill one row per observation and commit to a diagnosis before reading the debrief.
| # | Signal to read | Your reading |
|---|---|---|
| 1 | TTFT at low concurrency versus high concurrency | |
| 2 | Inter-token latency at the same two points | |
| 3 | Cache occupancy as a fraction of the budget | |
| 4 | Eviction and recompute rate over the run | |
| 5 | Gap between p50 and p95 latency | |
| 6 | Your one-sentence diagnosis and the first lever you would pull |
Worked solution
The characteristic pattern is that inter-token latency stays roughly flat while time to first token climbs sharply with concurrency, occupancy sits close to the budget ceiling, and the eviction/recompute rate becomes non-zero exactly where TTFT breaks. That combination identifies cache pressure rather than a slow decode loop: evicted sequences must be re-prefilled, and prefill is what TTFT measures. The p95 to p50 gap widens far more than the medians move, which is the usual reason a service looks acceptable on averages and fails its latency objective in practice. The correct first lever is to reduce cached tokens — bound the retained context, and admit fewer sequences than the budget allows rather than evicting them later — since recompute after eviction is pure wasted work.