1/18
CPU vs GPU and quantization
Hardware fit depends on workload shape; quantization can reduce weight memory, but compatibility, latency, and task quality still require measurement.
Presenter notes
Read this sentence as written and tell learners it is the whole session in one line. Do not open with vendor names.
2/18
What this session is not
- Not a ranking of chips
- Not a recommendation of a quantization format
- Not a benchmark you can quote next week
Presenter notes
Setting this boundary early prevents the two derailments that cost the most time: vendor arguments and format arguments.
3/18
What a CPU is good at
General control flow, preprocessing, orchestration, and modest workloads where latency targets are loose and volume is low.
Presenter notes
Give one concrete case: a single-user internal tool with short prompts where a CPU meets the target and a GPU sits idle.
4/18
What a GPU is good at
Many parallel arithmetic units and high memory bandwidth for dense tensor operations.
Presenter notes
Stress bandwidth as much as arithmetic. Most learners arrive thinking only about raw compute.
5/18
Two phases, two bottlenecks
| Phase | Parallelism | Usual bottleneck |
|---|---|---|
| Prefill | All prompt positions at once | Compute |
| Decode | One token position at a time | Memory bandwidth |
| Both | Depends on batch and context | Measure, do not assume |
Presenter notes
Draw the timeline on the whiteboard while showing this. The sequential nature of decode is the single most load-bearing idea of the first hour.
6/18
Why decode stays sequential
Each generated token is conditioned on the previous one, so positions cannot be unrolled in advance.
Presenter notes
Ask the room what would have to be true for decode to parallelise. The answer they reach is speculative decoding; mention it exists and move on.
7/18
The first memory estimate
Weight memory is approximately parameters x bits per weight / 8.
Presenter notes
Do the 7B at 16 bits calculation live on the board so the 14 GB figure is derived, not announced.
8/18
Precision comparison, 7B model
| Precision | Bytes per weight | Weight estimate for 7B |
|---|---|---|
| 16-bit (FP16/BF16) | 2 | about 14 GB |
| 8-bit (INT8) | 1 | about 7 GB |
| 4-bit | 0.5 | about 3.5 GB |
Presenter notes
Say explicitly that these are orders of magnitude for weights only. Someone will otherwise write 3.5 GB into a capacity plan.
9/18
What the estimate excludes
- KV cache and activations
- Allocator overhead and runtime buffers
- Metadata and sometimes unquantized layers
Presenter notes
Build this as a stack on the whiteboard with weights at the bottom. Keep the stack visible for the rest of the session.
10/18
Lab A — estimate, then measure
Fill the table at three precisions, then read the resident memory the runtime actually reports.
Presenter notes
20 minutes in pairs. Circulate and reject any answer that reports the estimate without the measured figure next to it.
11/18
The gap is the lesson
The estimate is a floor. Resident memory is higher, and the difference grows with context and concurrency.
Presenter notes
Debrief Lab A here. Ask which pair saw the largest gap and have them state their context length and sequence count.
12/18
Counter-intuitive truth 1
A smaller model file does not guarantee lower latency.
Presenter notes
Pause after this line. It contradicts the intuition almost everyone brings, so let the silence do some work.
13/18
Why smaller can be slower
- Kernel support for the format on the target hardware
- Dequantization cost on the critical path
- Workload shape: batch size, context length, concurrency
Presenter notes
These three mechanisms are enough to explain nearly every surprising result learners will hit in Lab B.
14/18
Counter-intuitive truth 2
Weight quantization does not automatically shrink the KV cache: cache precision and weight precision are separate choices.
Presenter notes
Tie this back to the memory stack drawing. Point at the KV-cache layer physically and say it did not move.
15/18
Lab B — fix the workload, vary the precision
Same machine, same prompts, same output length, same concurrency. Only precision changes.
Presenter notes
25 minutes in pairs. Insist that baseline and rollback threshold are written down before anything is loaded.
16/18
The quantization release gate
- Compare against a higher-precision baseline on representative tasks
- Record quality, TTFT, inter-token latency, throughput, peak memory
- Record hardware, runtime, and artifact hash
Presenter notes
This slide is the deliverable. Have learners copy it verbatim before Lab C.
17/18
Measure at the real load
Concurrency 1 tells you almost nothing about a service that will run at 20 concurrent sequences with long contexts.
Presenter notes
Connect back to the KV-cache layer of the stack: it is load that turns a comfortable budget into an out-of-memory event.
18/18
Honest boundary
- Formats, kernels, and runtimes change on a monthly cadence
- Any specific number here is an order of magnitude, not a benchmark
- The protocol survives; the leaderboard does not
Presenter notes
Close by asking each learner to name one measurement they will run on their own stack this week. Take two answers aloud.