Learning objectives
- Explain, in one sentence, what a tokenizer does to text before the model sees any of it.
- Tokenize the same passage with two different tokenizers and account for the difference in token counts.
- Enumerate every component of a request that consumes the context window, including the reserved output.
- Compute a token budget for a realistic request and state whether it fits a given context window.
- Choose and justify a truncation or summarization policy when the budget is exceeded.
- Demonstrate that non-English text, source code and emoji cost more tokens than equivalent English prose.
- Pin a model revision together with its matching tokenizer and chat template, and explain why the pair must move together.
Materials
- One device per participant with a Python environment and a tokenizer library installed ahead of time.
- Two tokenizers from different model families, downloaded before the session so nobody waits on the network.
- A prepared text pack: one English paragraph, its French translation, a short code snippet, and a line of emoji.
- A projector for the live tokenization demo, with a font large enough to show individual token boundaries.
- A printed or shared budget worksheet: system prompt, history, tool schemas, retrieved passages, user input, reserved output.
- Offline fallback: pre-computed token counts for the whole text pack, exported to a CSV so both labs still run with no network access.
Minute-by-minute plan
| Time | Duration | Segment |
|---|---|---|
| 0:00 | 10 min | Framing: the model never sees your text, only token identifiers. |
| 0:10 | 15 min | Concept 1: what a token actually is — words, fragments, punctuation, whitespace, bytes. |
| 0:25 | 20 min | Lab A: tokenize the same passage with two tokenizers and compare the counts. |
| 0:45 | 10 min | Debrief Lab A: why the same text yields different counts, and what that costs. |
| 0:55 | 5 min | Break. |
| 1:00 | 25 min | Lab B: build a full request budget and decide what to cut when it overflows. |
| 1:25 | 15 min | Concept 2: truncation and summarization policies, and the reserved output. |
| 1:40 | 10 min | Concept 3: operational controls — pinning revisions, testing edge cases, measuring for real. |
| 1:50 | 10 min | Lab C and close: the pinning checklist and the honest boundary. |
Key messages to land
- The model consumes token identifiers, not characters. Every reasoning step about cost, latency and limits happens in token space.
- Tokens are model-specific. The same string tokenized by two different tokenizers gives two different counts, and neither is wrong.
- A token is not a word. It can be a whole word, a word fragment, a punctuation mark, a leading space, or a run of raw bytes.
- The budget covers the entire request: system instructions, conversation history, tool schemas, retrieved passages, user input, and the output you reserve.
- If the total exceeds the context window, something is dropped. Choose the truncation or summarization policy explicitly, or the runtime will choose one for you.
- Character-based estimates are planning aids only. Anything that gates a request must be measured with the exact tokenizer paired to the model revision.
Common pitfalls
Learners assume one token equals one word and stop counting.
Tokenize a single sentence live and read the pieces out loud, including the leading spaces. Seeing " token" and "izer" split on screen ends the assumption faster than any explanation.
Someone estimates with a fixed characters-per-token ratio and treats the number as exact.
Have them apply the same ratio to French, to JSON, and to a line of emoji, then compare against the measured counts. The ratio is a planning aid, not a limit check.
The budget worksheet only counts the user prompt and ignores the reserved output.
Ask what happens when the model is asked for a 900-token answer with 200 tokens left in the window. The truncated reply makes the omission concrete.
Tool schemas and system instructions are forgotten because they are invisible in the interface.
Print the fully rendered request once, chat template included, and count it. The gap between the visible prompt and the actual payload is usually the surprise of the session.
A participant uses a tokenizer from a different revision than the model they are budgeting for.
Show that the counts diverge, and require the tokenizer identifier to be written next to every count on the worksheet. Numbers without a tokenizer identifier are not results.
The session drifts into comparing vendors on cost per token.
State up front that prices and limits change and that any comparison is dated. Teach the measurement method, which stays valid, not the current price list.