1/18
Tokenizers and token budgets
The model never sees your text. It sees a sequence of integers produced by a tokenizer.
Presenter notes
Open cold with this claim and let it sit for a beat. Ask who has ever counted the tokens of a prompt they shipped. Two minutes, no slides beyond this one.
2/18
Why this session comes first
- Cost, latency and limits are all measured in tokens
- Context overflow is a silent failure, not an error
- Every later session assumes you reason in token space
Presenter notes
Set expectations: this is plumbing, and it is the plumbing that breaks. Three minutes, then move.
3/18
What a token is
- A whole word, for frequent words
- A word fragment, for rarer ones
- A punctuation mark or a leading space
- Raw bytes, when nothing else matches
Presenter notes
Tokenize one sentence live and read the pieces aloud, spaces included. This single demo removes the one-token-equals-one-word assumption for good. Five minutes.
4/18
Tokens are model-specific
The same string tokenized by two different tokenizers gives two different counts, and neither is wrong.
Presenter notes
Ask the room to guess the token count of a short sentence before you reveal both numbers. The spread in the guesses is the teaching moment.
5/18
A count without a tokenizer is not a result
Any token figure must carry the identifier of the tokenizer and the revision that produced it.
Presenter notes
Make this a rule for the rest of the session: every number written on the worksheet gets a tokenizer label next to it. Enforce it during Lab A.
6/18
Lab A — the same text, two tokenizers
Five samples, two tokenizers, one table. Compare the counts and the ratios.
Presenter notes
20 minutes, in pairs. Circulate and check that people are labelling their columns. Collect two or three ratios for the debrief.
7/18
What the ratios tell you
| Content type | Typical cost versus English prose | Why |
|---|---|---|
| English prose | Baseline, roughly 1.2 to 1.4 tokens per word | Best covered by the vocabulary |
| French prose, same meaning | Commonly 15 to 30 percent more | Less frequent sequences fragment further |
| Source code | Substantially more per visible character | Indentation, punctuation, identifier splits |
| Emoji and rare scripts | Several tokens per character | Byte-level fallback |
Presenter notes
Present these as defensible orders of magnitude, not constants. Say explicitly that they depend on the tokenizer and must be re-measured for yours.
8/18
The English default is a hidden cost
A non-English product pays a token tax on every request, which compounds into cost, latency and lost context.
Presenter notes
Relevant to this audience specifically. Ask whether anyone has sized a French-language product using English benchmarks. Two minutes.
9/18
Everything in the window is billed
- System instructions and tool schemas
- Conversation history, turn after turn
- Retrieved passages and attached files
- The user input, usually the smallest part
Presenter notes
Point out that the first three are invisible in most interfaces, which is exactly why they are forgotten. Print a rendered request if you have one ready.
10/18
Reserve the output
The generated answer must fit in the same window as the input, so budget it before you send.
Presenter notes
Ask what happens when 200 tokens remain and a 900-token answer is requested. Let someone say it out loud: the reply is cut mid-sentence.
11/18
The budget equation
System plus history plus schemas plus retrieval plus input plus reserved output must be under the context window.
Presenter notes
Write it on the board and leave it there. Learners will copy it onto the Lab B worksheet.
12/18
Lab B — budget a full request
Assemble, measure, total, compare, then decide what to cut.
Presenter notes
25 minutes, in pairs. Insist they measure the rendered template, not the raw text. Circulate for the overflow moment, which is where the learning happens.
13/18
When it does not fit
- Keep the system prompt intact
- Summarize the oldest history turns
- Drop retrieved passages from the lowest rank upward
- Never cut a tool schema mid-object
Presenter notes
Debrief Lab B here. Ask each pair to name the exact content they sacrificed, then push on whether the user would notice.
14/18
Truncation is a design decision
If you do not choose a policy, the runtime chooses one for you and does not tell you.
Presenter notes
This is the sentence to repeat. Pair it with the point that overflow produces degraded answers, not exceptions.
15/18
Pin the whole pair
- Model revision
- Tokenizer revision
- Chat template version
Presenter notes
These three are one atomic unit. State that a mismatch runs without error, which is what makes it dangerous. Lead into Lab C.
16/18
Test at the edges
- Multilingual text and non-Latin scripts
- Source code and structured formats
- Emoji, including composed sequences
- Special tokens and truncation direction
Presenter notes
Ask which of these are already in the room’s test suites. The honest answer is usually none of them. Three minutes.
17/18
Estimates versus measurements
Characters-per-token ratios are for capacity planning. Anything that gates a request uses the real tokenizer.
Presenter notes
Close the loop with the ratio someone applied to French in Lab A and compare it against the measured count.
18/18
Honest boundary
- Every number here is an order of magnitude, not a constant
- Vocabularies, limits and prices change between revisions
- The method stays valid; re-measure with your own tokenizer
Presenter notes
Close on the method rather than the figures. Tell learners the one thing to take away: measure your own pipeline, and label every count with its tokenizer.