Lab A — The same text, two tokenizers
Duration: 20 minutes · Format: pairs, one machine each · Sensitive data: none
Tokenize each item of the prepared text pack with two tokenizers from different model families. Record the token count for each cell, then compute the ratio between the two tokenizers and the ratio between English and French. Write the tokenizer identifier at the top of each column; a count without an identifier is not a result.
| Text sample | Tokenizer A count | Tokenizer B count |
|---|---|---|
| English paragraph (about 100 words) | ||
| Its French translation, same meaning | ||
| A 15-line Python snippet | ||
| One line of 10 emoji | ||
| A JSON tool schema, formatted |
Worked solution
The English paragraph lands near 120 to 140 tokens for roughly 100 words, so the count exceeds the word count even in the best case. Its French translation costs measurably more with the same tokenizer, commonly 15 to 30 percent, because accented and less frequent sequences split into more pieces. The Python snippet is dominated by indentation, punctuation and identifier fragments, so it costs far more per visible character than prose. Each emoji typically consumes several tokens because it falls back to byte-level pieces, and skin-tone or flag sequences cost more again. Across the two tokenizers the totals differ by roughly 10 to 25 percent on the same rows, which is the point of the lab: the number belongs to the tokenizer, not to the text.
Lab B — Budget a full request
Duration: 25 minutes · Format: pairs, one worksheet per pair · Sensitive data: none
Build the complete token budget of a realistic assistant request against a stated context window, then decide what to cut when it does not fit.
- Assemble a realistic request: a system prompt, six turns of conversation history, two tool schemas, three retrieved passages, and a user question.
- Measure each component separately with the real tokenizer, including the fully rendered chat template rather than the raw text.
- Add a reserved output allowance sized for the answer you actually expect, and total everything.
- Compare the total against a declared context window, for instance 8,000 tokens, and record the overflow or the headroom.
- If it overflows, write an explicit policy stating what is dropped, what is summarized, and in which direction truncation happens, then re-measure.
Worked solution
The rendered request is consistently larger than the sum of the visible parts, because the chat template adds role markers and special tokens on every turn. In a typical build the two tool schemas alone account for 400 to 900 tokens and the three retrieved passages for 900 to 2,000, so the user question is usually the smallest line on the worksheet. With a 1,000-token reserved output, most pairs overflow an 8,000-token window on their first attempt. The workable policies are to keep the system prompt intact, summarize the oldest history turns, and drop retrieved passages from the lowest-ranked one upward, never truncating a tool schema mid-object since that breaks parsing. After re-measurement the total should sit below the window with visible headroom, and the pair should be able to name exactly which content was sacrificed.
Lab C — Pin the pair and break it on purpose
Duration: 10 minutes · Format: demonstration led by the instructor, learners follow along · Sensitive data: none
Load a tokenizer pinned to an explicit model revision, tokenize a short prompt with its chat template, then deliberately swap in a tokenizer from another family and observe what changes.
- Load the tokenizer by pinning an exact revision identifier rather than a moving tag such as the default branch.
- Render a two-turn conversation through the chat template and print the resulting token identifiers together with the decoded pieces.
- Locate the special tokens in the output and note their positions relative to the message content.
- Swap in a tokenizer from a different model family, re-render the same conversation, and compare identifiers, counts and special-token placement.
- Write down the three fields that must appear in your configuration: model revision, tokenizer revision, chat template version.
Worked solution
The pinned load produces a stable identifier sequence that can be re-run tomorrow and give the same result, which is the entire purpose of pinning. The decoded pieces show leading spaces attached to words and the special tokens wrapping each role segment, so the rendered payload is visibly longer than the raw message text. After the swap, the identifier sequence is completely different, the token count moves by a noticeable margin, and the special tokens are either renamed, repositioned, or absent. Nothing raises an error: the mismatched pipeline runs and returns text, which is exactly why this failure survives into production. The takeaway is that model revision, tokenizer revision and chat template version are one atomic unit and must be versioned together.