A tokenizer maps text to token IDs before a language model sees it. Token budgets constrain context, output, latency, and cost.
What you will be able to do
2-hour session
- Explain, in one sentence, what a tokenizer does to text before the model sees any of it.
- Tokenize the same passage with two different tokenizers and account for the difference in token counts.
- Enumerate every component of a request that consumes the context window, including the reserved output.
- Compute a token budget for a realistic request and state whether it fits a given context window.
- Choose and justify a truncation or summarization policy when the budget is exceeded.
- Demonstrate that non-English text, source code and emoji cost more tokens than equivalent English prose.
- Pin a model revision together with its matching tokenizer and chat template, and explain why the pair must move together.
Tokens are model-specific
A token may be a word, a word fragment, punctuation, whitespace, or bytes. The same text can have different counts across tokenizers. Use the exact tokenizer and chat template paired with the model revision.
Budget the whole request
Count system instructions, conversation history, tool schemas, retrieved passages, user input, and reserved output. If the total exceeds the context window, choose an explicit truncation or summarization policy.
Operational checks
Pin model and tokenizer revisions. Test multilingual text, code, emoji, special tokens, and truncation direction. Measure counts with the real tokenizer; character-based estimates are planning aids only.
Open the interactive — tokenizer
Quick check
Can two models assign different token counts to the same sentence?
Show answer
Yes. Vocabulary, normalization, and segmentation rules differ by tokenizer.
What must fit in the context window?
Show answer
The complete serialized request plus the output tokens reserved for generation.
Practise this
Work the three labs, then check yourself against the 10-question quiz.