Tokenizers and token budgets

10 questions, one correct answer each. The goal is to check whether you can reason in token space: what a tokenizer produces, what consumes the context window, and what must be measured rather than estimated.

1. What does a tokenizer do?

A) It compresses text to reduce storage cost
B) It converts text into a sequence of token identifiers the model can consume
C) It translates text into English before inference
D) It removes punctuation and whitespace from the prompt

Show answer

B — The model operates on integer identifiers drawn from a fixed vocabulary. The tokenizer is the mapping between text and that sequence of identifiers, in both directions.

2. The same sentence gives 14 tokens with one tokenizer and 19 with another. What follows?

A) One of the two tokenizers is faulty
B) The sentence is ambiguous
C) Token counts are tokenizer-specific, so both can be correct
D) The longer count always indicates a larger model

Show answer

C — Vocabularies and merge rules differ between model families. A count is only meaningful when the tokenizer that produced it is identified.

3. Which of these can be a single token?

A) Only complete words
B) Only words and punctuation marks
C) A word, a word fragment, a punctuation mark, a leading space, or raw bytes
D) Only sequences of at least three characters

Show answer

C — Subword vocabularies contain fragments, punctuation, whitespace-prefixed pieces and byte-level fallbacks, which is how unknown text is still encodable.

4. Which items must be included in a request token budget?

A) The user input only
B) The user input and the system prompt
C) System instructions, history, tool schemas, retrieved passages, user input, and reserved output
D) Everything except the output, which is billed separately

Show answer

C — The context window holds the whole rendered request, and the generated output has to fit as well. Omitting the reserved output is the most common budgeting error.

5. A request needs 7,000 input tokens and a 1,500-token answer, in a 8,000-token window. What happens?

A) It fits, because input and output are counted separately
B) It does not fit; roughly 500 tokens must be freed or the answer will be cut short
C) The model automatically summarizes the history
D) The window silently expands to accommodate the request

Show answer

B — Input plus reserved output must fit in the same window. Here 8,500 exceeds 8,000, so either the input is reduced or the generation stops early.

6. Why does French prose usually cost more tokens than the equivalent English prose?

A) French is a more verbose language by nature
B) Tokenizer vocabularies are typically fitted on corpora dominated by English, so other languages fragment into more pieces
C) Accented characters are forbidden and must be escaped
D) The model translates French to English internally, doubling the count

Show answer

B — Merge rules are learned from training data. Text that is under-represented in that data is split into smaller, more numerous pieces, often 15 to 30 percent more for French, though the exact figure depends on the tokenizer.

7. What is the correct use of a characters-per-token rule of thumb?

A) To enforce the context limit at request time
B) To bill customers precisely
C) To size capacity and plan roughly, before measuring with the real tokenizer
D) To replace the tokenizer in production

Show answer

C — The ratio varies with language, formatting and content type. It is useful for early sizing, but any check that gates or truncates a request must use the actual tokenizer.

8. Why must the tokenizer and the chat template be pinned to the same model revision?

A) To reduce the download size
B) Because a mismatch changes token identifiers and special-token placement, silently degrading output
C) Because the licence requires it
D) To make inference deterministic across hardware

Show answer

B — A revision can change the vocabulary, the special tokens or the template formatting. A mismatched pair still runs, which is what makes the failure hard to notice.

9. The budget is exceeded. What is the sound engineering response?

A) Let the runtime truncate however it does by default
B) Define an explicit policy: what is dropped, what is summarized, and in which direction truncation happens
C) Always drop the system prompt first, since it is boilerplate
D) Retry the request until it fits

Show answer

B — Something is always dropped when the window overflows. Making the policy explicit turns an invisible, arbitrary loss into a reviewable design decision.

10. Which test set best exercises a tokenization pipeline?

A) Plain English prose only
B) The longest document available
C) Multilingual text, source code, emoji, special tokens, and boundary cases for truncation
D) Randomly generated character strings

Show answer

C — Failures concentrate at the edges: non-Latin scripts, byte-level fallbacks, template special tokens, and text cut in the middle of a multi-token character.

Scoring

Scoring: 8-10 correct — you can budget a request and defend the numbers; move on to the retrieval and context-assembly sessions. 5-7 correct — the concepts hold but the budget arithmetic is shaky; redo Lab B with a different request before continuing. Below 5 — re-read the key messages and repeat Lab A with your own text; the rest of the track assumes you reason in token space by default.

Learner course · Go to the exercises