1/18
Choosing an LLM surface
Chat, search, code, and voice — the same model, four different context policies.
Presenter notes
Open by naming the tension the session resolves: people compare products when the decision that matters is what enters the context window. Promise a selection rule by the end.
2/18
The question behind every surface
What is actually inside the context window right now?
Presenter notes
Ask for a show of hands: who can list what is in the window on their last prompt? Almost nobody can. That gap is the session.
3/18
A surface is a context policy
- Plain chat: a self-contained token stream, bounded by a knowledge cutoff
- Search: retrieved passages are injected
- Code: repository files are injected
- Documents: an uploaded corpus is injected
Presenter notes
Draw the window on the whiteboard and physically add each source as you name it. Keep the drawing up for the rest of the session.
4/18
Choosing is not clicking
Selecting a surface decides what occupies the window — an engineering decision, not a UI preference.
Presenter notes
This is the sentence to repeat. If learners remember one line, make it this one.
5/18
Failure mode 1: stale recall
With no retrieval, the model answers from parameters fixed at training time.
Presenter notes
Emphasise that fluency is not calibration: the confident tone is identical whether the fact is current or three years old.
6/18
Lab A — the cutoff experiment
Same question, two surfaces. Record both answers, then open every citation.
Presenter notes
20 minutes, in pairs. Circulate and push people to actually open the sources rather than reading the quoted snippet.
7/18
Failure mode 2: unfaithful summarization
- Which sources were found?
- Which passages were selected?
- Is the summary faithful to them?
Presenter notes
Debrief Lab A here. Ask specifically who found a claim supported by no single cited source — there is usually at least one pair.
8/18
A citation is a promise, not proof
The claim is verified only when a human opens the source.
Presenter notes
Anticipate the pushback that this is slow. Answer: you only need to verify the claims you will act on.
9/18
Retrieval is a trade, not an upgrade
| Surface | Risk it removes | Risk it adds |
|---|---|---|
| No retrieval | Source noise, latency, cost | Stale or fabricated recall |
| Retrieval | Staleness | Bad sources, unfaithful summary |
| Code execution | Mental-arithmetic error | Wrong program, hidden assumption |
Presenter notes
Give a counter-example where retrieval is the wrong choice: a stable, well-known fact where it only adds latency and a mediocre source.
10/18
Failure mode 3: the unverified computation
When a surface runs code, the arithmetic becomes exact — and a new artifact appears.
Presenter notes
Segue into Lab B. The shift from "predicted number" to "executed program" is the conceptual jump.
11/18
Read the program, not the number
- Which rows were selected?
- How were nulls and malformed values handled?
- Which column was actually used?
Presenter notes
These three questions catch the majority of silent errors in delegated computation.
12/18
Lab B — delegated computation
One aggregate, one extrapolation. Read both programs before reading either result.
Presenter notes
20 minutes, in pairs. Keep the dataset tiny so this does not become a debugging session.
13/18
Exact arithmetic, assumed world
A linear fit computed perfectly still assumes the trend continues. The data cannot say that.
Presenter notes
Debrief Lab B. Insist that the assumption be written next to the figure whenever the figure is reported.
14/18
Context is metered
Attention and decoding scale with the tokens held in the window — paid on every turn.
Presenter notes
Use the tokenizer interactive live here. Show the full document, then the excerpt that would actually have answered the question.
15/18
Bound what you paste
The excerpt that answers the question is usually a small fraction of the document you attached.
Presenter notes
Have two or three learners report their ratio from Lab C. The gap is consistently larger than people expect.
16/18
Persistence is a privacy surface
- Memory carries state across sessions
- Custom instructions apply silently to every turn
- Saved configurations accumulate by default
Presenter notes
Audit a memory panel live on the projector, using a throwaway account. Never use a personal one.
17/18
The selection rule
Choose the cheapest surface that can answer correctly, then perform the verification that surface demands.
Presenter notes
This is the deliverable of the session. Have learners write it down before the closing slide.
18/18
Honest boundary
- Consumer features change monthly; any comparison is dated
- This session teaches selection criteria, not a leaderboard
- Verification effort scales with the consequence of being wrong
Presenter notes
Close on the attribution: the framing follows Karpathy's public lecture, restated and adapted here. Point learners to the original for his full treatment.