Tokenizers and the context budget
Chapter 13 told you models bill and think in tokens. This lesson shows what a token actually is — because token count is a budget you engineer against everywhere: prompts, RAG chunks, context rot, training cost.
Run the miniature tokenizer
Real tokenizers (the BPE family — byte-pair encoding — used across modern LLMs) build a vocabulary of frequent character chunks and encode text into them. The editor's toy uses greedy longest-match against a hand-made vocab — a simplification (true BPE applies learned merge rules), but the cost behavior it demonstrates is the same:
"unhappiness"→un / happi / ness— 3 tokens. Subwords let a fixed vocabulary cover unlimited words by composition."the cat"→ whole common words are single tokens (frequency earns vocabulary slots)."xyzzy"→ falls apart into per-character junk — rare strings are expensive. This is why weird IDs, base64 blobs, and unusual languages eat context: they tokenize inefficiently.
Two working rules that survive contact with real tokenizers: token
counts are roughly proportional to text length for ordinary
English prose, and anything rare or structured costs more than it
looks. Never hand-estimate for billing-critical paths — count with
the model's actual tokenizer (APIs return usage, chapter 13).
The context budget
The context window (chapter 22's 128k/200k/1M numbers) is
measured in these tokens, and it's a budget shared by everything:
system prompt, tools, retrieved chunks, history, and the answer.
Chapter 30's context-rot lesson showed that spending it all is
its own failure. So the discipline: know your big line items
(chapter 23's caching table told you where the stable spend is),
retrieve chunks by relevance rather than stuffing (chapter 22),
and compact history before the budget forces it (chapter 19's
/compact).
Sequence models also have a related training-time knob: the maximum sequence length they were trained to handle. Attention's cost grows fast with sequence length (next lesson shows the mechanism), which is why long context is an engineering feature, not a free dial.
Where AI specifically gets this wrong
- Characters ≈ tokens. Generated cost estimates that divide characters by a magic constant drift badly on code, JSON, and non-English text. Count, don't estimate.
- Chunking by characters without token awareness. A "500-char chunk" of dense JSON and one of prose are wildly different token spends — chapter 22's chunking works better in token units. Then there's the diagnostic miss: ignoring tokenization when things look weird. Models mangling exact strings, IDs, or arithmetic often trace to how those strings tokenized. When output garbles a specific string, look at its tokens first.