promptdojo_

Tokenizers and context budget — step 5 of 7

This tokenizer bills "unhappiness" at ELEVEN tokens even though 'un', 'happi', and 'ness' are all in the vocabulary. Look at the size loop: it tries the SHORTEST candidate first, so the single letter 'u' always wins before 'un' is ever considered. Greedy tokenization must try longest-first. Fix the loop direction.

The break is on line 7 — but read the whole snippet first.

full-screen editor opens — close anytime to keep reading.