promptdojo_

Tokenizers and context budget — step 6 of 7

Write the greedy longest-match tokenize(text) against the given vocab, then price two same-order-of-length strings: ordinary prose and a structured ID. Print f"{text!r}: {n} tokens for {len(text)} chars" for each. The prose rides whole-word vocabulary entries; the ID shatters into per-character pieces — chars-per-token is not a constant.

full-screen editor opens — close anytime to keep reading.