skip to content
promptdojo
_
[ save your spot ]
[ follow on x ]
Tokenizers and context budget — step 3 of 7
ch 44 · cnns, transformers, and useful llm internals
3/7
promptdojo
_
›
phase 06 · ml systems
›
ch 44 · cnns, transformers, and useful llm internals
lesson 2 of 5 · tokenizers and context budget
step 3 of 7 in this lesson
Greedy longest-match tokenization against a 5-piece vocabulary. What does this print?
your prediction
Check prediction
Or run the code →
Show hint 1
ch 44 · cnns, transformers, and useful llm internals
3/7
promptdojo
_
›
phase 06 · ml systems
›
ch 44 · cnns, transformers, and useful llm internals
lesson 2 of 5 · tokenizers and context budget
step 3 of 7 in this lesson
Greedy longest-match tokenization against a 5-piece vocabulary. What does this print?
your prediction
Check prediction
Or run the code →
Show hint 1
park a thought