The four breakage classes — sort any LLM failure before you touch the prompt (step 9/9) · debugging broken ai output

Checkpoint

One last thing before we move on. Same surface as a write step — but the lesson doesn't complete until this passes.

Final drill. You got three user-reported failures from the support inbox this morning. Each has a trace attached. Classify them so the right team gets the right ticket — retrieval team, prompt team, ML team (hallucination), or backend team (parse).

Write classify_failure(trace) using the same priority order from the previous step:

retrieved_chunks non-empty + retrieved_chunks_match_query False → "retrieval"
raw_output != output_after_postprocess → "parse"
retrieved_chunks empty → "hallucination"
else → "prompt"

Then print one line per trace in this format: ticket_<id>: <class>

Expected output:

ticket_8821: retrieval
ticket_8822: prompt
ticket_8823: parse

Checkpoint

One last thing before we move on. Same surface as a write step — but the lesson doesn't complete until this passes.

Write classify_failure(trace) using the same priority order from the previous step:

retrieved_chunks non-empty + retrieved_chunks_match_query False → "retrieval"
raw_output != output_after_postprocess → "parse"
retrieved_chunks empty → "hallucination"
else → "prompt"

Then print one line per trace in this format: ticket_<id>: <class>

Expected output:

ticket_8821: retrieval
ticket_8822: prompt
ticket_8823: parse

full-screen editor opens — close anytime to keep reading.

The four breakage classes — sort any LLM failure before you touch the prompt — step 9 of 9