promptdojo_

GitHub Actions for model checks — what a green check does and doesn't mean — step 1 of 7

ch 17 · git and github cli1/7

CI runs your checks on every push — but only the checks you wrote

You ask Claude to "add CI" and it drops a file into .github/workflows/. From then on, every push gets a green check or a red X next to it on GitHub. Feels like safety. Now the uncomfortable part: for an ML project, the default setup checks almost nothing you actually care about.

What a workflow file actually is

A workflow is a YAML file with three parts:

name: model-checks
on: [pull_request]          # WHEN to run
jobs:
  check:
    runs-on: ubuntu-latest  # a fresh rented Linux box
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - run: pytest -q

No magic. on says when it triggers (push, pull_request, a cron schedule). Each run: line is a shell command executed on a throwaway machine. A step passes when its command exits with code 0. That exit code is the entire API of CI. Green check = every step exited 0. That's all a green check has ever meant.

Why "tests pass" is not "model is fine"

pytest exercises your code: does preprocess() handle empty input, does the API return 200. Your model's quality lives somewhere pytest never looks — in the data and the weights. Accuracy can crater from 91% to 74% while every unit test stays green, because no test asserts anything about predictions. A schema change upstream, a "harmless" preprocessing tweak, a retrained checkpoint — all invisible to a code-only workflow.

The fix is to add steps that fail on purpose:

  • A data validation step — a script that checks row counts, null rates, and column schema, and calls sys.exit(1) when the data looks wrong.
  • An eval gate — a script that runs the model on a fixed eval set, computes the score, and exits nonzero if it's below a threshold you chose. Below 0.80? The PR physically cannot merge.

Each is ten lines of Python plus one run: line in the YAML.

Reading a red X

Click the X. GitHub shows the job's steps in order: green ticks down to the first red one, then grey "skipped" for everything after — a job stops at the first failing step. Open the red step's log and read from the bottom up; the exit reason is almost always in the last 20 lines. Don't rerun it hoping. The X is the system doing its job.

Run the editor. We parse a model-checks workflow into a dict — the shape we'll evaluate gates against for the rest of this lesson.

open the editor to run it.