CSV, JSONL, Parquet - which file earns the job — step 1 of 7

CSV vs JSONL vs Parquet: pick the file for the job

Chapter 10 introduced CSV and JSONL as file formats. A dataset pipeline forces the question: which one do you store your data in at each stage? The answer is a tradeoff, not a favorite.

What each format actually buys you

• CSV — universal, human-openable, and typeless. Run the editor: a float and a bool go in, strings come out. Every consumer re-parses types, and every consumer can do it differently. Nested data doesn't fit at all.
• JSONL — one JSON object per line (chapter 10). Keeps types and nesting, streams line-by-line, appends cheaply. The workhorse for raw event capture and LLM training/eval data. Costs: verbose (keys repeated on every line), and reading one column means parsing every row.
• Parquet — a columnar, typed, compressed binary format. The schema travels with the file, columns compress tightly, and a reader can load just the two columns it needs instead of the whole table. The standard for analytics-stage data that pandas and every warehouse can read (pd.read_parquet). Costs: not human-openable, needs a library, and appending a few rows means writing new files rather than tacking on a line.

The pipeline shape that uses all three

A common, sane layout — not a law, but a good default:

ingest (JSONL, append-only, raw and ugly)
   → clean/typed tables (Parquet, columnar, schema'd)
      → small hand-off extracts (CSV, when a human or a
        spreadsheet is the consumer)

Raw stays raw so you can always re-derive; the typed middle layer is where analysis and training read from; CSV exists at the edges where humans live.

Where AI specifically gets this wrong

• CSV as the system of record. Types silently degrade (this editor's demo), and one comma inside a value untangles a naive parser.
• Hand-rolled CSV parsing. If Cursor writes line.split(","), stop it — chapter 10's rule: use the csv module, or better, don't use CSV here at all.
• Re-parsing raw JSONL on every run. Fine at 10k rows, painful at 10M. The typed middle layer exists so the expensive parse happens once.
• Guessing schemas per file. Ten CSVs, ten inferred schemas, one column that's int in nine files and str in the tenth. Parquet's travelling schema — or an explicit contract (lesson 03) — is the fix.