Behind the pipelines course
How we built the cohort project for the data pipelines course, and why it runs against real traffic.
By Priya Nair
When we built the data pipelines course, the first decision was the project. A toy dataset gives you a toy pipeline. We wanted a project that breaks the way production breaks, which meant real traffic, real drift, and real on-call.
Why real traffic
A pipeline against a static CSV cannot teach you about backfill, drift, or the SLA you owe a downstream team. A pipeline against a live (small, sandboxed) dataset can. The cohort runs the same pipeline against the same source for two weeks and rotates on-call.
The tradeoff is cost. Real traffic means a real environment, which means more setup. We cut setup to a single script and a single command, so the cohort spends time on the pipeline, not the environment.
More reading
- The pipeline after the notebook — 7 min
- Design systems nobody uses — 6 min