The pipeline after the notebook

A notebook is a wonderful place to find an answer. It is a terrible place to keep one. The difference is what the notebook was built for: exploration, not operation. When you move a notebook into a schedule without changing how it thinks, you inherit every assumption the notebook made once and only once.

The second stage is where it breaks

The first stage of a pipeline is exciting. The data arrives, the transform runs, something lands in a table. The second stage is when someone else depends on it. That is the stage that breaks, because the notebook never had to answer to anyone.

The fix is not more tooling. The fix is four habits that make the pipeline legible to the next person, including future you.

• Name your stages so a stranger can read them in order
• Write the contract before you write the transform
• Make retries safe by making them idempotent
• Alert on the SLA, not on the error

Contracts before transforms

A data contract is a small, boring document. It says what fields will arrive, in what shape, and what it means when they do not. Writing it before the transform forces you to decide what you are promising. It also gives the downstream team something to push back on before you ship, which is the cheapest moment to change.

The pipeline you can defend at 3am is the one you wrote a contract for at noon.

In the cohort we write a contract for a real dataset before we touch Airflow. It is the single change that moves pipelines from fragile to durable, and it costs almost nothing to start.