Demo · Lesson player

Production data pipelines with Airflow

A demo of the player. Lesson list, searchable transcript, and mark-as-done that saves to your browser and survives a reload.

4 of 9 lessons done

Transcript

0:00
Priya
A notebook is a wonderful place to find an answer. It is a terrible place to keep one.
0:18
Priya
The difference is what the notebook was built for: exploration, not operation.
0:34
Priya
When you move a notebook into a schedule without changing how it thinks, you inherit every assumption it made once and only once.
0:58
Priya
The first stage of a pipeline is exciting. The second stage is when someone else depends on it.
1:15
Priya
That second stage is where it breaks, because the notebook never had to answer to anyone.
1:34
Priya
The fix is not more tooling. The fix is four habits that make the pipeline legible to the next person, including future you.
1:52
Priya
First: name your stages so a stranger can read them in order. Second: write the contract before you write the transform.
2:10
Priya
Third: make retries safe by making them idempotent. Fourth: alert on the SLA, not on the error.
2:28
Priya
A data contract is a small, boring document. It says what fields will arrive, in what shape, and what it means when they do not.
2:48
Priya
Writing it before the transform forces you to decide what you are promising.
3:05
Priya
It also gives the downstream team something to push back on before you ship, which is the cheapest moment to change.