Airflow module
Data Engineering · Module 01
Apache Airflow
Orchestrate batch jobs as DAGs: install Airflow with Docker, learn the components, write a first TaskFlow DAG, and debug a failed run — on your machine.
Beginner7 lessons~4 hours plus a demo
Pipelines are graphs: extract, then transform, then load — on a schedule, with retries and logs. Apache Airflow is how many data teams write that graph in Python and watch it run. This is the first Data Engineering module, after the foundations spine.
You will start Airflow with Docker Compose (LocalExecutor), map the API server, scheduler, DAG processor, and metadata database, then write TaskFlow DAGs on your machine. The site never starts Airflow for you.
Start here
Finish Docker Getting Started so docker compose works, and keep a terminal from Linux Getting Started. Python and Git from the foundations modules help you read the DAG files. Then do this module’s Getting Started — we create~/dcubes/airflow-lab and bring the stack up.
On-ramp
ReadyYour Airflow practice environment
Start a local Apache Airflow 3 stack with Docker Compose and open the UI on your machine.
45 min
Lesson 1
ReadyWhy Apache Airflow
What Airflow is for in data work — DAGs, retries, and logs — and what it is not.
20 min
Lesson 2
ReadyArchitecture and components
Map the Airflow 3 API server, scheduler, DAG processor, executor, and metadata database to your Compose services.
30 min
Lesson 3
ReadyYour first DAG
Write a two-task TaskFlow DAG, wait for the processor to parse it, unpause, and trigger a run in the UI.
40 min
Lesson 4
ReadyOperators and dependencies
Chain TaskFlow tasks, mix in a BashOperator, and keep XCom payloads small.
30 min
Lesson 5
ReadySchedules and runs
Cron and timetable schedules, pause vs trigger, catchup, and why a DAG did not run when you expected.
25 min
Lesson 6
ReadyWhen tasks fail
Read task logs, set retries, clear a failed run, and fix a DAG that never appears.
30 min