Why Apache Airflow

What you will be able to do

• Explain why data teams orchestrate jobs as DAGs instead of a pile of cron entries
• Name what Airflow owns (order, schedule, retries, logs) versus what it does not (compute, storage)
• See the map of this module

Why this matters for data work

A warehouse job is rarely one command. You extract a file, check row counts, transform, load a table, then ping a downstream model. If step three fails at 03:00 you need retries, a graph of what ran, and logs — not a silent crontab and a Slack message the next morning.

Apache Airflow is a workflow orchestrator. You describe work as a DAG (directed acyclic graph): tasks with dependencies, a schedule, and a place to watch runs. Python is the language of those files. You already practiced Python scripts and Docker Compose; Airflow is those two skills glued to a scheduler.

Cron vs an orchestrator

Cron (or a single script)	Airflow
One clock, one command	A graph: B waits for A
Logs if you remembered to redirect	Per-task logs in the UI
Failures are exit codes you might miss	Failed tasks stay red; you retry or clear
“Run yesterday again” is a custom flag	Backfill and reruns are first-class
Secrets in the crontab	Connections and variables (later modules)

Cron is still fine for one backup.sh. The moment jobs depend on each other, you want a DAG.

What Airflow is not

• Not a warehouse. Tables and grain live in Postgres, BigQuery, Snowflake — later on this roadmap.
• Not Spark. Airflow can trigger a Spark job; it does not replace distributed compute.
• Not a place to shove big data. Task return values (XCom) are for small handles: a path, a row count, a date. Files and tables carry the payload.
• Not production by default. The Compose stack in Getting Started is a laptop lab. Real clusters use a proper executor, secrets, and often Kubernetes.

Words you will keep using

DAG — the workflow file. Acyclic: no loops. Airflow 3 UI often writes Dag.

Task — one unit of work (a Python function, a bash command, a provider operator).

DAG run — one execution of the whole graph, for a logical date.

Operator — the template for a task. TaskFlow @task is the Pythonic form; BashOperator is the classic form.

Executor — how tasks actually run (subprocess on this machine vs Celery workers vs Kubernetes pods).

Map of this module

1. Getting Started — you already did this (~/dcubes/airflow-lab, UI on port 18080).
2. Architecture — which container does what.
3. Your first DAG — two TaskFlow tasks, manual trigger.
4. Operators and dependencies.
5. Schedules and runs.
6. When tasks fail.
7. Demo: First Airflow shift

Practice on your machine

Confirm the lab is still the working directory and the UI still answers.

cd ~/dcubes/airflow-lab
docker compose ps

# From your browser, not from this site:
# http://127.0.0.1:18080

If the stack is down:

docker compose up -d

What you should see

Three Airflow services plus Postgres Up. The Dags list is still empty. That is the right starting point.

Common mistakes

• Treating Airflow as “the place data lives.” Metadata is in lab Postgres; your CSV and warehouse tables are elsewhere.
• One giant task. Split extract / transform / load so a failure has a name.
• Skipping Docker. pip install apache-airflow on the host is a dependency maze. This module stays in Compose.

Next

Architecture and components