Architecture and components

What you will be able to do

• Name the required Airflow 3 components and what each one does
• Match those names to docker compose ps in the lab
• Contrast LocalExecutor with Celery and Kubernetes (concepts only)

Why this matters for data work

When a DAG “does not appear” or a task sits in queued, you debug which process is stuck — not “Airflow” as a blob. The lab is small enough that you can see every box.

Airflow 3 split some jobs that Airflow 2 hid inside the scheduler or “webserver.” Learn the 3.x names; old blog posts still say webserver.

Concepts

A minimal Airflow 3 install is four ideas plus disk:

  ./dags/*.py
        |
        v
  DAG processor  ----serializes DAG---->  metadata DB (Postgres)
                                               ^
                                               |
  scheduler  --reads DAG structure, queues tasks
        |
        |  LocalExecutor: subprocess on the scheduler
        v
      task code  --state via-->  API server  --UI at :18080

Component	Job	In this lab
DAG files	Python you write	bind mount ./dags
DAG processor	Parse those files, write the graph into the database	airflow-dag-processor
Metadata DB	DAG runs, task state, connections	postgres (Airflow’s own DB)
Scheduler	Decide what should run now, hand work to the executor	airflow-scheduler
Executor	How the task process starts	LocalExecutor env on the scheduler
API server	UI, REST API, and the Task Execution API workers talk to	airflow-apiserver

Triggerer (not in this lab) watches deferrable sensors without occupying a worker. Skip it until you need sensors.

Workers in Celery or Kubernetes are separate containers or pods. LocalExecutor forks subprocesses inside the scheduler instead.

Metadata Postgres is not your warehouse

The postgres service stores Airflow’s brain: serialized DAGs, run history, XCom. It is not the retail lab on port 15432. Do not load analytics tables into it.

Executors you will hear about

Executor	Where tasks run	Typical use
Local	Subprocess next to the scheduler	Laptop, tiny single-node
Celery	Pool of workers + Redis/RabbitMQ	Always-on worker fleet
Kubernetes	A pod per task	Isolation and elastic scale

Official Airflow Docker docs ship a Celery compose file (Redis + worker). We use LocalExecutor so the lab fits a beginner machine. The DAG files you write are the same.

Practice on your machine

cd ~/dcubes/airflow-lab
docker compose ps

NAME                              STATUS
airflow-lab-postgres-1            Up (healthy)
airflow-lab-airflow-apiserver-1   Up (healthy)   0.0.0.0:18080->8080/tcp
airflow-lab-airflow-scheduler-1   Up (healthy)
airflow-lab-airflow-dag-processor-1 Up (healthy)

Exact names vary (project-service-1). You should not see Redis, Flower, or airflow-worker.

Inspect one service’s command:

docker compose logs airflow-scheduler --tail 20
docker compose logs airflow-dag-processor --tail 20
docker compose logs airflow-apiserver --tail 20

You are looking for “healthy” processes, not a specific sentence. Errors about missing dags/ files are normal while the folder is empty.

What you should see

• Four running containers (Postgres + three Airflow roles).
• UI still at http://127.0.0.1:18080.
• No Celery worker.

Common mistakes

• Calling the API server “the webserver.” Airflow 2 name. In 3 it is api-server.
• Assuming the scheduler parses DAG files. The DAG processor does. If a new file never shows up, read airflow-dag-processor logs first.
• Publishing metadata Postgres to 5432. This lab does not. Leave it internal.
• Loading example DAGs to “see if it works.” They clutter the list. Your first DAG in the next lesson is the check.

Next

Your first DAG