Your Airflow practice environment

What you will be able to do

• Create ~/dcubes/airflow-lab with dags, logs, and a Compose file
• Initialize the metadata database and start Airflow 3.3.1
• Log in to the UI at http://127.0.0.1:18080

Why this exists

Airflow is a set of processes (API server, scheduler, DAG processor) plus a Postgres database that stores DAG runs — not a single pip install on your laptop. Docker Compose is how you run that set locally. This site never starts Airflow for you.

Do Docker Getting Started first so docker and docker compose work. Keep the same terminal you used for Linux.

Lab folder

mkdir -p ~/dcubes/airflow-lab/{dags,logs,config,plugins,data}
cd ~/dcubes/airflow-lab
pwd

pwd should end in dcubes/airflow-lab.

User id for bind mounts

On Linux and WSL, files Airflow writes into logs/ should be owned by you, not root. Compose reads AIRFLOW_UID from .env.

cd ~/dcubes/airflow-lab
echo "AIRFLOW_UID=$(id -u)" > .env
cat .env

cd ~/dcubes/airflow-lab
echo "AIRFLOW_UID=50000" > .env
cat .env

Compose file

This lab uses LocalExecutor: the scheduler runs tasks as subprocesses on the same container. That is enough to learn DAGs. Production often uses Celery or Kubernetes — we cover those names in Architecture, not in this stack.

The image is pinned to Airflow 3.3.1. We turn example DAGs off so the UI starts empty. The UI is published on 18080 so it does not steal port 8080. Metadata Postgres stays on the Compose network (no host port) so it cannot collide with the SQL lab on 15432.

The &airflow-common block is YAML reuse: each Airflow service copies the same image, env, and volumes.

cd ~/dcubes/airflow-lab
cat > compose.yaml << 'EOF'
x-airflow-common: &airflow-common
  image: apache/airflow:3.3.1
  environment: &airflow-common-env
    AIRFLOW_UID: "${AIRFLOW_UID:-50000}"
    AIRFLOW__CORE__EXECUTOR: LocalExecutor
    AIRFLOW__CORE__AUTH_MANAGER: airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager
    AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
    AIRFLOW__CORE__FERNET_KEY: wtiOQFxjBjHaAAJpJnUIDYB5I0_nrG-mrNyOXqXdJaA=
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: "true"
    AIRFLOW__CORE__LOAD_EXAMPLES: "false"
    AIRFLOW__CORE__EXECUTION_API_SERVER_URL: http://airflow-apiserver:8080/execution/
    AIRFLOW__API_AUTH__JWT_SECRET: dcubes-airflow-lab-jwt-not-for-prod
    AIRFLOW__SCHEDULER__ENABLE_HEALTH_CHECK: "true"
    AIRFLOW_CONFIG: /opt/airflow/config/airflow.cfg
  volumes:
    - ./dags:/opt/airflow/dags
    - ./logs:/opt/airflow/logs
    - ./config:/opt/airflow/config
    - ./plugins:/opt/airflow/plugins
    - ./data:/opt/airflow/data
  user: "${AIRFLOW_UID:-50000}:0"
  depends_on:
    postgres:
      condition: service_healthy

services:
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: airflow
      POSTGRES_DB: airflow
    volumes:
      - postgres-db-volume:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U airflow -d airflow"]
      interval: 10s
      retries: 5
      start_period: 5s
    restart: unless-stopped

  airflow-init:
    <<: *airflow-common
    entrypoint: /bin/bash
    command:
      - -c
      - |
        mkdir -p /opt/airflow/{logs,dags,plugins,config,data}
        /entrypoint airflow version
        chown -R "${AIRFLOW_UID:-50000}:0" /opt/airflow/{logs,dags,plugins,config,data}
    environment:
      <<: *airflow-common-env
      _AIRFLOW_DB_MIGRATE: "true"
      _AIRFLOW_WWW_USER_CREATE: "true"
      _AIRFLOW_WWW_USER_USERNAME: airflow
      _AIRFLOW_WWW_USER_PASSWORD: airflow
    user: "0:0"
    restart: "no"

  airflow-apiserver:
    <<: *airflow-common
    command: api-server
    ports:
      - "18080:8080"
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8080/api/v2/monitor/health"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 40s
    restart: unless-stopped
    depends_on:
      postgres:
        condition: service_healthy
      airflow-init:
        condition: service_completed_successfully

  airflow-scheduler:
    <<: *airflow-common
    command: scheduler
    healthcheck:
      test: ["CMD-SHELL", 'airflow jobs check --job-type SchedulerJob --hostname "$${HOSTNAME}"']
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 40s
    restart: unless-stopped
    depends_on:
      postgres:
        condition: service_healthy
      airflow-init:
        condition: service_completed_successfully
      airflow-apiserver:
        condition: service_healthy

  airflow-dag-processor:
    <<: *airflow-common
    command: dag-processor
    healthcheck:
      test: ["CMD-SHELL", 'airflow jobs check --job-type DagProcessorJob --hostname "$${HOSTNAME}"']
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 40s
    restart: unless-stopped
    depends_on:
      postgres:
        condition: service_healthy
      airflow-init:
        condition: service_completed_successfully

volumes:
  postgres-db-volume:
EOF

The Fernet key and JWT secret in this file are lab-only. Never reuse them outside ~/dcubes/airflow-lab.

Initialize, then start

First up downloads apache/airflow:3.3.1 (often around a gigabyte). Wait for it.

cd ~/dcubes/airflow-lab
docker compose up airflow-init

What you should see: logs that mention creating directories, printing an Airflow version, migrating the database, and creating user airflow. The init container should exit 0.

Then start the long-running services:

docker compose up -d
docker compose ps

Wait until airflow-apiserver, airflow-scheduler, and airflow-dag-processor show healthy (this can take a minute after the image is already local).

Open the UI

In a browser on the same machine:

http://127.0.0.1:18080

Log in with:

username: airflow
password: airflow

You should land on the Dags list. It will be empty until you add a file in dags/ in a later lesson. That is success.

If 18080 is already in use, change the left side of "18080:8080" in compose.yaml and run docker compose up -d again.

Ready checklist

cd ~/dcubes/airflow-lab
docker compose ps

You want Postgres plus three Airflow services Up (init may already have exited). Then:

ls dags logs config plugins data

Those five directories exist. You can sign in at http://127.0.0.1:18080.

Stop and start tomorrow

Stop (keeps the metadata volume):

cd ~/dcubes/airflow-lab
docker compose down

Start again:

cd ~/dcubes/airflow-lab
docker compose up -d

Wipe the metadata database (you will re-run airflow-init):

docker compose down -v
docker compose up airflow-init
docker compose up -d

Common mistakes

• Docker Desktop not running — Cannot connect to the Docker daemon. Start Desktop (or the engine) and wait.
• Skipping airflow-init — API server crash-loops until the database has tables. Run init once per volume.
• Port 18080 busy — change the published port, or stop whatever owns 18080.
• Opening localhost:8080 — this lab publishes 18080. 8080 is inside the container.
• Expecting example DAGs — we set LOAD_EXAMPLES to false on purpose.

Next

Why Apache Airflow