When tasks fail

What you will be able to do

• Trigger a failing task and read its log in the UI
• Add retries and clear a task to rerun
• Use dag-processor logs when a DAG file does not show up

Why this matters for data work

Orchestration is mostly failure handling. The happy path is a green graph. The job is: see red, read the log, decide retry vs skip vs fix the file, leave an audit trail. Airflow is useful because that loop has names.

Concepts

Task states — queued, running, success, failed, upstream_failed, skipped, up_for_retry.

Retries — @task(retries=2) or retries= on an operator. A flake (network blip) should retry; a KeyError in your code should not retry five times hoping the key appears.

Logs — UI task log, and files under ~/dcubes/airflow-lab/logs. Start in the UI.

Clear — from the grid or graph, clear a failed task (and optionally downstream) to run it again with the same DAG run. You do not always need a new Trigger.

Import errors — if the DAG processor cannot import the file, the DAG is absent from the list (or listed under import errors). This is not a failed task. It is a broken module.

OOM — Docker Desktop out of memory kills containers. Scheduler health goes red. Free RAM; this is not a Python traceback.

Practice on your machine

Write a DAG that fails on purpose, then fix it.

cd ~/dcubes/airflow-lab
cat > dags/fail_then_fix.py << 'EOF'
from datetime import datetime

from airflow.sdk import dag, task


@dag(
    schedule=None,
    start_date=datetime(2024, 1, 1),
    catchup=False,
    tags=["dcubes"],
)
def fail_then_fix():
    @task(retries=0)
    def boom() -> None:
        raise ValueError("planted failure — you will fix this")

    boom()


fail_then_fix()
EOF

Wait for fail_then_fix. Unpause, trigger. boom should go red.

Open Logs. You should see ValueError: planted failure — you will fix this.

Now fix the file:

cd ~/dcubes/airflow-lab
cat > dags/fail_then_fix.py << 'EOF'
from datetime import datetime

from airflow.sdk import dag, task


@dag(
    schedule=None,
    start_date=datetime(2024, 1, 1),
    catchup=False,
    tags=["dcubes"],
)
def fail_then_fix():
    @task(retries=0)
    def boom() -> None:
        print("all good")

    boom()


fail_then_fix()
EOF

Wait a few seconds for parse. In the UI, clear the failed boom task on the existing run (or Trigger a new run). It should succeed and print all good.

Import error drill

Break the file:

cd ~/dcubes/airflow-lab
echo "this is not python" > dags/fail_then_fix.py

Wait, then:

docker compose logs airflow-dag-processor --tail 30

You want a traceback that names fail_then_fix.py. Restore the working file from the “all good” block above (or delete the DAG if you are done with it):

rm ~/dcubes/airflow-lab/dags/fail_then_fix.py

What you should see

• A red task with a readable ValueError.
• After the fix, a green task on clear or a new run.
• Processor logs that name a broken file when the DAG vanishes.

Common mistakes

• Re-triggering instead of reading the log. The error is already there.
• retries=5 on a syntax bug. You wait. Fix the code.
• Ignoring processor logs. “DAG not in the UI” is parse, not schedule.
• Full disk / no RAM. docker compose ps unhealthy. docker compose down other labs; give Desktop more memory.

Next

Capstone: First Airflow shift