Operators and dependencies

What you will be able to do

• Chain tasks with function calls and with >>
• Run a BashOperator next to a TaskFlow task
• Explain why XCom is for small values, not data frames

Why this matters for data work

Production DAGs mix styles: a Python transform you wrote, a bash that calls dbt, a provider operator that talks to S3. You need to read both TaskFlow and the older Operator + >> form. You also need a hard rule: pass paths and counts between tasks, not the dataset itself.

Concepts

TaskFlow call — shout(say_hello()) means shout waits for say_hello and receives its return value.

Bitshift — extract >> transform >> load is the classic dependency line. No data is passed unless you use XCom on purpose.

BashOperator — runs a shell string inside the task process (here: inside the scheduler container). Import from the standard provider:

from airflow.providers.standard.operators.bash import BashOperator

XCom — “cross-communication.” TaskFlow return values are XComs. Inspect them in the UI under XCom. They belong in the metadata database. A 2 MB JSON blob is already rude; a pandas frame is a bug.

Idempotence — running the task twice should not double-write. Prefer “write this file/table for this date” over “append forever.”

Practice on your machine

Add a second DAG that mixes TaskFlow and bash.

cd ~/dcubes/airflow-lab
cat > dags/mix_operators.py << 'EOF'
from datetime import datetime

from airflow.providers.standard.operators.bash import BashOperator
from airflow.sdk import dag, task


@dag(
    schedule=None,
    start_date=datetime(2024, 1, 1),
    catchup=False,
    tags=["dcubes"],
)
def mix_operators():
    @task
    def path_for_today() -> str:
        return "/opt/airflow/data"

    list_data = BashOperator(
        task_id="list_data",
        bash_command="ls -la {{ ti.xcom_pull(task_ids='path_for_today') }}",
    )

    path_for_today() >> list_data


mix_operators()
EOF

Wait for mix_operators in the UI. Unpause, trigger, open the graph.

path_for_today succeeds and stores a short string XCom. list_data runs ls on /opt/airflow/data (empty except maybe lost+found). The log should show a directory listing, not an error.

Optional: open the XCom tab on path_for_today and confirm the value is the path string.

What you should see

path_for_today  >>  list_data
XCom: /opt/airflow/data
list_data log: a listing of /opt/airflow/data

hello_airflow from the previous lesson is still there. Leave it.

Common mistakes

• Returning a list of rows from extract. Write a file under /opt/airflow/data (or object storage, later) and return the path.
• BashOperator with from airflow.operators.bash import .... Airflow 3 moved it to airflow.providers.standard.
• Jinja in TaskFlow Python. Bash strings are templated ({{ ti.xcom_pull(...) }}). Plain @task functions are Python — use arguments, not {{ }}.
• Cycles. a >> b >> a will not parse. The graph must stay acyclic.

Next

Schedules and runs