Introduction to Apache Airflow® Dags

In Apache Airflow®, a Dag is a data pipeline or workflow. Dags are the main organizational unit in Airflow; they contain a collection of tasks and dependencies that you want to execute on a schedule.

Without a Dag, pipeline steps run independently with no awareness of each other. If an extraction step fails, downstream transformations might still run on stale or missing data. Dags solve this by defining explicit dependencies between tasks, so failures halt dependent steps and alert you to the problem.

A Dag is defined in Python code and visualized in the Airflow UI. Dags can be as simple as a single task or as complex as hundreds or thousands of tasks with complicated dependencies.

The following screenshot shows a complex Dag run graph in the Airflow UI. After reading this guide, you’ll be able to understand the elements in this graph, as well as know how to define Dags and use Dag parameters.

Assumed knowledge

To get the most out of this guide, you should have an understanding of:

• What Airflow is. See Introduction to Apache Airflow.

What is a Dag?

A Dag (directed acyclic graph) is a mathematical structure consisting of nodes and edges. In Airflow, a Dag represents a data pipeline or workflow with a start and an end.

The mathematical properties of Dags make them useful for building data pipelines:

• Directed: There is a clear direction of flow between tasks. A task can be either upstream, downstream, or parallel to another task.

  

• Acyclic: There are no circular dependencies in a Dag. This means that a task cannot depend on itself, nor can it depend on a task that ultimately depends on it.

  

• Graph: A Dag is a graph, which is a structure consisting of nodes and edges. In Airflow, nodes are tasks and edges are dependencies between tasks. Defining workflows as graphs helps you visualize the entire workflow in a way that’s easy to navigate and conceptualize.

Beyond these requirements, a Dag can be as simple or as complicated as you need. You can define tasks that run in parallel or sequentially, implement conditional branches, and visually group tasks together in task groups.

For example, a common Dag might extract data from an API, anonymize sensitive fields, check for duplicate records, insert cleaned data into a database, and run a SQL query to update a dashboard. Each of these steps is a task, and the Dag ensures they run in the correct order.

Each task in a Dag should perform one unit of work. Tasks can be anything from a simple Python function to a complex data transformation or a call to an external service. They are defined using Airflow operators or Airflow decorators. The dependencies between tasks can be set in different ways (see Managing Dependencies).

The following screenshot shows a simple Dag graph with 3 sequential tasks.

1
"""
2
## Astronaut ETL example DAG
3
4
This DAG queries the list of astronauts currently in space from the
5
Open Notify API and prints each astronaut's name and flying craft.
6
7
There are three tasks, one to get the data from the API and save the results,
8
one to print the results and a final task to react. The first two tasks are
9
written in Python using Airflow's TaskFlow API, which allows you to easily turn
10
Python functions into Airflow tasks, and automatically infer dependencies and pass data.
11
12
The second task uses dynamic task mapping to create a copy of the task for
13
each Astronaut in the list retrieved from the API. This list will change
14
depending on how many Astronauts are in space, and the DAG will adjust
15
accordingly each time it runs.
16
17
The third task is defined using the BashOperator, which is a traditional
18
operator, allowing you to run a bash command.
19
"""
20
21
from airflow.decorators import dag, task
22
from airflow.operators.bash import BashOperator
23
from airflow.models.baseoperator import chain
24
from pendulum import datetime
25
import requests
26
27
28
# Define the basic parameters of the DAG, like schedule and start_date
29
@dag(
30
    start_date=datetime(2024, 1, 1),
31
    schedule="@daily",
32
    catchup=False,
33
    doc_md=__doc__,
34
    default_args={"owner": "Astro", "retries": 3},
35
    tags=["example"],
36
)
37
def example_astronauts_three_tasks():
38
    # Define tasks
39
    @task
40
    def get_astronauts(**context) -> list[dict]:
41
        """
42
        This task uses the requests library to retrieve a list of Astronauts
43
        currently in space. The results are pushed to XCom with a specific key
44
        so they can be used in a downstream pipeline. The task returns a list
45
        of Astronauts to be used in the next task.
46
        """
47
        r = requests.get("http://api.open-notify.org/astros.json")
48
        number_of_people_in_space = r.json()["number"]
49
        list_of_people_in_space = r.json()["people"]
50
51
        context["ti"].xcom_push(
52
            key="number_of_people_in_space", value=number_of_people_in_space
53
        )
54
        return list_of_people_in_space
55
56
    @task
57
    def print_astronaut_craft(greeting: str, person_in_space: dict) -> None:
58
        """
59
        This task creates a print statement with the name of an
60
        Astronaut in space and the craft they are flying on from
61
        the API request results of the previous task, along with a
62
        greeting which is hard-coded in this example.
63
        """
64
        craft = person_in_space["craft"]
65
        name = person_in_space["name"]
66
67
        print(f"{name} is currently in space flying on the {craft}! {greeting}")
68
69
    # define a task using a traditional operator
70
    # this task will run a bash command
71
    print_reaction = BashOperator(
72
        task_id="print_reaction",
73
        bash_command="echo This is awesome!",
74
    )
75
76
    # Use dynamic task mapping to run the print_astronaut_craft task for each
77
    # Astronaut in space
78
    print_astronaut_craft_obj = print_astronaut_craft.partial(
79
        greeting="Hello! :)"
80
    ).expand(
81
        person_in_space=get_astronauts()  # Define dependencies using TaskFlow API syntax
82
    )
83
84
    # set the dependency between the second and third task explicitly
85
    chain(print_astronaut_craft_obj, print_reaction)
86
87
88
# Instantiate the DAG
89
example_astronauts_three_tasks()

Why use Dags for data pipelines?

Structuring your data pipelines as Dags provides several advantages over running disconnected scripts or using separate interface-driven tools:

• Reliability: Tasks execute in a guaranteed order every run. If a task fails, dependent downstream tasks don’t execute, preventing data corruption from stale or incomplete data.
• Visibility: The Dag structure gives you a visual map of your entire pipeline in the Airflow UI, making it easier to debug failures and understand data flow at a glance.
• Testability: Because Dags define deterministic execution paths, you can test individual tasks in isolation and validate expected outcomes against known inputs.

What is a Dag run?

A Dag run is an instance of a Dag running at a specific point in time. A task instance is an instance of a task running at a specific point in time. Each Dag run has a unique run_id and contains one or more task instances. The history of previous Dag runs is stored in the Airflow metadata database.

In the Airflow UI, you can view previous runs of a Dag in the Grid view and select individual Dag runs by clicking on their respective duration bar.

A Dag run graph looks similar to the Dag graph, but includes additional information about the status of each task instance in the Dag run.

Learn more about how to navigate the Airflow UI in the An introduction to the Airflow UI guide.

Each Dag run is associated with a Dag version. Each time you make structural changes to a Dag and create a new Dag run, the Dag version is incremented. This allows you to track changes to the Dag over time and better understand previous Dag runs. For more information on Dag versions, see Dag Versioning and Dag Bundles.

Dag run properties

A Dag run graph in the Airflow UI contains information about the Dag run, as well as the status of each task instance in the Dag run. The following screenshot shows the same Dag as in the previous section, but with annotations explaining the different elements of the graph.

• dag_id: The unique identifier of the Dag.
• logical date: The point in time after which this particular Dag run can run. This date and time is not necessarily the same as the actual moment the Dag run is executed. See Scheduling for more information.
• task_id: The unique identifier of the task.
• task state: The status of the task instance in the Dag run. Possible states are running, success, failed, skipped, restarting, up_for_retry, upstream_failed, queued, scheduled, none, removed, deferred, and up_for_reschedule, they each cause the border of the node to be colored differently. See the OSS documentation on task instances for an explanation of each state.

There are four ways you can trigger a Dag run:

• Backfill: Backfilling is a mechanism by which you can create several Dag runs for dates in the past using the Airflow UI, API or CLI. Backfilled Dag runs include a curved arrow on their Dag run duration bar.
• Scheduled: Dag runs created based on a Dag’s schedule (for example @daily, @hourly) are created by the Airflow scheduler. The Dag run duration bar does not have an additional icon.
• Manual: You can trigger manual runs of a Dag in the Airflow UI, or by using the Airflow CLI or API. Manually triggered Dag runs include a play icon on the Dag run duration bar.
• Asset triggered: Dags can be scheduled using data-aware scheduling. This means a Dag runs as soon as one or more Airflow assets are updated. These updates can come from tasks inside of the same Airflow instance, a call to the Airflow REST API, be made manually using the Airflow UI, or triggered based on messages in a message queue. The Dag run duration bar includes an asset icon.

A Dag run can have the following statuses:

• Queued: The time after which the Dag run can be created has passed but the scheduler has not created task instances for it yet.
• Running: The Dag run is eligible to have task instances scheduled.
• Success: All task instances are in a terminal state (success, skipped, failed or upstream_failed) and all leaf tasks (tasks with no downstream tasks) are either in the state success or skipped. The duration bar of a successful Dag run is green.
• Failed: All task instances are in a terminal state and at least one leaf task is in the state failed or upstream_failed. The duration bar of a failed Dag run is red.

Complex Dag runs

When you start writing more complex Dags, you will see additional Airflow features that are visualized in the Dag run graph. The following screenshot shows the same complex Dag as in the overview but with annotations explaining the different elements of the graph. Don’t worry if you don’t know about all these features yet. You will learn about them as you become more familiar with Airflow.

1
"""
2
## Example of a complex DAG structure
3
4
This DAG demonstrates how to set up a complex structures including:
5
- Branches with Labels
6
- Task Groups
7
- Dynamically mapped tasks
8
- Dynamically mapped task groups
9
10
The tasks themselves are empty or simple bash statements.
11
"""
12
13
from airflow.providers.standard.operators.bash import BashOperator
14
from airflow.providers.standard.operators.empty import EmptyOperator
15
from airflow.sdk import Asset, Label, chain, chain_linear, dag, task, task_group
16
17
18
# Define the DAG
19
@dag
20
def complex_dag_structure():
21
22
    start = EmptyOperator(task_id="start")
23
24
    # Dynamically mapped tasks: .partial() contains all parameters that stay the same
25
    # between mapped task instances. .expand() contains the parameters that change
26
    # one mapped task instance will be created for each value in the list provided to `bash_command`
27
    sales_data_extract = BashOperator.partial(task_id="sales_data_extract").expand(
28
        bash_command=["echo 1", "echo 2", "echo 3", "echo 4"]
29
    )
30
    internal_api_extract = BashOperator.partial(task_id="internal_api_extract").expand(
31
        bash_command=["echo 1", "echo 2", "echo 3", "echo 4"]
32
    )
33
34
    # Branch task that picks which branch to follow
35
    @task.branch
36
    def determine_load_type() -> str:
37
        """Randomly choose a branch. The return value is the task_id of the branch to follow."""
38
        import random
39
40
        if random.choice([True, False]):
41
            return "internal_api_load_full"
42
        return "internal_api_load_incremental"
43
44
    sales_data_transform = EmptyOperator(task_id="sales_data_transform")
45
46
    # When using the TaskFlow API it is common to assign the called task to an object
47
    # to use in several dependency definitions without creating several instances of the same task
48
    determine_load_type_obj = determine_load_type()
49
50
    sales_data_load = EmptyOperator(task_id="sales_data_load")
51
    internal_api_load_full = EmptyOperator(task_id="internal_api_load_full")
52
    internal_api_load_incremental = EmptyOperator(
53
        task_id="internal_api_load_incremental"
54
    )
55
56
    # defining a task group that task a parameter (a) to allow for dynamic task group mapping
57
    @task_group
58
    def sales_data_reporting(a):
59
        # the trigger_rule of the first task in the task group can be set to "all_done"
60
        # to ensure that it runs even if one of the upstream tasks (`internal_api_load_full`)
61
        # is skipped due to the branch task
62
        prepare_report = EmptyOperator(
63
            task_id="prepare_report", trigger_rule="all_done"
64
        )
65
        publish_report = EmptyOperator(task_id="publish_report")
66
67
        # setting dependencies within the task group
68
        chain(prepare_report, publish_report)
69
70
    # dynamically mapping the task group with `a` being the mapped parameter
71
    # one task group instance will be created for each value in the list provided to `a`
72
    sales_data_reporting_obj = sales_data_reporting.expand(a=[1, 2, 3, 4, 5, 6])
73
74
    # defining a task group that does not use any additional parameters
75
    @task_group
76
    def cre_integration():
77
        # the trigger_rule of the first task in the task group can be set to "all_done"
78
        # to ensure that it runs even if one of the upstream tasks (`internal_api_load_full`)
79
        # is skipped due to the branch task
80
        cre_extract = EmptyOperator(task_id="cre_extract", trigger_rule="all_done")
81
        cre_transform = EmptyOperator(task_id="cre_transform")
82
        cre_load = EmptyOperator(task_id="cre_load")
83
84
        # setting dependencies within the task group
85
        chain(cre_extract, cre_transform, cre_load)
86
87
    # calling the task group to instantiate it and assigning it to an object
88
    # to use in dependency definitions
89
    cre_integration_obj = cre_integration()
90
91
    @task_group
92
    def mlops():
93
        # the trigger_rule of the first task in the task group can be set to "all_done"
94
        # to ensure that it runs even if one of the upstream tasks (`internal_api_load_incremental`)
95
        # is skipped due to the branch task
96
        set_up_cluster = EmptyOperator(
97
            task_id="set_up_cluster", trigger_rule="all_done"
98
        )
99
        # the outlets parameter is used to define which datasets are updated by this task
100
        train_model = EmptyOperator(task_id="train_model")
101
        tear_down_cluster = EmptyOperator(task_id="tear_down_cluster")
102
103
        # setting dependencies within the task group
104
        chain(set_up_cluster, train_model, tear_down_cluster)
105
106
        # turning the `tear_down_cluster`` task into a teardown task and set
107
        # the `set_up_cluster` task as the associated setup task
108
        tear_down_cluster.as_teardown(setups=set_up_cluster)
109
110
    mlops_obj = mlops()
111
112
    end = EmptyOperator(task_id="end", outlets=[Asset("dag_completed")])
113
114
    # --------------------- #
115
    # Defining dependencies #
116
    # --------------------- #
117
118
    chain(
119
        start,
120
        sales_data_extract,
121
        sales_data_transform,
122
        sales_data_load,
123
        [sales_data_reporting_obj, cre_integration_obj],
124
        end,
125
    )
126
    chain(
127
        start,
128
        internal_api_extract,
129
        determine_load_type_obj,
130
        [internal_api_load_full, internal_api_load_incremental],
131
        mlops_obj,
132
        end,
133
    )
134
135
    chain_linear(
136
        [sales_data_load, internal_api_load_full],
137
        [sales_data_reporting_obj, cre_integration_obj],
138
    )
139
140
    # Adding labels to two edges
141
    chain(
142
        determine_load_type_obj, Label("additional data"), internal_api_load_incremental
143
    )
144
    chain(
145
        determine_load_type_obj, Label("changed existing data"), internal_api_load_full
146
    )
147
148
149
# Calling the DAG function will instantiate the DAG
150
complex_dag_structure()

Some more complex features visible in this Dag graph are:

• Dynamically mapped tasks: A dynamically mapped task is created dynamically at runtime based on user-defined input. The number of dynamically mapped task instances is shown in brackets ([]) behind the task ID.
• Branching tasks: A branching task creates a conditional branch in the Dag. See Branching in Airflow for more information.
• Edge labels: Edge labels appear on the edge between two tasks. These labels are often helpful to annotate branch decisions in a Dag graph.
• Task groups: A task group is a tool to logically and visually group tasks in an Airflow Dag. See Airflow task groups for more information.
• Setup/teardown tasks: When using Airflow to manage infrastructure, it can be helpful to define tasks as setup and teardown tasks to take advantage of additional intelligent dependency behavior. Setup and teardown tasks appear with diagonal arrows next to their task IDs and are connected with a dotted line. See Use setup and teardown tasks in Airflow for more information.
• Assets: Assets are shown in the Dag graph. If a Dag is scheduled on an asset, it is shown upstream of the first task of the Dag. If a task in the Dag updates an asset, it is shown after the respective task as in the previous screenshot. See Airflow assets for more information.

You can learn more about how to set complex dependencies between tasks and task groups in the Managing Dependencies guide.

Write a Dag

A Dag can be defined with a Python file placed in an Airflow project’s Dag bundle. When using the Astro CLI with default settings this is your dags folder. Airflow automatically parses all files in this folder every 5 minutes to check for new Dags, and it parses existing Dags for code changes every 30 seconds. You can force a new Dag parse using airflow dags reserialize, or astro dev run dags reserialize using the Astro CLI.

There are two types of syntax you can use to structure your Dag:

• TaskFlow API: The TaskFlow API contains the @dag decorator. A function decorated with @dag defines a Dag. Note that you need to call the function at the end of the script for Airflow to register the Dag. All tasks are defined within the context of the Dag function.
• Traditional syntax: You can create a Dag by instantiating a Dag context using the DAG class and defining tasks within that context.

TaskFlow API and traditional syntax can be freely mixed. See Mixing TaskFlow decorators with traditional operators for more information. Additionally, it is also possible to create one-task Dags with the @asset decorator, for more information see Airflow assets.

The following is an example of the same Dag written using each type of syntax.

1
# Import all packages needed at the top level of the DAG
2
from airflow.sdk import dag, task, chain
3
from pendulum import datetime
4
5
6
# Define the DAG function a set of parameters
7
@dag(
8
    start_date=datetime(2025, 4, 1),
9
    schedule="@daily",
10
)
11
def taskflow_dag():
12
    # Define tasks within the DAG context
13
    @task
14
    def my_task_1():
15
        import time  # import packages only needed in the task function
16
17
        time.sleep(5)
18
        print(1)
19
20
    @task
21
    def my_task_2():
22
        print(2)
23
24
    # Define dependencies and call task functions
25
    chain(my_task_1(), my_task_2())
26
27
28
# Call the DAG function
29
taskflow_dag()

1
# Import all packages needed at the top level of the DAG
2
from airflow.sdk import DAG
3
from airflow.providers.standard.operators.python import PythonOperator
4
from pendulum import datetime
5
6
7
def my_task_1_func():
8
    import time  # import packages only needed in the task function
9
10
    time.sleep(5)
11
    print(1)
12
13
14
# Instantiate the DAG
15
with DAG(
16
    dag_id="traditional_syntax_dag",
17
    start_date=datetime(2025, 4, 1),
18
    schedule="@daily",
19
):
20
    # Instantiate tasks within the DAG context
21
    my_task_1 = PythonOperator(
22
        task_id="my_task_1",
23
        python_callable=my_task_1_func,
24
    )
25
26
    my_task_2 = PythonOperator(
27
        task_id="my_task_2",
28
        python_callable=lambda: print(2),
29
    )
30
31
    # Define dependencies
32
    my_task_1 >> my_task_2

Tip

Astronomer recommends creating one Python file for each Dag and naming it after the dag_id as a best practice for organizing your Airflow project. For certain advanced use cases it may be appropriate to dynamically generate Dags using Python code, see Dynamically generate Dags in Airflow for more information.

Dag-level parameters

In Airflow, you can configure when and how your Dag runs by setting parameters in the Dag object. Dag-level parameters affect how the entire Dag behaves, as opposed to task-level parameters which only affect a single task.

The Dags in the previous section have the following basic parameters defined:

• dag_id: The name of the Dag. This must be unique for each Dag in the Airflow environment. When using the @dag decorator and not providing the dag_id parameter name, the function name is used as the dag_id.
• start_date: The date and time after which the Dag starts being scheduled. Defaults to None.
• schedule: The schedule for the Dag. There are many different ways to define a schedule, see Scheduling in Airflow for more information. Defaults to None.

There are many more Dag-level parameters that let you configure anything from resource usage to the Dag’s appearance in the Airflow UI. See Dag-level parameters for a complete list.

FAQ

What does Dag stand for?

Dag stands for “directed acyclic graph,” a mathematical term for a graph with directed edges and no cycles. The Airflow project historically wrote this as the acronym “DAG” but now treats “Dag” as a standalone term.

What is a Dag used for?

In Airflow, a Dag defines a data workflow as a series of tasks with explicit dependencies. This gives you control over task ordering, visibility into pipeline execution through the Airflow UI, and automatic handling of failures across dependent tasks.

Can a Dag have parallel tasks?

Yes. Tasks without dependencies between them run in parallel, assuming your Airflow instance is set up to support parallel tasks. Only tasks with explicit upstream or downstream relationships run sequentially. You control parallelism by how you define task dependencies.

See also

• Get started with Apache Airflow tutorial for a hands-on introduction to writing your first simple Dag.
• Airflow operators and Introduction to the TaskFlow API and Airflow decorators for more information on how to define tasks in a Dag.