For AI agents: a documentation index is available at the root level at /llms.txt and /llms-full.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.
That said, following best practices and taking full advantage of Astro features will help you:
Write Dags more efficiently.
Improve your development workflow.
Better organize your deployment and Dags.
Optimize resource usage and task execution efficiency.
This guide describes best practices for taking advantage of Astro features when writing Dags. While this focuses on writing Airflow Dags for Astro, all general Airflow best practices are also recommended.
New to Airflow?
If you are new to Airflow, Astronomer suggests starting with the following resources:
Astro alerts. No-code alerting on Astro configurable for various trigger types and communication channels.
Best practice guidance
The guidance in this section is split into two categories: writing Dag code for Astro and managing Dag code on Astro.
Write and develop Dags for Astro
Astro users have several tools available when it comes to developing and writing Dags: the Astro CLI, the TaskFlow API, Dag Factory, and Cosmos:
The Astro CLI is recommended for those who want to develop code locally in a containerized setup using their preferred IDE. It is especially useful for those migrating existing Dags to Astro.
The TaskFlow API is a functional API for using decorators to define Dags and tasks, which simplifies the process for passing data between tasks and defining dependencies. The TaskFlow API simplifies the Dag authoring experience by eliminating the boilerplate code required by traditional operators.
Dag Factory is an open source tool for dynamic Dag generation from YAML files that makes it easy for people who don’t know Python to use Airflow. See the Dag Factory Documentation for more information.
Cosmos is an open source tool that turns dbt workflows into Airflow Dags. See the Cosmos documentation for more information.
See Dags for detailed information on structuring your Airflow data pipelines.
Use the Astro CLI for local development
When writing Dags intended for Astro Deployments, use the Astro CLI for containerized local development. The Astro CLI is an open-source interface you can use to:
Test Airflow Dags locally.
Deploy code to Astro.
Automate key actions as part of a CI/CD process.
To get the most out of the Astro CLI:
Use the dags directory to store your Dags. If you have multiple Dags, you can use subdirectories to keep your environment organized.
Modularize your code and store all supporting classes, files and functions in the include directory.
If you are unable to install the Astro CLI on your local machine, due to company policy or for other reasons, you can use it in GitHub Codespaces by forking the Astro CLI Codespaces repository.
Learn more and get started by following the guidance in Astro CLI.
Manage Dag code on Astro
Compared to open-source Airflow, Astro offers a distinct feature set for optimizing your Airflow pipelines. Some of these features impact individual Dags, and you should consider them not only when writing new Dag code but also when migrating Dags to Astro. You don’t need to make changes to existing Dags when you migrate them, but by implementing some small changes you can get access to an upgraded experience when it comes to managing connections, defining and storing environment variables, allocating resources, and implementing alerts.
Making use of the following features can require Dag code changes or adding Airflow configuration environment variables in Astro instead of Airflow. Astronomer recommends considering these features during Dag development to streamline the development process and make use of everything Astro has to offer.
Astro Cloud UI Environment Manager for managing connections. This allows you to define connections once and use them in multiple deployments (with or without overrides for individual fields), as well as in your local development environment. Connections in the Environment Manager are stored in the Astronomer managed secrets backend. Using the Environment Manager requires adding the relevant Airflow provider packages to your Astro project and adding configuring the connection in the Astro UI.
Deployment environment variables in the Astro UI for storing environment variables. Common uses of environment variables include adding tokens or URLs required by your Dags, integrating with third-party tooling to export metrics, customizing core settings of Airflow or Airflow Providers, and storing Airflow connections and variables. A number of approaches are supported. For example, you can mark variables as secret for storage in the Astronomer managed secrets backend. You can store Airflow variables, which are Airflow native key-value pairs, as environment variables as well by using the format AIRFLOW_VAR_MYVARIABLENAME. You can also manage environment variables using your Astro project Dockerfile and, locally, your project .env file. Astro project modification and additional configuration in Astro may be required depending on the strategy chosen.
Worker queues for optimizing task execution efficiency. Worker queues allow you to define sets of workers with specific resources that execute tasks in parallel. You can assign tasks to specific worker queues to optimize resource usage. This feature is an improvement over OSS Airflow pools, which only manage concurrency and not resources. You can use both worker queues and pools in combination. Config in Astro and Dag code modification are required to use this feature.
Astro Alerts for setting up alerts for your Dags. Astro alerts add another level of observability to Airflow’s notification systems. For example, you can configure an alert to notify you in Slack when a Dag run completes or fails. Unlike Airflow callbacks and SLAs, Astro alerts require no changes to Dag code. For more information, see When to use Airflow or Astro alerts for your pipelines on Astro. Config in the Astro UI required.
