For AI agents: a documentation index is available at the root level at /llms.txt and /llms-full.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.
Manage Airflow connections, variables, and environment variables
Airflow connections are used for storing credentials and other information necessary for connecting to external services. Airflow variables are a generic way to store and retrieve arbitrary content or settings as a simple key value store within Airflow. Environment variables are key-value configurations that can be used to configure Airflow settings, store credentials, or pass configuration to your DAGs.
Use this document to select the right Airflow connection, variable, and environment variable management strategies for your team.
Airflow supports several different methods for managing connections, variables, and environment variables. Each of these strategies has benefits and limitations related to their security and ease of use. The strategies you choose should be compatible with both your local environments and Astro Deployments, allowing you to import and export objects between the two contexts.
Because connections serve different purposes in Airflow, you might want to use a different strategy for each object type. For example, you can use a secrets backend for connections and use combination of a json files and the Airflow UI for variables.
If you use a mix of strategies for managing connections, it’s important to know which connection value that Airflow gives precedence to in the case of a conflict. The order of precedence for connections is:
Secrets Backend
Astro Environment Manager
Environment Variables
Airflow’s metadata database (Airflow UI)
Airflow checks for connections in order from highest (secrets backend) to lowest (Airflow UI) precedence. Airflow uses the first configuration for a given connection ID that it finds.
For example, if you store a connection with the same connection ID in both a secrets backend and the Airflow UI, Airflow reads and uses the connection configured in the secrets backend first, instead of using a connection configuration with the same connection ID stored in the Airflow UI.
If you only want to test connections or export connections in a JSON or URI format, use the Airflow UI to manage your connection. You can then use the Astro CLI commands to export the connections in a URI or JSON format. See Import and export connections and variables.
Compare strategies
The following sections explain the benefits, limitations, and implementations of each strategy in more detail.
Astro Environment Manager
Astro includes a built-in secrets backend for managing connections, Airflow variables, and environment variables. After you create a connection, variable, or environment variable in the Astro UI, you can share it with multiple Deployments in a Workspace, override values on a per-Deployment basis, or hide secret values after creating them. See Create connections in Astro, Create variables in Astro, or Create environment variables in Astro for setup steps.
For Remote Execution Deployments, you can only manage environment variables and Metrics export with the Environment Manager. You cannot create, update, or assign Connections or Airflow Variables.
Benefits
It securely stores your connection, variable, and environment variable configuration and authentication information on Astro, without a secrets backend.
You can create a configuration once and then add it to multiple Deployments in a Workspace.
Authenticate connections using credentials other than what the Airflow UI supports.
Hide the values for Airflow Variable keys and environment variables in the Astro UI, after you finish creating them.
After you configure a connection, variable, or environment variable, the authentication information is securely stored in an Astro-managed secrets backend.
Control whether connections, Airflow variables, or environment variables can be used in a single Deployment or shared across Deployments in a Workspace.
Override certain fields in connections, variable values, or environment variable values per Deployment, so you can create a general configuration and then customize its behavior.
Limitations
If you create a connection in the Astro UI, you also need to add its related provider package to the requirements.txt file in your Astro project.
Only available with Astro Runtime 9.3.0 and greater.
You can’t see connections, Airflow variables, or environment variables defined in the Astro UI in the Airflow UI.
You need WORKSPACE_OPERATOR or WORKSPACE_OWNER user permissions.
You can’t programmatically import connections, Airflow variables, or environment variables to the Environment Manager from your local environment.
You can create Airflow connections and variables is through the Airflow UI. This experience is the same for both local Airflow environments and Astro Deployments. Astronomer recommends this method if you’re just getting started with Airflow, you want to get your dags running quickly, or if you want to export connections in a URI/JSON format.
Benefits
The UI has features for correctly formatting and testing your connections.
It’s easy to change variables or connections to test different use cases on the fly.
Connections and variables are encrypted and stored in the Airflow metadata database.
Limitations
You cannot export or import connections from the UI for security reasons.
Managing many connections or variables can become unwieldy.
In a local environment, you lose your connections and variables if you delete your metadata database with astro dev kill.
Secrets backend
A secrets backend is the most secure way to store connections and variables. You can access a secrets backend both locally and on Astro by configuring the appropriate credentials in your Airflow environment. Astronomer recommends this method for all staging and production deployments. See the following documentation for setup steps:
Store objects in a centralized location alongside other secrets used by your organization.
Comply with internal security postures and policies that protect your organization.
Recover objects if your Airflow environments go down.
Share secrets across different Airflow environments.
Allow selective access to connections and variables by using connections_prefix and variables_prefix.
Limit the number of open connections to your metadata database, especially if you are using your connections and variables outside of task definitions.
Limitations
A third-party secrets manager is required.
Separate configurations might be required for using a secrets backend locally and on Astro.
You cannot use the Airflow UI to view connections and variables.
You are responsible for ensuring that secrets are encrypted.
Environment variables
You can use Airflow’s system-level environment variables to store connections and variables, or to configure general Airflow settings and pass configuration to your DAGs. There are two ways to manage environment variables on Astro:
Workspace-level environment variables (Astro Environment Manager): Create environment variables at the Workspace level and share them across multiple Deployments. This is the recommended approach when you want to standardize environment variables across Deployments. See Create environment variables in Astro.
Deployment-level environment variables: Set environment variables specific to a single Deployment through the Deployment UI or your Dockerfile. This is recommended when you don’t have a secrets backend, but you still want to take advantage of security and RBAC features. See Manage environment variables on Astro.
Airflow connections and variables are stored in the Airflow metadata database. Calling them outside of task definitions and operators requires an additional connection to the Airflow metadata database which is used every time the scheduler parses a dag. By adding connections and variables as environment variables, you can lower the amount of open connections and improve the performance of your database and resources.
Benefits
If you use an .env file for your local Airflow environment and your local metadata database is corrupted or accidentally deleted, you still have access to all of your connections and variables.
This approach limits the number of open connections to your metadata database, especially if you are using your connections and variables outside of task definitions.
Workspace-level environment variables can be shared across multiple Deployments with override capabilities per Deployment.
Limitations
You can’t view connections and variables from the Airflow UI when they are stored as environment variables.
You must restart your local environment using astro dev restart whenever you make changes to your .env file.
When stored in your .env file or Dockerfile, environment variables are defined in plain text.
Connections must be formatted as either a URI or serialized JSON.
Deployment-level environment variables are not as secure or centralized compared to a secrets backend or the Workspace Environment Manager.
You cannot directly export Deployment-level environment variables from the Astro UI to a local Airflow environment. See Import and export Airflow objects.
Other strategies
While it’s possible to manage Airflow connections and variables with these strategies, Astronomer doesn’t recommend them at scale:
You can use the Airflow REST API to programmatically create Airflow connections and variables for a Deployment. Airflow objects created with the API are stored in the Airflow metadata database and visible in the Airflow UI.
For local Astro projects, you can use airflow_settings.yaml for defining your connections and variables. See Configure airflow_settings.yaml for more details.
