You can create and manage Airflow variables for Deployments with the Astro Environment Manager in the Astro UI. The Environment Manager uses an Astro-managed secrets backend to store Airflow variable key-value pairs as Kubernetes Secrets.
Using the Environment Manager, you can quickly and securely create Airflow variables once and share them to multiple Deployments without having to set up your own secrets backend. You can also create a variable once and use it across multiple Airflow Deployments.
For example, you can configure a variable with the access token credentials for a sandbox or development environment. Then, you can later configure your variable to be applied to all Deployments in the workspace by default. This means that when you create new Deployments, they automatically have access to your development environment. Later, you can override the variable to use your production credentials in production Deployments by using a field override or edit.
Compared to creating a variable in the Airflow UI, when you create a variable in the Astro UI, you can:
Workspace Owners and Operators can create and assign variables, while Workspace Authors can view configured variables and use them in Deployments.
When you create an Airflow variable in the Environment Manager, Astro stores Airflow variable details in an Astronomer-hosted secrets manager, and then applies variables to Deployments as Kubernetes Secrets. Specifically the following steps occur:
This process occurs every time you create or update a variable.
You can create Airflow variables both at the Deployment and Workspace level. When you create a variable at the Deployment level, the variable details are available only to that specific Deployment. When you create a variable at the Workspace level, you can apply the variable to several Deployments and override its value as needed for each Deployment.
To create a variable at the Workspace level:
To create a variable at the Deployment level:
After you create a variable, you can reference its Airflow Variable Key from Dag code like you would with any Airflow variable created through the Airflow UI.
After you create a variable at the Workspace level, you can link it to multiple Deployments. Linking variables is useful for standardizing external resource usage across your entire team.
For the most flexibility, you can set default variables and override the variable details per-Deployment based on details like the Deployment’s usage and environment type (production or development).
You can configure Astro to link Workspace-level variables to all Deployments in the Workspace by default.
This is useful, for example, when you need to configure a variable for development environments that all Deployments in a Workspace should start with. Then, when you create new Deployments, they automatically have a default variable to your development resources.
When you’re ready to connect your Deployments to production resources, you can either replace the variable or override the variable field values with your production resource information.
If you toggle the Automatically link to all deployments setting from Off to On, Astro respects any variable value overrides that you might have configured for existing linked Deployments.
If you create a variable at the Workspace level and link it to a Deployment, you can later edit the variable within the Deployment to specify a value override. When you override the variable’s value, you specify the value that you want to use for one Deployment, but not for others. This way, you can configure the variable a single time, but still have the flexibility to customize variables at the Deployment level.
For example, you might have created a variable that points to a dev token, and then add a value override to specify a staging or production token for your staging and production Deployments to use.