This page provides steps for using HashiCorp Vault as a secrets backend for both local development and on Astro.
To do this, you will:
If you use a different secrets backend tool or want to learn the general approach on how to integrate one, see Configure a Secrets Backend.
http://127.0.0.1:8200/.values.yaml file from the Register Agents modal in your Deployments>Agents page.If you don’t already have a Vault server deployed but would like to test this feature, Astronomer recommends that you either:
To use Vault as a secrets backend, Astronomer recommends configuring a Vault AppRole with a policy that grants only the minimum necessary permissions for Astro. For Remote Execution Deployments, you can use any Vault authentication method you prefer, for example Kubernetes auth if your agents and Vault are running on Kubernetes.
To do this:
Run the following command to create a Vault policy that Astro can use to access a Vault server:
Run the following command to create a Vault AppRole:
Run the following command to retrieve the secret-id for your AppRole:
Save this value. You’ll use this later to complete the setup.
To start, create an Airflow variable or connection in Vault that you want to store as a secret. It can be either a real or test value. You will use this secret to test your backend’s functionality.
You can use an existing mount point or create a new one to store your Airflow connections and variables. For example, to create a new mount point called airflow, run the following Vault CLI command:
To store an Airflow variable in Vault as a secret at the path variables, run the following Vault CLI command with your own values:
To store an Airflow connection in Vault as a secret at the path connections, first format the connection as a URI. Then, run the following Vault CLI command with your own values:
To format existing connections in URI format, see Import and export connections.
value for all Airflow variables and the key name conn_uri for all Airflow connections as shown in the previous commands.To confirm that your secret was written to Vault successfully, run:
In your Astro project, add the HashiCorp Airflow provider to your project by adding the following to your requirements.txt file:
Then, add the following environment variables to your .env file:
If you run Vault on HashiCorp Cloud Platform (HCP):
http://host.docker.internal:8200 with https://<your-cluster>.hashicorp.cloud:8200."namespace": "admin" as an argument after url.This tells Airflow to look for variable and connection information at the airflow/variables/* and airflow/connections/* paths in your Vault server. You can now run a Dag locally to check that your variables are accessible using Variable.get("<your-variable-key>").
If your Astro environment runs in AWS, you can authenticate to Vault using AWS IAM with an STS assume role instead of an AppRole. This lets Vault verify the assumed AWS IAM role rather than requiring you to manage a long-lived AppRole secret. For more details, see Vault authentication with AWS Assume Role STS in the Apache Airflow documentation.
To use this authentication method, set auth_type to aws_iam and provide assume_role_kwargs with the IAM role to assume. For example:
For the full list of supported parameters in assume_role_kwargs, see the AWS STS assume_role reference.
Run the following commands to export your environment variables to Astro:
Run the following command to push your updated requirements.txt file to Astro:
(Optional) Remove the environment variables from your .env file or store your .env file in a safe location to protect your credentials in AIRFLOW__SECRETS__BACKEND_KWARGS.
Now, any Airflow variable or connection that you write to your Vault server can be successfully accessed and pulled by any Dag in your Deployment on Astro.