Configure an external secrets backend on Astronomer Software

In this section, you’ll learn how to use Hashicorp Vault as a secrets backend for both local development and on Astronomer Software. To do this, you will:

• Create an AppRole in Vault which grants Astronomer minimal required permissions.
• Write a test Airflow variable or connection as a secret to your Vault server.
• Configure your Astro project to pull the secret from Vault.
• Test the backend in a local environment.
• Deploy your changes to Astronomer Software.

Prerequisites

• A Deployment on Astronomer.
• The Astro CLI.
• A Hashicorp Vault server.
• An Astro project initialized with astro dev init.
• The Vault CLI.
• Your Vault Server’s URL. If you’re using a local server, this should be http://127.0.0.1:8200/.

If you do not already have a Vault server deployed but would like to test this feature, Astronomer recommends that you either:

• Sign up for a Vault trial on Hashicorp Cloud Platform (HCP) or
• Deploy a local Vault server. See Starting the server in Hashicorp documentation.

Step 1: Create a Policy and AppRole in Vault

To use Vault as a secrets backend, Astronomer recommends configuring a Vault AppRole with a policy that grants only the minimum necessary permissions for Astronomer Software. To do this:

1. Create a Vault policy with the following permissions:

   1
   path "secret/data/variables/*" {
   2
     capabilities = ["read", "list"]
   3
   }
   4
   5
   path "secret/data/connections/*" {
   6
     capabilities = ["read", "list"]
   7
   }

2. Create a Vault AppRole and attach the policy you just created to it.

3. Retrieve the role-id and secret-id for your AppRole by running the following commands:

   1
   vault read auth/approle/role/<your-approle>/role-id
   2
   vault write -f auth/approle/role/<your-approle>/secret-id

   Save these values for Step 3.

Step 2: Write an Airflow variable or connection to Vault

To test whether your Vault server is set up properly, create a test Airflow variable or connection to store as a secret.

To store an Airflow variable in Vault as a secret, run the following Vault CLI command with your own values:

1
vault kv put secret/variables/<your-variable-key> value=<your-variable-value>

To store a connection in Vault as a secret, run the following Vault CLI command with your own values:

1
vault kv put secret/connections/<your-connection-id> conn_uri=<connection-type>://<connection-login>:<connection-password>@<connection-host>:5432

To confirm that your secret was written to Vault successfully, run:

1
# For variables
2
$ vault kv get secret/variables/<your-variable-key>
3
# For connections
4
$ vault kv get secret/connections/<your-connection-id>

Step 3: Set up Vault locally

In your Astro project, add the Hashicorp Airflow provider to your project by adding the following to your requirements.txt file:

apache-airflow-providers-hashicorp

Then, add the following environment variables to your Dockerfile:

1
# Make sure to replace `<your-approle-id>` and `<your-approle-secret>` with your own values.
2
ENV AIRFLOW__SECRETS__BACKEND=airflow.providers.hashicorp.secrets.vault.VaultBackend
3
ENV AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_path": "connections", "variables_path": "variables", "config_path": null, "url": "http://host.docker.internal:8200", "auth_type": "approle", "role_id":"<your-approle-id>", "secret_id":"<your-approle-secret>"}

This tells Airflow to look for variable and connection information at the secret/variables/* and secret/connections/* paths in your Vault server. In the next step, you’ll test this configuration in a local Airflow environment.

For more information on the Airflow provider for Hashicorp Vault and how to further customize your integration, see the Apache Airflow documentation.

Step 4: Run an example DAG to test Vault locally

To test Vault, write a simple DAG which calls your test secret and add this DAG to your project’s dags directory. For example, you can use the following DAG to print the value of a variable to your task logs:

1
from airflow import DAG
2
from airflow.hooks.base import BaseHook
3
from airflow.models import Variable
4
from airflow.operators.python import PythonOperator
5
from datetime import datetime
6
7
def print_var():
8
    my_var = Variable.get("<your-variable-key>")
9
    print(f'My variable is: {my_var}')
10
11
with DAG('example_secrets_dags', start_date=datetime(2022, 1, 1), schedule=None) as dag:
12
13
  test_task = PythonOperator(
14
      task_id='test-task',
15
      python_callable=print_var,
16
)

Once you’ve added this DAG to your project:

1. Run astro dev restart to push your changes to your local Airflow environment.

2. In the Airflow UI (http://localhost:8080/admin/), trigger your new DAG.

3. Click on test-task > View Logs. If you ran the example DAG above, you should see the contents of your secret in the task logs:

   {logging_mixin.py:109} INFO - My variable is: my-test-variable

Once you confirm that the setup was successful, you can delete this example DAG.

Step 5: Deploy on Astronomer Software

Once you’ve confirmed that the integration with Vault works locally, you can complete a similar set up with a Deployment on Astronomer Software.

1. In the Software UI, add the same environment variables found in your Dockerfile to your Deployment environment variables. Specify AIRFLOW__SECRETS__BACKEND_KWARGS as secret to ensure that your Vault credentials are stored securely.
2. In your Astro project, delete the environment variables from your Dockerfile.
3. Deploy your changes to Astronomer Software.

Now, any Airflow variable or connection that you write to your Vault server can be successfully accessed and pulled by any DAG in your Deployment on Astronomer Software.