Configure Google Cloud Secret Manager as a secrets backend on Astro Private Cloud

In this section, you’ll learn how to use Google Cloud Secret Manager as a secrets backend on Astro Private Cloud.

Prerequisites

• A Deployment.
• The Astro CLI.
• An Astro project initialized with astro dev init.
• Cloud SDK.
• A Google Cloud environment with Secret Manager configured.
• A service account with the Secret Manager Secret Accessor role on Google Cloud.
• A JSON service account key for the service account.

Step 1: Write an Airflow variable or connection to Google Cloud Secret Manager

To start, add an Airflow variable or connection as a secret to Google Cloud Secret Manager. You can do so in the Cloud Console or the gcloud CLI.

Secrets must be formatted such that:

• Airflow variables are set as airflow-variables-<variable-key>.
• Airflow connections are set as airflow-connections-<connection-id>.

For example, to add an Airflow variable with a key my-secret-variable, you would run the following gcloud CLI command:

1
gcloud secrets create airflow-variables-<my-secret-variable> \
2
    --replication-policy="automatic"

For more information on creating secrets in Google Cloud Secret Manager, see the Google Cloud documentation.

Step 2: Set up Secret Manager locally

To test Google Secret Manager locally, configure it as a secrets backend in your Astro project.

First, install the Airflow provider for Google by adding the following to your project’s requirements.txt file:

apache-airflow-providers-google

Then, add the following environment variables to your project’s Dockerfile:

1
ENV AIRFLOW__SECRETS__BACKEND=airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
2
ENV AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "gcp_keyfile_dict": <your-key-file>}

Make sure to paste your entire JSON service account key in place of <your-key-file>. In the next step, you’ll test that this configuration is valid locally.

Step 3: Run an example Dag to test Secret Manager locally

To test Secret Manager, create a secret containing either an Airflow variable or connection for testing.

Once you create a test secret, write a simple Dag which calls the secret and add this Dag to your project’s dags directory. For example, you can use the following Dag to print the value of a variable to your task logs:

1
from datetime import datetime
2
3
from airflow import DAG
4
from airflow.models import Variable
5
from airflow.operators.python import PythonOperator
6
7
def print_var():
8
    my_var = Variable.get("<your-variable-key>")
9
    print(f'My variable is: {my_var}')
10
11
with DAG('example_secrets_dags', start_date=datetime(2022, 1, 1), schedule=None) as dag:
12
13
  test_task = PythonOperator(
14
15
      task_id='test-task',
16
      python_callable=print_var,
17
)

To test your changes:

1. Run astro dev stop followed by astro dev start to push your changes to your local Airflow environment.

2. In the Airflow UI (http://localhost:8080/admin/), trigger your new Dag.

3. Click test-task > View Logs. If you ran the example Dag above, you should see the contents of your secret in the task logs:

   {logging_mixin.py:109} INFO - My variable is: my-test-variable

Once you confirm that the setup was successful, you can delete this Dag.

Step 4: Deploy to Astro Private Cloud

Once you’ve confirmed that the integration with Google Cloud Secret Manager works locally, you can complete a similar set up with a Deployment on Astro Private Cloud.

1. In the Astro Private Cloud UI, add the same environment variables found in your Dockerfile to your Deployment environment variables. Specify both AIRFLOW__SECRETS__BACKEND and AIRFLOW__SECRETS__BACKEND_KWARGS as Secret to ensure that your credentials are stored securely.
2. In your Astro project, delete the environment variables from your Dockerfile.
3. Deploy your changes to Astro Private Cloud.

You now should be able to see your secret information being pulled from Secret Manager on Astronomer. From here, you can store any Airflow variables or connections as secrets on Secret Manager and use them in your project.