Set up AWS Systems Manager (SSM) Parameter Store

Prerequisites

An Astro project with version 5.1.0+ of apache-airflow-providers-amazon. See Add Python and OS-level packages.

An IAM role with access to the Parameter Store that your Astro cluster can assume.

(Remote Execution Only) Helm installed

(Remote Execution Only) The values.yaml file from the Register Agents modal in your Deployments>Agents page.

Step 1: Create Airflow secrets directories in Parameter Store

Create directories for Airflow variables and connections in Parameter Store that you want to store as secrets.

Variables and connections should be stored in /airflow/variables and /airflow/connections, respectively. For example, if you’re setting a secret variable with the key my_secret, it should be stored in the /airflow/connections/ directory. If you modify the directory paths, make sure you change the values for variables_prefix and connections_prefix in Step 2.

Step 2: Set up Parameter Store locally

Astro

Remote Execution

Add the following environment variables to your Astro project’s .env file:

AIRFLOW__SECRETS__BACKEND=airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend
AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables",  "role_arn": "<your-role-arn>", "region_name": "<your-region>"}

You can now run a dag locally to check that your variables are accessible using Variable.get("<your-variable-key>").

Step 3: Deploy configuration

Astro

Remote Execution

Run the following commands to export your secrets backend configurations as environment variables to Astro.

1 $ astro deployment variable create --deployment-id <your-deployment-id> AIRFLOW__SECRETS__BACKEND=airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend
2 
3 $ astro deployment variable create --deployment-id <your-deployment-id> AIRFLOW__SECRETS__BACKEND_KWARGS='{"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables",  "role_arn": "<your-role-arn>", "region_name": "<your-region>"}' --secret

(Optional) Remove the environment variables from your .env file or store your .env file in a safe location to protect your credentials in AIRFLOW__SECRETS__BACKEND_KWARGS.

1	$ astro deployment variable create --deployment-id <your-deployment-id> AIRFLOW__SECRETS__BACKEND=airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend
2
3	$ astro deployment variable create --deployment-id <your-deployment-id> AIRFLOW__SECRETS__BACKEND_KWARGS='{"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables", "role_arn": "<your-role-arn>", "region_name": "<your-region>"}' --secret