Create a Microsoft Entra Workload ID connection in Airflow

Info

This page has not yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.

A workload identity is an identity you can assign to your Airflow environment which is authorized to access external services and resources. On Azure, a single workload identity can be authorized to multiple Azure resources through Azure resource groups.

The new generic Azure connection type lets you assign a workload identity to your Airflow environment so that Airflow can access multiple Azure resources using a single Airflow connection. This configuration greatly simplifies the number of credentials and connections you need to manage for Azure workflows.

This guide explains how to set up an Azure Workload Identity connection using the Azure connection type on Astro. Astronomer recommends using this connection type for most Azure workflows.

Prerequisites

• The Astro CLI.
• The Azure CLI or access to the Azure web portal.
• An Astro project.
• (Optional) An Astro Deployment.
• A Microsoft Entra managed identity.

Info

If you want to use Microsoft Entra Workload ID with a generic Apache Airflow project, your setup steps might vary. See Airflow documentation.

Get connection details

To create a workload identity for your Airflow environment, you first need to link your Airflow environment to your Entra ID managed identity.

If you’re using Astro, follow the steps in Authorize Deployments to Cloud resources to create a workload identity for your Deployment.

If you’re using Apache Airflow outside of Astro, your setup will vary based on your cloud and the environment you’re running Airflow in. Generally speaking, the setup will be similar to the following:

1. In your Azure portal, open the Managed Identities menu.

2. Search for your managed identity, click Properties, then copy its Name, Client ID, Tenant ID, and Resource group name.

3. Run the following command to create a workload identity for your Airflow environment, replacing the <managed-identity> and <resource-group> values with your managed identity Name and Resource group name respectively.

   $
   workloads=( scheduler triggerer worker )
   $
   for workload in "${workloads[@]}"; do
   $
       az identity federated-credential create --name <credential-name>-$workload --identity-name <managed-identity> --resource-group <resource-group> --issuer <your-issuer> --subject <your-service-account>
   $
   done
   $
   az identity federated-credential create --name <credential-name> --identity-name <managed-identity> --resource-group <resource-group> --issuer <your-issuer> --subject <your-service-account>

Create your connection

To create your connection in Astro, follow the steps to create a new connection in the Astro Environment Manager. Select the Azure workload identity connection type and enter your Client ID and Tenant ID. If you need to specify a Subscription ID for a specific service, you can open the More options dropdown menu and add it there.

Alternatively, to create your connection in the Airflow UI:

1. In the Airflow UI, go to Admin > Connections.
2. Click + to add a new connection, then select Azure as the connection type.
3. Enter the clientId and tenantId fields you retrieved from Get connection details and enter them into the Managed Identity Client ID and Workload Identity Tenant ID fields respectively. You can also specify a subscriptionId for a specific service if required.
4. Click Save.

After you create your connection, any DAGs using the connection will have the same permissions and access you defined in your managed identity.