Apache Airflow is an extensible orchestration tool that offers multiple ways to define and orchestrate data workflows. For users looking to automate actions around those workflows, Airflow exposes a stable REST API in Airflow 2 and an experimental REST API for users running Airflow 1.10. You can use both on Astronomer.
To externally trigger DAG runs without needing to access your Airflow Deployment directly, for example, you can make an HTTP request in Python or cURL to the corresponding endpoint in the Airflow REST API that calls for that exact action.
To get started, you need a service account on Astronomer to authenticate. Read below for guidelines.
The first step to calling the Airflow REST API on Astronomer is to create a Deployment-level Service Account, which will assume a user role and set of permissions and output an API key that you can use to authenticate with your request.
You can use the Software UI or the Astro CLI to create a Service account.
https://app.<BASE-DOMAIN>/token (e.g. https://app.astronomer.example.com/token) and skip to Step 2.Log in to the Software UI.
Select a Deployment and then click the Service Accounts tab.
Click New Service Account and then complete the following fields:
Click Create Service Account.
Copy and save the API key in the API Key field. You can store this key in an environment variable or in a secrets management tool.
Click Service Accounts to return to the Service Accounts page.
To use the Astro CLI to create a Deployment-level Service Account:
Authenticate to the Astro CLI by running:
To identify your <BASE-DOMAIN>, run astro cluster list and select the domain name that corresponds to the cluster you’re working in.
Identify your Airflow Deployment’s Deployment ID. To do so, run:
This will output the list of Airflow Deployments you have access to and their corresponding Deployment ID in the DEPLOYMENT ID column.
With that Deployment ID, run:
The <deployment-role> must be either editor or admin.
Save the API key that was generated. Depending on your use case, you might want to store this key in an Environment Variable or secret management tool.
With the information from Step 1, you can now execute requests against any supported endpoints in the Airflow Rest API Reference at the following base URL:
In the following examples, replace the following values with your own:
<BASE-DOMAIN>: The base domain for your organization on Astronomer Software. For example: mycompany.astronomer.io.<DEPLOYMENT-RELEASE-NAME>: The release name of your Deployment. For example: galactic-stars-1234.<API-KEY>: The API key for your Deployment Service account.<DAG-ID>: Name of your DAG (case-sensitive).The example requests listed below are made with cURL and Python, but you can make requests with any standard method. In all cases, your request will have the same permissions as the role of the service account you created on Astronomer.
To trigger a DAG, execute a POST request to the dagRuns endpoint of the Airflow REST API:
This request will trigger a DAG run for your desired DAG with an execution_date value of NOW(), which is equivalent to clicking the “Play” button in the DAGs view of the Airflow UI.
To set a specific execution_date for your DAG, you can pass in a timestamp with the parameter’s JSON value ("-d'{}').
The string needs to be in the following format (in UTC):
Where, YYYY represents the year, MM represents the month, DD represents the day, HH represents the hour, mm represents the minute, and SS represents the second of your timestamp. For example, "2021-11-16T11:34:00" would create a DAG run with an execution_date of November 16, 2021 at 11:34 AM.
Here, your request would be:
The execution_date parameter was replaced with logical_date in Airflow 2.2. If you run Airflow 2.2+, replace execution_date with logical_date and add a “Z” to the end of your timestamp. For example, "logical_date": "2019-11-16T11:34:00Z".
For more information, see Apache Airflow documentation.
To list all Airflow pools for your Deployment, execute a GET request to the pools endpoint of the Airflow REST API:
As of its momentous 2.0 release, the Apache Astro project now supports an official and more robust Stable REST API. Among other things, Airflow’s new REST API:
To convert a call from the Airflow experimental API, simply update the URL to use the endpoint specified in the Airflow Stable REST API reference.
For example, requests to the Get current configuration endpoint are different depending on which version of Airflow you run.
Prior to Airflow 2, a cURL request to this endpoint would be:
With the stable REST API in Airflow 2, your cURL request would be: