Astro executor

Overview

The Astro executor is the default for all Airflow 3.x Deployments, and is the only executor you can use for Deployments in remote execution mode. The Astro executor consists of Agents that pull work from an API server. The API server manages the Agent lifecycle and controls task assignment logic, enabling more efficient workload distribution and scaling. This design differs from the Celery executor, where workers fetch tasks directly from a queue, and the Kubernetes executor, where the scheduler launches pods for each task.

By centrally managing both Agent scaling and task assignment, the Astro executor offers increased reliability, improved performance, and cost efficiency compared to other Airflow execution models.

Remote Execution mode

In a remote execution mode Deployment, the Remote Execution Agents, data sources, code, secrets, and logs are housed in your environment while the API server is hosted in Astro’s Orchestration Plane. To configure workers and scaling in Remote Execution mode, see Remote Execution Agents.

For more information on Remote versus Hosted execution mode, see Execution mode.

Astro worker autoscaling logic in Hosted execution mode

If your Deployment is in Hosted execution mode, the number of Astro workers running per worker queue on your Deployment at a given time is based on two values:

• The total number of tasks in a queued or running state.
• The worker queue’s setting for Concurrency.

The calculation is made based on the following expression:

[Number of workers]= ([Queued tasks]+[Running tasks])/(Concurrency)

Kubernetes Event Driven Autoscaling (KEDA) computes these calculations every ten seconds. When KEDA determines that it can scale down a worker, it waits for five minutes after the last running task on the worker finishes before terminating the worker Pod.

When you push a new image to a Deployment, workers running tasks from before the code push don’t shut down until those tasks are complete. To learn more about how changes to a Deployment can affect worker resource allocation, see What happens during a code deploy.

Configure the Astro executor in Hosted execution mode

In Hosted execution mode, you can configure Astro executor in the following ways with the Astro UI:

• The type and size of your workers.
• The minimum and maximum number of workers that your Deployment can run at a time.
• The number of tasks that each worker can run at a time.

You can set these configurations per worker queue. With the Astro executor, you can configure multiple worker queues for different types of tasks and assign tasks to those queues in your Dag code.

The following document explains basic Astro executor configurations for a single worker queue. For instructions on how to configure multiple worker queues, see Create a worker queue.

Configure Astro worker scaling in Hosted execution mode

For each worker queue on your Deployment, you have to specify certain settings that affect worker scaling behavior. If you’re new to Airflow, Astronomer recommends using the defaults in Astro for each of these settings.

1. In the Astro UI, select a Workspace, click Deployments, and then select a Deployment.

2. Click the Details tab and then click Edit in the Execution section to edit a worker queue.

3. Configure the following settings:

   • Worker type: Choose the amount of resources that each worker will have.
   • Concurrency: The maximum number of tasks that a single worker can run at a time. If the number of queued and running tasks exceeds this number, a new worker is added to run the remaining tasks. This value is equivalent to the Apache Airflow worker concurrency setting. It is 16 by default.
   • Storage: Choose the amount of ephemeral storage in GiB that each worker has. This storage volume is transient and allows for temporary storage and processing of data within the worker. The worker is assigned the minimum 10 GiB by default. The maximum quota is 100 GiB. Only ephemeral storage requests that are greater than the default minimum of 10 GiB are chargeable. Note that this feature is in Preview.
   • Worker Count (Min-Max): The minimum and maximum number of workers that can run at a time. The number of running workers changes based on Concurrency and the current number of tasks in a queued or running state. By default, the minimum number of workers is 1 and the maximum is 10.

4. Click Update Queue.

See also

• Configure worker queues.