Scale Airflow resources | Astronomer Documentation

Configure CPU, memory, and replica settings for Airflow Deployment components including scheduler, webserver, workers, triggerer, Dag processor, and API server.

Resource values are plain integers: CPU in millicpu and memory in MiB.

Component resources

Scheduler

The scheduler orchestrates Dag runs and task scheduling.

1 scheduler:
2   replicas: 1
3   resources:
4     requests:
5       cpu: 500
6       memory: 1920
7     limits:
8       cpu: 1000
9       memory: 3840

Scaling considerations

Add additional replicas for high availability.
Increase memory for complex Dag dependencies.
Set safeToEvict: false to prevent cluster autoscaler eviction.

Webserver

1 webserver:
2   resources:
3     requests:
4       cpu: 500
5       memory: 1920
6     limits:
7       cpu: 1000
8       memory: 3840

API server (Airflow 3+)

1 apiServer:
2   replicas: 1
3   resources:
4     requests:
5       cpu: 1000
6       memory: 3840
7     limits:
8       cpu: 2000
9       memory: 7680

Dag processor (Airflow 2.3+, required in Airflow 3)

1 dagProcessor:
2   enabled: true
3   replicas: 1
4   resources:
5     requests:
6       cpu: 1000
7       memory: 3840
8     limits:
9       cpu: 2000
10       memory: 7680

In Airflow 2, the Dag processor defaults to 0 replicas and must be explicitly enabled. In Airflow 3, the APC API automatically sets dagProcessor.enabled: true and enforces a minimum of 1 replica regardless of configuration.

Triggerer

1 triggerer:
2   replicas: 1
3   resources:
4     requests:
5       cpu: 500
6       memory: 1920
7     limits:
8       cpu: 1000
9       memory: 3840

Workers (Celery executor)

1 workers:
2   replicas: 2
3   resources:
4     requests:
5       cpu: 1000
6       memory: 3840
7     limits:
8       cpu: 2000
9       memory: 7680
10   terminationGracePeriodSeconds: 600

Sizing recommendations

Small workloads (fewer than 50 Dags)

1 scheduler:
2   resources:
3     requests: { cpu: 500, memory: 1920 }
4     limits: { cpu: 1000, memory: 3840 }
5 workers:
6   replicas: 1
7   resources:
8     requests: { cpu: 1000, memory: 3840 }

Medium workloads (50–200 Dags)

1 scheduler:
2   resources:
3     requests: { cpu: 500, memory: 1920 }
4     limits: { cpu: 1000, memory: 3840 }
5 dagProcessor:
6   enabled: true
7   resources:
8     requests: { cpu: 1000, memory: 3840 }
9 workers:
10   replicas: 3
11   resources:
12     requests: { cpu: 1000, memory: 3840 }

Large workloads (more than 200 Dags)

1 scheduler:
2   replicas: 2
3   resources:
4     requests: { cpu: 1000, memory: 3840 }
5     limits: { cpu: 2000, memory: 7680 }
6 dagProcessor:
7   enabled: true
8   replicas: 2
9   resources:
10     requests: { cpu: 1000, memory: 3840 }
11 workers:
12   replicas: 10

Autoscale workers with KEDA

Kubernetes Event-driven Autoscaling (KEDA) scales Celery workers based on task queue depth. Enable KEDA for a Deployment using the updateDeploymentKedaConfig mutation:

1 mutation {
2   updateDeploymentKedaConfig(
3     deploymentUuid: "<deployment-uuid>"
4     state: true
5   ) {
6     id
7     label
8   }
9 }

Monitor resources

$ # View current resource usage
$ kubectl top pods -n <deployment-namespace>
$ 
$ # Check resource limits
$ kubectl describe pod <pod-name> -n <deployment-namespace>

1	scheduler:
2	replicas: 1
3	resources:
4	requests:
5	cpu: 500
6	memory: 1920
7	limits:
8	cpu: 1000
9	memory: 3840

1	webserver:
2	resources:
3	requests:
4	cpu: 500
5	memory: 1920
6	limits:
7	cpu: 1000
8	memory: 3840

1	apiServer:
2	replicas: 1
3	resources:
4	requests:
5	cpu: 1000
6	memory: 3840
7	limits:
8	cpu: 2000
9	memory: 7680

1	dagProcessor:
2	enabled: true
3	replicas: 1
4	resources:
5	requests:
6	cpu: 1000
7	memory: 3840
8	limits:
9	cpu: 2000
10	memory: 7680

1	triggerer:
2	replicas: 1
3	resources:
4	requests:
5	cpu: 500
6	memory: 1920
7	limits:
8	cpu: 1000
9	memory: 3840

1	workers:
2	replicas: 2
3	resources:
4	requests:
5	cpu: 1000
6	memory: 3840
7	limits:
8	cpu: 2000
9	memory: 7680
10	terminationGracePeriodSeconds: 600

1	scheduler:
2	resources:
3	requests: { cpu: 500, memory: 1920 }
4	limits: { cpu: 1000, memory: 3840 }
5	workers:
6	replicas: 1
7	resources:
8	requests: { cpu: 1000, memory: 3840 }

1	mutation {
2	updateDeploymentKedaConfig(
3	deploymentUuid: "<deployment-uuid>"
4	state: true
5	) {
6	id
7	label
8	}
9	}

$	# View current resource usage
$	kubectl top pods -n <deployment-namespace>
$
$	# Check resource limits
$	kubectl describe pod <pod-name> -n <deployment-namespace>