Overprovision Airflow components

Overprovisioning in Astro Private Cloud (APC) is a cluster-level setting that sets resource requests for selected Airflow components to a fraction of their limits. This allows the scheduler to pack more pods on nodes (bin-packing) while still allowing bursts up to the limit when needed.

How overprovisioning works

You configure three values in your cluster’s deployment config:

• overProvisioningFactorCPU: Fraction of the CPU limit used as the CPU request (0 < value ≤ 1).
• overProvisioningFactorMem: Fraction of the memory limit used as the memory request (0 < value ≤ 1).
• overProvisioningComponents: List of component names that receive the overprovisioning factor.

For each component in the list, requests are set as:

• requests.cpu = limits.cpu × overProvisioningFactorCPU
• requests.memory = limits.memory × overProvisioningFactorMem

For example, if a component has limits: { cpu: "2000m", memory: "4Gi" } and both factors are 0.5, then requests become { cpu: "1000m", memory: "2Gi" }. Default factors of 1 mean no change (requests equal limits).

Configure overprovisioning

Set these keys in your cluster config (for example, in the cluster’s deployment config or the values used when registering the cluster). Values must be greater than 0 and less than or equal to 1.

1
overProvisioningFactorCPU: 0.5
2
overProvisioningFactorMem: 0.5
3
overProvisioningComponents:
4
  - scheduler
5
  - webserver
6
  - apiServer
7
  - workers
8
  - triggerer
9
  - flower
10
  - pgbouncer
11
  - statsd
12
  - dagProcessor

Only components that have resources defined in the deployment config are affected. Components not listed in overProvisioningComponents keep their existing request/limit values.

Supported components

You can include any of these in overProvisioningComponents:

Choose factor values

• 1 (default): No overprovisioning; requests equal limits. Use when you want predictable capacity and no bin-packing.
• 0.5: Requests are half of limits. Common choice for better bin-packing while keeping headroom.
• Lower values (for example, 0.25): More aggressive bin-packing; ensure your workloads can tolerate less guaranteed CPU/memory.

Start with 0.5 for CPU and memory and adjust based on utilization and scheduling behavior.

Best practices

• Apply factors per cluster: Overprovisioning applies at the cluster level; all deployments on the cluster use the same factors for the listed components.
• Include only components that have resources: Only components with resources.limits (and optionally resources.requests) in the deployment config receive modifications; listing others has no effect.
• Monitor utilization: After enabling, watch pod scheduling and resource usage to confirm the factors match your workload and node capacity.