Customize Deployment CPU and management resources per component

When you create a new Astro Private Cloud Deployment, you now specify the exact amount of CPU and memory that power its core components.

For example:

• You might need to allocate significantly more memory than CPU to your worker Pods if you run memory-intensive tasks.
• At the same time, your Scheduler may need more memory than CPU.

Deployment Admins can:

• Assign exact CPU and memory values to each component.
• Ensure that Pods use these values as both requests and limits, providing predictable scheduling and resource enforcement.

In Astro Private Cloud 1.0, you configure resources by setting CPU and memory directly for each Airflow component in the UI or with the Houston API.

Set CPU and Memory Resources in the Astro Private Cloud UI

Configure Worker resources

1. Navigate to your Deployment in the APC UI.
2. Open the Settings page for that Deployment.
3. In the Execution Environment section, under Workers, set the desired CPU and memory values. These values will be applied to all worker Pods.
4. Click Deploy Changes.

Configure core component resources

1. Navigate to your Deployment in the APC UI.
2. Open the Settings page for that Deployment.
3. In the Core Resources section, set CPU and memory for the following components individually:
   • Webserver / API Server (Airflow 3 only)
   • Scheduler
   • Triggerer
   • Dag Processor

4. Click Deploy Changes.

Set Deployment component resources with Houston API

You can also set resources specifications with the Houston API.

Setting defaults in the config

You can set custom defaults, limits, and minimums for Airflow components by defining them in the config file. When you define a component using the example below, Deployments are created with those resource values, and the UI reflects the same defaults.

1
astronomer:
2
  houston:
3
    config:
4
      components:
5
        - name: scheduler
6
          custom:
7
            default:
8
              cpu: 1000
9
              memory: 2000
10
            minimum:
11
              cpu: 500
12
              memory: 1000
13
            limit:
14
              cpu: 6000
15
              memory: 12000
16
          extra:
17
            - name: replicas
18
              default: 1
19
              minimum: 1
20
              limit: 4
21
              minAirflowVersion: "2.0.0"

Set resource quotas per deployment

Astro Private Cloud lets you override default quota calculations by specifying resource quotas directly in the Houston API’s Deployment payload. Use the quotas object to set custom CPU and memory requests/limits for your deployment. If you do not provide quotas, Astronomer uses the platform’s default quota logic.

Default quotas config

Include the requests and limits in your quotas parameter of the deployment upsert API payload. If you set quotas, make sure your values are not less than required platform minimums and do not exceed the allowed platform maximums.

The following example shows the quotas object and the platform defaults for CPU and memory requests and limits.

1
"quotas": {
2
  "requests": {
3
    "cpu": 1,
4
    "memory": "1920Mi"
5
  },
6
  "limits": {
7
    "cpu": 2,
8
    "memory": "1920Mi"
9
  }
10
}

Configuration options

Use custom quotas if you need to guarantee or constrain Deployment resources beyond the system-determined logic. If you do not set quotas, the deployment will use platform default resource constraints.

Configure Deployment-level limits for resource usage

Astro Private Cloud limits the amount of resources that can be used by all Pods in a Deployment by creating and managing a LimitRange and ResourceQuota for the namespace associated with each Deployment.

These values are automatically adjusted to account for the resource requirements of various components.

You can add additional resources, beyond the standard amount allocated based on the resource-requirements of standing components, to the LimitRange and ResourceQuota. Add resources by configuring astronomer.houston.config.deployments.maxExtraCapacity and astronomer.houston.config.deployments.maxExtraPodCapacity to account for the requirements of KubernetesExecutor and KubernetesPodOperator tasks.

1
astronomer:
2
  houston:
3
    config:
4
      deployments:
5
        maxExtraCapacity:
6
          cpu: 40000 # in milliCPUs (m)
7
          memory: 153600 # in MiB (Mi)
8
        maxPodCapacity:
9
          cpu: 3500 # in milliCPUs (m)
10
          memory: 13440 # in MiB (Mi)

Configurable Components

Configurable components include:

Airflow Scheduler

1
- name: scheduler
2
  custom:
3
    default:
4
      cpu: 1000
5
      memory: 2000
6
    minimum:
7
      cpu: 500
8
      memory: 1000
9
    limit:
10
      cpu: 6000
11
      memory: 12000
12
  extra:
13
    - name: replicas
14
      default: 1
15
      minimum: 1
16
      limit: 4
17
      minAirflowVersion: "2.0.0"

Airflow Dag processor

1
astronomer:
2
  houston:
3
    config:
4
      deployments:
5
        dagProcessorEnabled: true

1
houston:
2
  config:
3
    deployments:
4
      helm:
5
        airflow:
6
          dagProcessor:
7
            extraContainers:
8
              - name: <extra-container>
9
                env:
10
                  - name: DAGS_LOCATION
11
                    value: /dags
12
                  - name: SECRET_LOCATION
13
                    value: /secrets
14
                  - name: SYNC_INTERVAL
15
                    value: "15"
16
                image: ubuntu:latest
17
                command: ["/bin/bash", "-c", "--"]
18
                args: ["<arg-1>; <arg-2>; <arg-3>;"]

1
- name: dagProcessor
2
  custom:
3
    default:
4
      cpu: 1000
5
      memory: 3840
6
    minimum:
7
      cpu: 1000
8
      memory: 3840
9
    limit:
10
      cpu: 3000
11
      memory: 11520
12
  extra:
13
    - name: replicas
14
      default: 0
15
      minimum: 0
16
      limit: 3

Airflow Webserver

1
- name: webserver
2
 custom:
3
   default:
4
     cpu: 1000
5
     memory: 2000
6
   minimum:
7
     cpu: 500
8
     memory: 1000
9
   limit:
10
     cpu: 6000
11
     memory: 12000

StatsD

1
- name: statsd
2
  custom:
3
    default:
4
      cpu: 200
5
      memory: 768
6
    minimum:
7
      cpu: 200
8
      memory: 768
9
    limit:
10
      cpu: 3000
11
      memory: 11520

Database Connection Pooler (PgBouncer)

1
- name: pgbouncer
2
  custom:
3
    default:
4
      cpu: 200
5
      memory: 768
6
    minimum:
7
      cpu: 200
8
      memory: 768
9
    limit:
10
      cpu: 200
11
      memory: 768

Celery Diagnostic Web Interface (Flower)

1
- name: flower
2
  custom:
3
    default:
4
      cpu: 200
5
      memory: 768
6
    minimum:
7
      cpu: 200
8
      memory: 768
9
    limit:
10
      cpu: 200
11
      memory: 768

Redis

1
- name: redis
2
 custom:
3
   default:
4
     cpu: 200
5
     memory: 768
6
   minimum:
7
     cpu: 200
8
     memory: 768
9
   limit:
10
     cpu: 200
11
     memory: 768

Celery Workers

1
- name: workers
2
 custom:
3
   default:
4
     cpu: 1000
5
     memory: 3840
6
   minimum:
7
     cpu: 100
8
     memory: 384
9
   limit:
10
     cpu: 3000
11
     memory: 11520
12
 extra:
13
   - name: terminationGracePeriodSeconds
14
     default: 600
15
     minimum: 0
16
     limit: 36000
17
   - name: replicas
18
     default: 1
19
     minimum: 1
20
     limit: 20

Triggerer

1
- name: triggerer
2
  custom:
3
    default:
4
      cpu: 1000
5
      memory: 2000
6
    minimum:
7
      cpu: 500
8
      memory: 1000
9
    limit:
10
      cpu: 6000
11
      memory: 12000
12
  extra:
13
    - name: replicas
14
      default: 1
15
      minimum: 0
16
      limit: 4
17
      minAirflowVersion: "2.2.0"