Configure XCom backend

Remote Execution Agents require a custom XCom backend to pass data between tasks. This guide covers configuring AWS S3, Azure Blob Storage, or GCP Cloud Storage as your XCom backend.

Overview

XCom (cross-communication) enables data exchange between Airflow tasks. By default, Airflow stores XCom values in the metadata database. With Remote Execution, the metadata database resides in the Astro orchestration plane while agents execute tasks in your infrastructure.

A custom XCom backend stores XCom data in object storage accessible to all agents in your execution plane. This allows tasks running on different agents to share data through a common storage location.

Prerequisites

• Remote Execution Agent installed and registered
• Object storage bucket in your cloud provider
• Workload identity configured for your Kubernetes cluster (IRSA for AWS, Managed Identity for Azure, Workload Identity for GCP)
• Required Python packages installed in your agent image (see cloud-specific sections below)

Configure XCom backend

All XCom backend configurations use the XComObjectStorageBackend class from the Common IO provider. Choose your cloud provider below for specific configuration instructions.

1
{
2
  "Version": "2012-10-17",
3
  "Statement": [
4
    {
5
      "Effect": "Allow",
6
      "Action": [
7
        "s3:GetObject",
8
        "s3:PutObject",
9
        "s3:DeleteObject",
10
        "s3:ListBucket"
11
      ],
12
      "Resource": [
13
        "arn:aws:s3:::your-xcom-bucket",
14
        "arn:aws:s3:::your-xcom-bucket/*"
15
      ]
16
    }
17
  ]
18
}

1
xcomBackend: "airflow.providers.common.io.xcom.backend.XComObjectStorageBackend"
2
3
commonEnv:
4
  - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH
5
    value: "s3://<connection-id>@<bucket-name>/<path-to-xcom>"
6
  - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD
7
    value: "0"  # Always store XComs in object storage
8
  - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_COMPRESSION
9
    value: "zip"
10
11
annotations:
12
  eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/<IAM_ROLE_NAME>

$
helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml

Configuration options

XCom path format

The AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH parameter defines where XCom objects are stored:

<protocol>://<connection-id>@<bucket-or-container>/<path-prefix>

• protocol: s3:// for AWS, abfs:// for Azure, gs:// for GCP
• connection-id: Airflow connection ID for authentication
• bucket-or-container: Storage bucket or container name
• path-prefix: Optional path prefix for organizing XCom objects

XCom threshold

The AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD parameter controls when XCom values are stored in object storage:

• 0: Always store in object storage (recommended for Remote Execution)
• >0: Store in object storage only if value size exceeds threshold in bytes

Set this to 0 to ensure all XCom values are accessible to remote agents.

Compression

The AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_COMPRESSION parameter controls compression:

• zip: Compress XCom values (recommended)
• None: No compression

Compression reduces storage costs and transfer time for large XCom values.

Verify configuration

After applying your configuration, verify XCom backend functionality:

1. Trigger a test DAG that uses XCom to pass data between tasks
2. Check that tasks complete successfully
3. Verify XCom objects appear in your storage bucket
4. Review agent logs for any XCom-related errors

Next steps

After configuring XCom backend:

• Configure secrets backend - Store Airflow connections and variables securely
• Configure dag sources - Define how agents access DAG code
• Configure logging - Export task logs to external platforms

Related documentation

• XCom backend strategies
• Remote Execution overview
• Helm chart configuration reference