Export logs to a Secondary S3 Bucket

Export logs to a Secondary S3 Bucket

By forwarding Airflow task logs from your Astro Deployment to an additional, customer‑managed S3 bucket, you can keep redundant copies of your execution history, integrate with existing log‑processing pipelines, and satisfy compliance or retention requirements that extend beyond the built‑in Astro logs.

Prerequisites

• Your Deployment must run Astro Runtime 11.7.0 or later. See Upgrade Astro Runtime.
• Your image must include astronomer-providers-logging==1.6.2 or later. The secondary S3 logging feature was introduced in this version.
• You need an AWS account where you can create an S3 bucket and IAM resources.

Overview of configuration options

Astro supports two ways to grant your Deployment permissions to write to the secondary bucket:

1. Customer‑provided Workload Identity: Recommended if you already use this mechanism for other AWS services.
2. Assume Role: Allows your Deployment to assume a dedicated IAM role that has access to the bucket.

Both options require the same base environment variables:

Option 1 : Customer‑Provided Workload Identity

If your Deployment already uses a customer‑managed AWS Workload Identity, you can attach S3 write permissions directly to that role.

1. Configure a Workload Identity for the Deployment if you don’t have one yet.
2. Grant that identity permissions to write to the bucket. Example policy:

1
    "Version": "2012-10-17",
2
    "Statement": [
3
        {
4
            "Sid": "ExternalS3Policy",
5
            "Effect": "Allow",
6
            "Action": [
7
                "s3:PutObject",
8
                "s3:GetObject",
9
                "s3:ListBucket",
10
                "s3:DeleteObject"
11
            ],
12
            "Resource": [
13
                "arn:aws:s3:::my-bucket",
14
                "arn:aws:s3:::my-bucket/*",
15
            ]
16
        }
17
    ]
18
}

3. Set the following environment variables in the Deployment:

1
AIRFLOW__ASTRO_SECONDARY_LOGS__S3_BUCKET_ENABLED=true
2
AIRFLOW__ASTRO_SECONDARY_LOGS__S3_BASE_LOG_FOLDER="s3://my-bucket/logs"
3
# Optional if bucket is in the same region as the deployment
4
AIRFLOW__ASTRO_SECONDARY_LOGS__REGION="us-east-1"

Option 2: Assume Role

With this approach, your Deployment’s existing identity assumes a separate IAM role that has write access to the bucket.

Additional variable

1. Create an IAM role with S3 write permissions (same JSON policy as above).
2. Configure the role’s trust relationship to allow your Deployment to assume it:

1
{
2
  "Version": "2012-10-17",
3
  "Statement": [
4
    {
5
      "Effect": "Allow",
6
      "Principal": {
7
        "AWS": "arn:aws:iam::111122223333:role/astro-deployment-role"
8
      },
9
      "Action": "sts:AssumeRole"
10
    }
11
  ]
12
}

3. Set the environment variables:

1
AIRFLOW__ASTRO_SECONDARY_LOGS__S3_BUCKET_ENABLED=true
2
AIRFLOW__ASTRO_SECONDARY_LOGS__S3_BASE_LOG_FOLDER="s3://my-bucket/logs"
3
AIRFLOW__ASTRO_SECONDARY_LOGS__ROLE_ARN="arn:aws:iam::123456789012:role/SecondaryS3LoggerRole"
4
# Optional if bucket is in the same region
5
AIRFLOW__ASTRO_SECONDARY_LOGS__REGION="us-east-1"

Troubleshooting

If task logs are not appearing in the bucket:

1. Confirm that all required environment variables are set exactly as documented.
2. Verify that the IAM role attached to the Deployment, or the assumed role, includes s3:PutObject for the bucket path.
3. Check the role’s trust relationship (Assume Role options only).
4. Inspect the deployment worker logs for external S3 logging-related errors. Ensure that the Worker is selected in the dropdown - the default is Scheduler

If issues persist, contact Astronomer Support with the Deployment ID and any relevant error output.