Forward logs to Amazon S3

Astro Private Cloud (APC) uses Vector for log collection and forwarding. You can configure Vector to send Airflow task logs to Amazon S3 for long-term storage, compliance, or integration with other analytics tools.

Architecture

Vector continues forwarding logs to Elasticsearch for the Airflow UI while also sending copies to S3.

Prerequisites

• An existing S3 bucket
• AWS IAM credentials with S3 write access
• APC 2.0 or later

1
{
2
  "Version": "2012-10-17",
3
  "Statement": [
4
    {
5
      "Effect": "Allow",
6
      "Action": [
7
        "s3:ListBucket"
8
      ],
9
      "Resource": "arn:aws:s3:::<YOUR_LOGS_BUCKET>"
10
    },
11
    {
12
      "Effect": "Allow",
13
      "Action": [
14
        "s3:PutObject",
15
        "s3:GetObject"
16
      ],
17
      "Resource": "arn:aws:s3:::<YOUR_LOGS_BUCKET>/*"
18
    }
19
  ]
20
}

1
{
2
  "Version": "2012-10-17",
3
  "Statement": [
4
    {
5
      "Effect": "Allow",
6
      "Principal": {
7
        "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/oidc.eks.<AWS_REGION>.amazonaws.com/id/<OIDC_ID>"
8
      },
9
      "Action": "sts:AssumeRoleWithWebIdentity",
10
      "Condition": {
11
        "StringEquals": {
12
          "oidc.eks.<AWS_REGION>.amazonaws.com/id/<OIDC_ID>:sub": "system:serviceaccount:astronomer:astronomer-vector"
13
        }
14
      }
15
    }
16
  ]
17
}

1
vector:
2
  serviceAccount:
3
    annotations:
4
      eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/<VECTOR_S3_ROLE>

1
vector:
2
  extraSinks:
3
    s3_logs:
4
      type: aws_s3
5
      inputs:
6
        - transform_remove_fields
7
      bucket: "<YOUR_LOGS_BUCKET>"
8
      region: "<AWS_REGION>"
9
      key_prefix: "airflow-logs/{{ "{{ namespace }}" }}/{{ "{{ release }}" }}/%Y/%m/%d/"
10
      compression: gzip
11
      encoding:
12
        codec: json
13
      batch:
14
        max_bytes: 10485760
15
        timeout_secs: 300
16
      request:
17
        retry_attempts: 5

$
helm upgrade astronomer astronomer/astronomer \
>
  -f values.yaml \
>
  --namespace astronomer

$
kubectl rollout status daemonset/astronomer-vector -n astronomer

$
kubectl logs -n astronomer -l app=vector --tail=100 | grep -i s3

$
aws s3 ls s3://<YOUR_LOGS_BUCKET>/airflow-logs/ --recursive | head -20

$
aws s3 cp s3://<YOUR_LOGS_BUCKET>/airflow-logs/path/to/file.json.gz - | gunzip | head -5

Advanced configuration

Filter logs by severity

Only forward ERROR and WARNING logs to S3 using a Vector Remap Language (VRL) filter condition:

1
vector:
2
  extraTransforms:
3
    filter_errors:
4
      type: filter
5
      inputs:
6
        - transform_remove_fields
7
      condition:
8
        type: vrl
9
        source: '.level == "ERROR" || .level == "WARNING"'
10
11
  extraSinks:
12
    s3_errors:
13
      type: aws_s3
14
      inputs:
15
        - filter_errors
16
      bucket: "<YOUR_LOGS_BUCKET>"
17
      # ... rest of config

Partition by deployment

Organize logs by deployment namespace:

1
vector:
2
  extraSinks:
3
    s3_logs:
4
      type: aws_s3
5
      inputs:
6
        - transform_remove_fields
7
      bucket: "<YOUR_LOGS_BUCKET>"
8
      key_prefix: "deployments/{{ "{{ namespace }}" }}/{{ "{{ pod }}" }}/%Y/%m/%d/"
9
      # ... rest of config

Multiple destinations

Forward to both S3 and another system:

1
vector:
2
  extraSinks:
3
    s3_archive:
4
      type: aws_s3
5
      inputs:
6
        - transform_remove_fields
7
      bucket: "<ARCHIVE_BUCKET>"
8
      # ... config
9
10
    splunk_realtime:
11
      type: splunk_hec
12
      inputs:
13
        - transform_remove_fields
14
      endpoint: "<SPLUNK_ENDPOINT>"
15
      token: "${SPLUNK_TOKEN}"

S3 lifecycle policies

Configure S3 lifecycle rules to manage log retention:

1
{
2
  "Rules": [
3
    {
4
      "ID": "ArchiveOldLogs",
5
      "Status": "Enabled",
6
      "Filter": {
7
        "Prefix": "airflow-logs/"
8
      },
9
      "Transitions": [
10
        {
11
          "Days": 30,
12
          "StorageClass": "STANDARD_IA"
13
        },
14
        {
15
          "Days": 90,
16
          "StorageClass": "GLACIER"
17
        }
18
      ],
19
      "Expiration": {
20
        "Days": 365
21
      }
22
    }
23
  ]
24
}

Apply via AWS CLI:

$
aws s3api put-bucket-lifecycle-configuration \
>
  --bucket <YOUR_LOGS_BUCKET> \
>
  --lifecycle-configuration file://lifecycle.json

Troubleshooting

Logs not appearing in S3

1. Check Vector pod logs:

   $
   kubectl logs -n astronomer -l app=vector | grep -i error

2. Verify AWS credentials:

   $
   kubectl exec -n astronomer -it ds/astronomer-vector -c vector -- \
   >
     sh -c 'echo $AWS_ACCESS_KEY_ID'

3. Inspect the logs for credential errors or permission issues.

   Look for lines containing CredentialsNotLoaded (no credentials found) or Invalid credentials (credentials rejected by AWS). For example:

   2026-04-16T18:27:48.827213Z ERROR vector::topology::builder: msg="Healthcheck failed." error=Invalid credentials component_kind="sink" component_type="aws_s3" component_id=s3_logs

   To see which credentials Vector loaded, look for lines matching aws_config::profile::credentials:

   2026-04-16T18:27:48.247566Z  INFO aws_config::profile::credentials: constructed abstract provider from config file chain=ProfileChain { base: AccessKey(Credentials { provider_name: "ProfileFile", access_key_id: "AKIA5WLLPVSPD7JDVSXF", secret_access_key: "** redacted **", expires_after: "never" }), chain: [] }

   These lines show the access key ID in use, which can help confirm whether the correct credentials are being picked up.

Permission denied errors

Verify your IAM policy includes both s3:PutObject and s3:ListBucket permissions. The bucket resource ARN shouldn’t include /* for ListBucket.

High latency

Adjust batch settings for faster delivery:

1
vector:
2
  extraSinks:
3
    s3_logs:
4
      batch:
5
        max_bytes: 5242880    # 5MB
6
        timeout_secs: 60      # 1 minute

Related documentation

• Logs configuration
• Vector Documentation
• AWS S3 Sink Reference