Show Remote Execution Agent task logs in Airflow UI

You can display task logs in the Airflow UI by exporting logs to object storage and configuring the Astro API Server to retrieve them. Start by enabling log display after task completion, then optionally extend the setup to stream logs in real time as tasks run.

This guide explains configuring post-task log display and expanding that configuration to support real-time log streaming.

Display task logs after task completion

Set up log uploading so logs are visible in the Airflow UI after task completion. This requires:

• Remote Execution Agent configuration (values.yaml)
• Astro UI Deployment configuration
• Workload identities: write access for the Remote Execution Agent, read access for the Astro API Server

The commonEnv block in the following procedures applies environment variables to all Airflow components in the Remote Execution Agent. These variables configure Airflow’s remote logging so that workers and the triggerer upload completed task logs to object storage:

• AIRFLOW__LOGGING__REMOTE_LOGGING: Turns on Airflow’s remote log upload.
• AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: Names the Airflow connection used to authenticate with object storage.
• AIRFLOW_CONN_<CONN_ID>: Defines that connection inline, so no separate connection record is required. A URI with no credentials (for example, s3://) causes the underlying cloud SDK to use its default credential chain, which picks up the workload identity attached to the Pod.
• AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: Sets the bucket and path prefix that Airflow writes task logs to. Use a Deployment-scoped path so logs from different Deployments don’t collide.
• AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS: Replaces Airflow’s default logging configuration with the Astronomer Runtime configuration, which installs the task log handler used to write logs to and read logs from object storage.
• ASTRONOMER_ENVIRONMENT: Set to cloud for Astro Deployments. Astronomer Runtime reads this value to select Astro-specific logging defaults.

1
commonEnv:
2
  - name: AIRFLOW__LOGGING__REMOTE_LOGGING
3
    value: "True"
4
  - name: AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID
5
    value: "astro_aws_logging"
6
  - name: AIRFLOW_CONN_ASTRO_AWS_LOGGING
7
    value: "s3://"
8
  - name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
9
    value: "s3://<bucket>/<deployment-id>"
10
  - name: AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS
11
    value: "astronomer.runtime.logging.logging_config"
12
  - name: ASTRONOMER_ENVIRONMENT
13
    value: "cloud"

1
  - name: ASTRO_LOGGING_AWS_WEB_IDENTITY_TOKEN_FILE
2
    value: "/tmp/logging-token"

Display task logs during task execution

Once you have post-completion log visibility, you can enable real-time log display. Remote Execution prevents the Airflow API server from reading logs directly from workers until they reach object storage. Use Vector, included in the Remote Execution Agent Helm chart, to upload partial logs while tasks are running.

Prerequisites

Before you configure Vector, ensure that your Remote Execution Deployment is already set up to upload task logs to object storage after task completion.

Enable Vector sidecar

Use Vector to watch for log file changes and upload updates to object storage during task execution.

In your Helm values.yaml:

• Set loggingSidecar.enabled to true to inject the Vector container into each worker and triggerer Pod:

1
loggingSidecar:
2
  enabled: true

• Configure loggingSidecar.volumeMounts to give Vector read access to the Airflow task log files and a writable location for its on-disk checkpoint state:

1
loggingSidecar:
2
  volumeMounts:
3
    - name: task-logs
4
      mountPath: /var/log/airflow/task_logs
5
      readOnly: true
6
    - name: vector-data
7
      mountPath: /var/lib/vector

The task-logs mount is the directory Vector reads from. The vector-data mount stores Vector’s file checkpoints so it can resume uploads after a restart without re-sending lines.

Configure log upload for your cloud provider

The loggingSidecar.config block is the Vector pipeline definition. Each cloud-specific config uses the same three-stage structure:

• sources.airflow_task_logs: Tails the local task log files written by Airflow workers and the triggerer. read_from: beginning ensures Vector starts at the top of each file so partial logs aren’t missed.
• transforms.strip_path_prefix: Removes the local mount path from each event’s file field and writes the result to log_path. This produces an object key that matches the layout Airflow uses when it uploads the complete log after task completion, so the Astro API Server can read both the partial and final logs from the same location.
• sinks.<cloud>: Uploads the transformed events to object storage. key_prefix (or blob_prefix on Azure) uses the log_path field from the transform to place each log under the right object key. batch.max_bytes and batch.timeout_secs control how often Vector flushes — smaller values stream logs to the UI faster but produce more small objects in storage. See Small file problem.

1
loggingSidecar:
2
  config: |
3
    # Vector configuration for Astronomer Remote Execution agents for uploading Airflow task logs to AWS S3
4
5
    sources:
6
      airflow_task_logs:
7
        type: file
8
        include:
9
          - /var/log/airflow/task_logs/**/*.log
10
        read_from: beginning
11
12
    transforms:
13
      strip_path_prefix:
14
        type: remap
15
        inputs: [airflow_task_logs]
16
        source: |
17
          .log_path, err = replace(.file, "/var/log/airflow/task_logs/", "")
18
          if err != null {
19
            abort
20
          }
21
22
    sinks:
23
      s3:
24
        # For more Vector AWS S3 configuration options, see https://vector.dev/docs/reference/configuration/sinks/aws_s3
25
        type: aws_s3
26
        inputs: [strip_path_prefix]
27
        bucket: # set bucket name e.g. airflow_logs
28
        region: # set AWS region e.g. us-east-2
29
        compression: none
30
        encoding:
31
          codec: text
32
        key_prefix: '{{ "{{" }} log_path {{ "}}" }}.'
33
        filename_time_format: "%Y-%m-%dT%H-%M-%S"
34
        filename_append_uuid: false
35
        batch:
36
          max_bytes: 1000000  # configure based on your storage costs and log frequency requirements - see Caveats section below
37
          timeout_secs: 10  # configure based on your storage costs and log frequency requirements - see Caveats section below

1
debug:
2
  type: console
3
  inputs: [strip_path_prefix]
4
  encoding:
5
    codec: json

The following volume configuration creates the shared emptyDir volumes that the Vector sidecar and the Airflow worker or triggerer container both mount. task-logs is the directory Airflow writes logs into and Vector reads from. vector-data holds Vector’s checkpoint state. The worker and triggerer mount task-logs at /usr/local/airflow/logs, which is where Airflow writes by default, while the Vector sidecar mounts the same volume at /var/log/airflow/task_logs to match its sources.airflow_task_logs.include glob.

• Configure workers[*].volumes:

1
volumes:
2
  - name: task-logs
3
    emptyDir: {}
4
  - name: vector-data
5
    emptyDir: {}

• Configure workers[*].volumeMounts:

1
volumeMounts:
2
  - name: task-logs
3
    mountPath: /usr/local/airflow/logs

• Configure triggerer.volumes:

1
volumes:
2
  - name: task-logs
3
    emptyDir: {}
4
  - name: vector-data
5
    emptyDir: {}

• Configure triggerer.volumeMounts:

1
volumeMounts:
2
  - name: task-logs
3
    mountPath: /usr/local/airflow/logs

• Set AIRFLOW__LOGGING__DELETE_LOCAL_LOGS in commonEnv so Airflow removes the local log file after it uploads the complete log to object storage. This keeps the shared task-logs volume from filling up on long-running Pods:

1
commonEnv:
2
  - name: AIRFLOW__LOGGING__DELETE_LOCAL_LOGS
3
    value: "True"

Log upload process

Partial logs are uploaded and displayed as follows:

1. Airflow worker or triggerer writes local task log files, as set by AIRFLOW__LOGGING__LOG_FILENAME_TEMPLATE.
2. Vector watches /var/log/airflow/task_logs/**/*.log and uploads log changes in chunks while the task runs.
3. Vector appends a timestamp to the file name before uploading each chunk.
4. Airflow scans object storage for log chunks when displaying the UI log view.
5. The UI displays all log content to the user.

Caveats

Duplicate log storage

After task completion, Airflow uploads the complete log to object storage and deletes the local copy. This causes duplication:

1. Partial logs from Vector
2. Complete log from Airflow

The Airflow API server deduplicates log lines by timestamp and message. Only storage usage is affected; logs are displayed once.

Small file problem

High-frequency, small log file uploads can create many small objects. This may increase storage costs, load on object storage, or trigger rate limits. Adjust file size and upload frequency in your Vector config to balance performance and cost.

• AWS bills object retrieval at a 128KB minimum on certain storage classes (source).
• A large number of small objects means more object requests (PUTs, GETs, LISTs) and more load on metadata/indexing; this can result in rate limits or latency issues.

Ensure a proper balance between filesize/timeout and log upload frequency in your Vector config.