Configure logging

Remote Execution Agents generate task logs in your Kubernetes cluster. By default, these logs remain in agent Pods and are lost when Pods terminate. Configure logging to preserve and access these logs.

Logging approaches

External logging provider (recommended)

Export logs to your logging platform (Splunk, Elasticsearch, CloudWatch, etc.) using a logging sidecar. Configure the Airflow UI to display links to external logs.

• See Configure logging sidecar
• See Export logs to external platforms

Object storage (Airflow UI display)

Store logs in object storage (S3, GCS, Azure Blob) and configure the Astro API server to fetch and display them in the Airflow UI. Logs appear after task completion.

• See Display logs in Airflow UI

Real-time streaming to object storage

Extend object storage logging with a Vector sidecar to stream partial logs while tasks run. Provides real-time log visibility in the Airflow UI.

• See Display logs in Airflow UI

Comparison

Prerequisites

• Remote Execution Agent installed and registered
• Workload identity configured for your Kubernetes cluster
• One of the following:
  • External logging: Logging platform endpoint
  • Object storage: S3, GCS, or Azure Blob container
  • Real-time streaming: Object storage + Vector configuration

Next steps

• Configure logging sidecar
• Export logs to external platforms
• Display logs in Airflow UI