This guide provides everything you need to set up Remote Execution on Astro, including prerequisites, configuration requirements, and a step-by-step checklist.
Remote Execution requires both required and optional components. Understand what you need before starting setup.
You must configure these components for Remote Execution to function:
All required components must be configured before agents can successfully execute tasks. The setup checklist below provides the recommended order.
These components enhance functionality but aren’t required for basic operation:
Follow these steps in order to set up Remote Execution:
Before you begin, ensure you have:
Configure singleProcessOOMKill: true in your kubelet configuration. With this setting, Kubernetes kills only the process that runs out of memory instead of the entire Pod. Without it, one task’s out-of-memory error kills all tasks on the worker and loses all task logs.
See Kubernetes documentation for details.
https://<your-cluster-id>.external.astronomer.run/ to your organization’s allowlist. To find your cluster ID, go to Settings > Clusters in the Astro UI.Install the Remote Execution Agent Helm chart in your Kubernetes cluster:
values.yaml fileSee Register and configure agents.
Configure a secrets backend so agents can access Airflow connections and variables:
See Configure secrets backend.
Configure object storage for passing data between tasks:
Define how agents access your Dag code:
Enable Sentinel to monitor agent health and report status to the orchestration plane. Sentinel detects Pod and component issues and monitors essential integrations such as XCom and secrets backends.
See Sentinel for Remote Execution Agents.
Set up task log export to view logs in external platforms or the Airflow UI:
See Configure logging.
Deploy your Airflow project to the Remote Execution deployment:
See Deploy a Remote Execution project.
Confirm agents are healthy and processing tasks:
After completing setup: