Astro Runtime is a production ready, data orchestration tool based on Apache Airflow that is distributed as a Docker image and is required by all Astronomer products. It is intended to provide organizations with improved functionality, reliability, efficiency, and performance.
Deploying Astro Runtime is a requirement if your organization is using Astro. Astro Runtime includes the following features:
openlineage-airflow package. OpenLineage standardizes the definition of data lineage, the metadata that forms lineage metadata, and how data lineage metadata is collected from external systems. This package enables data lineage on Astro. See OpenLineage and Airflow.For more information about the features that are available in Astro Runtime releases, see the Astro Runtime release notes.
Astro Runtime versions are released regularly and use semantic versioning. Astronomer ships major, minor, and patch releases of Astro Runtime in the format of major.minor.patch.
astronomer-providers and openlineage-airflow.astronomer-providers, and openlineage-airflow.Every version of Astro Runtime correlates to an Apache Airflow version. All Deployments must run only one version of Astro Runtime, but you can run different versions of Astro Runtime on different Deployments within a given cluster or Workspace.
For a list of supported Astro Runtime versions and more information on the Astro Runtime maintenance policy, see Astro Runtime versioning and lifecycle policy.
This table lists Astro Runtime releases and their associated Apache Airflow versions.
For version compatibility information, see the Runtime release notes.
Airflow 2.x images are still available under the original registry:
but can also be pulled from the new domain as an alternative:
The following table lists the Airflow environment variables that have different default values on Astro Runtime as of version 13.2.0. Unlike global environment variables set on the Astro data plane, you can override the values of these variables for specific use cases.
Astro Runtime includes the following pre-installed open source provider packages. Providers marked with an asterisk (*) are also installed by default on open source Apache Airflow.
apache-airflow-providers-common-io*apache-airflow-providers-common-sql*apache-airflow-providers-fab*apache-airflow-providers-ftp*apache-airflow-providers-http*apache-airflow-providers-imap*apache-airflow-providers-smtp*apache-airflow-providers-sqlite*apache-airflow-providers-amazonastronomer-providersastro-sdk-pythonapache-airflow-providers-celeryapache-airflow-providers-cncf-kubernetesapache-airflow-providers-datadogapache-airflow-providers-elasticsearchapache-airflow-providers-googleapache-airflow-providers-microsoft-azureapache-airflow-providers-mysqlapache-airflow-providers-openlineageapache-airflow-providers-postgresapache-airflow-providers-redisAstronomer users specific provider package versions in each release of Astro Runtime. See Astro Runtime provider package reference for a list of all providers installed in each release of Astro Runtime.
Starting with Astro Runtime 9, if you require a different version of Python than what’s included in the base image, you can use a Python versions of Astro Runtime. See Image types.
If you’re running Astro Runtime 6.0 (based on Airflow 2.4) to Runtime 8, Astronomer recommends that you use the ExternalPythonOperator to run different Python versions in Airflow. See ExternalPythonOperator.
If you’re currently using the KubernetesPodOperator or the PythonVirtualenvOperator in your DAGs, you can continue to use them to create virtual or isolated environments that can run tasks with different versions of Python.
Starting with Python version 3.12, Python has removed the module imp. This can cause errors for your DAGs if you Astro Runtime version 12 or higher, which supports Python Version 3.12 or higher.
See How do I migrate from imp guidance from Python for remediation steps. If you’re currently using the KubernetesPodOperator or the PythonVirtualenvOperator in your DAGs, you can continue to use them to create virtual or isolated environments that can run tasks with different versions of Python.
The following table shows which versions Postgres are compatible with each version of Astro Runtime. Note that Postgres versioning is handled automatically on Astro Hosted.
In Airflow, the executor is responsible for determining how and where a task is completed.
In all local environments created with the Astro CLI, Astro Runtime runs the Local executor. On Astro and Astronomer Software, you can use either the Celery executor or the Kubernetes executor.
Astro Runtime is distributed as a Debian-based Docker image. An Astro Runtime image must be specified in the Dockerfile of your Astro project. The default image tag is:
You can modify this image tag in your Astro project Dockerfile to use different versions of Astro Runtime. The following sections explain each image type and how to specify them. For a list of all Astro Runtime Docker images, see Quay.io.
The base Astro Runtime Docker image has the following format:
A base Astro Runtime image is recommended for complex use cases that require additional customization, such as installing Python packages from private sources.
For all other cases, Astronomer recommends using non-base images, which incorporate ONBUILD commands that copy and scaffold your Astro project directory so you can more easily pass those files to the images running each core Airflow component.
Starting with Astro Runtime 9, Astronomer maintains different Astro Runtime images for each supported Python version.
Python version images have the following format:
Starting with Astro Runtime 11.13.0 and 12.3.0, RHEL version images are also available for Python versions 3.11 and 3.12. To select a particular Python and RHEL version combination, use the following format:
Starting with Astro Runtime 11, Astronomer maintains a slim Astro Runtime image. Slim Astro Runtime images only include the dependencies required for the basic functionality of Astro. Providers marked with an asterisk (*) are also installed by default on open source Apache Airflow. The providers installed in the slim image are:
apache-airflow-providers-common-io*apache-airflow-providers-common-sql*apache-airflow-providers-fab*apache-airflow-providers-ftp*apache-airflow-providers-http*apache-airflow-providers-imap*apache-airflow-providers-smtp*apache-airflow-providers-sqlite*apache-airflow-providers-celeryapache-airflow-providers-elasticsearchapache-airflow-providers-mysqlapache-airflow-providers-postgresastronomer-kubernetes-executorUse the slim Astro Runtime image if you want faster local builds and deploys, smaller footprint for security vulnerabilities and dependency conflicts, or you don’t require the packages included in the default Astro Runtime distribution.
Image types are additive, meaning that you can combine multiple types in your image tag to specify an image with multiple variations.
Types must be specified in a specific order in your image tag. The order and format multi-type image tag is:
You can add or remove any types as needed. For example, to use the base, slim Astro Runtime image with Python 3.11, your image tag would be:
If you wanted the same base image using Python 3.11, but you didn’t want it to be slim, you would remove the -slim- from the image tag like in the following example:
The following table lists the operating systems and architectures supported by each Astro Runtime version.
¹ RHEL UBI 9 images are officially supported starting with Astro Runtime 12.5.0 in the 12.x series, and available experimentally starting with Astro Runtime 11.13.0 in the 11.x series.
² Astro Runtime 11.13.0 and later support RHEL UBI 9.
³ Astro Runtime 12.3.0 and later support RHEL UBI 9.
Astro Runtime 6.0.4 and later images are multi-arch and support AMD64 and ARM64 processor architectures for local development. Docker automatically uses the correct processor architecture based on the computer you are using.