Astro Runtime architecture
Astro Runtime is a production ready, data orchestration tool based on Apache Airflow that is distributed as a container image and is required by all Astronomer products. It is intended to provide organizations with improved functionality, reliability, efficiency, and performance.
Deploying Astro Runtime is a requirement if your organization is using Astro. Astro Runtime includes the following features:
- Timely support for new patch, minor, and major versions of Apache Airflow. This includes bug fixes that have not been released by the open source project but are backported to Astro Runtime and available to users earlier.
- Exclusive features to enrich the task execution experience, including smart task concurrency defaults and high availability configurations.
- Built-in lineage capabilities with OpenLineage Airflow provider package enabling data lineage features on Astro. OpenLineage standardizes the definition of data lineage, the metadata that forms lineage metadata, and how data lineage metadata is collected from external systems. See OpenLineage and Airflow.
- A custom Airflow UI that includes links to Astronomer resources and exposes the currently running the image tag in the footer of all UI pages.
- A custom logging module that ensures Airflow task logs are reliably available to the Astro data plane. (Astro only).
- A custom security manager that enforces user roles and permissions as defined by Astro. (Astro only).
For more information about the features that are available in Astro Runtime releases, see the Astro Runtime release notes.
Runtime versioning
Astro Runtime versions are released regularly and follow semantic versioning. For Airflow 3 versions of Astro Runtime, Astronomer uses the syntax major.minor-patch, including a hyphen between minor and patch, to avoid confusion with the corresponding Airflow major.minor version because Astro Runtime patch releases can include early-access fixes. For Airflow 2, the format is major.minor.patch.
- Major versions are released for significant feature additions. This includes new major or minor versions of Apache Airflow, as well as API or dag specification changes that are not backward compatible.
- Minor versions are released for functional changes. This includes API or dag specification changes that are backward compatible, which might include new minor versions of
astronomer-providers. - Patch versions are released for bug and security fixes that resolve unwanted behavior. This includes new patch versions of Apache Airflow and
astronomer-providers.
Every version of Astro Runtime correlates to an Apache Airflow version. All Deployments must run only one version of Astro Runtime, but you can run different versions of Astro Runtime on different Deployments within a given cluster or Workspace.
For a list of supported Astro Runtime versions and more information on the Astro Runtime maintenance policy, see Astro Runtime versioning and lifecycle policy.
Astro Runtime and Apache Airflow parity
This table lists Astro Runtime releases and their associated Apache Airflow versions.
For version compatibility information, see the Runtime release notes.
Provider packages
The Astro Runtime 3.0 and more recent versions includes the following pre-installed open source provider packages. Providers marked with an asterisk (*) are also installed by default on open source Apache Airflow. See Runtime Provider Reference for a complete list.
- Celery
apache-airflow-providers-celery - Common Compat
apache-airflow-providers-common-compat* - Common IO
apache-airflow-providers-common-io* - Common SQL
apache-airflow-providers-common-sql* - Elasticsearch
apache-airflow-providers-elasticsearch - MySQL
apache-airflow-providers-mysql - OpenLineage
apache-airflow-providers-openlineage - PostgreSQL (Postgres)
apache-airflow-providers-postgres - SMTP
apache-airflow-providers-smtp* - Standard
apache-airflow-providers-standard*
Provider package versioning
If an Astro Runtime release includes changes to an installed version of a provider package that is maintained by Astronomer (e.g., astronomer-providers), the version change is documented in the Astro Runtime release notes.
To determine the version of any provider package installed in your current Astro Runtime image, run:
For example, to find the version of the current package for Astronomer providers, run the following command:
Python versioning
Starting with Astro Runtime 9, if you require a different version of Python than what’s included in the base distribution, you can use a Python distribution of Astro Runtime. See Image types.
If you’re running Astro Runtime 6.0 (based on Airflow 2.4) to Runtime 8, Astronomer recommends that you use the ExternalPythonOperator to run different Python versions in Airflow. See ExternalPythonOperator.
If you’re currently using the KubernetesPodOperator or the PythonVirtualenvOperator in your dags, you can continue to use them to create virtual or isolated environments that can run tasks with different versions of Python.
Python version considerations
Starting with Python version 3.12, Python has removed the module imp. This can cause errors for your dags if you Astro Runtime version 12 or higher, which supports Python Version 3.12 or higher.
See How do I migrate from imp guidance from Python for remediation steps.
Postgres version compatibility
The following table shows which versions Postgres are compatible with each version of Astro Runtime. Note that Postgres versioning is handled automatically on Astro.
Executors
In Airflow, the executor is responsible for determining how and where a task is completed.
In all local environments created with the Astro CLI, Astro Runtime runs the Local executor. On Astro and Astronomer Software, you can use the Astro executor, Celery executor, or the Kubernetes executor.
Image tag conventions
Astro Runtime image tags encode the Runtime version, optional OS variant, optional Python version, and optional base designation into a single string. The general format differs between Airflow 3.x and 2.x.
Airflow 3.x:
Airflow 2.x:
All segments after the version are optional. When multiple segments are present, they must appear in the order shown above.
Floating tags
For Airflow 3.x, you can use a tag without a patch version to always pull the most recent patch release within that minor version. For example, 3.1 resolves to the latest patch of Runtime 3.1 with the default Python version, and 3.1-base resolves to the latest patch base image.
For production deployments, Astronomer recommends pinning a specific patch version for reproducibility.
Tag examples
Image types
Astro Runtime is distributed by default as a Debian-based container image. An Astro Runtime image must be specified in the Dockerfile of your Astro project.
Container registry URLs
Starting with Airflow 3.x, Runtime images are hosted at a new registry:
Airflow 2.x images are still available under the original registry:
but can also be pulled from the new domain as an alternative:
You can modify this image tag in your Astro project Dockerfile to use different versions of Astro Runtime. The following sections explain each image type and how to specify them.
Base images
The base Astro Runtime Docker image has the following format:
Airflow 3.x:
Airflow 2.x:
A base Astro Runtime image is recommended for complex use cases that require additional customization, such as installing Python packages from private sources.
For all other cases, Astronomer recommends using non-base images, which incorporate ONBUILD commands that copy and scaffold your Astro project directory so you can more easily pass those files to the images running each core Airflow component.
Python version images
Starting with Astro Runtime 9, Astronomer maintains different Astro Runtime images for each supported Python version. Python version images have the following format:
Airflow 3.x:
Airflow 2.x:
Operating system variant images
Starting with Astro Runtime 11, Astronomer offers images built on RHEL UBI 9 as an alternative to the default Debian-based images. RHEL-based images are useful for organizations with compliance or security requirements that mandate a Red Hat-certified base.
To select a RHEL UBI 9 image, add -ubi9 after the runtime version in your image tag. If you omit the OS variant segment, you get the Debian-based image (the default).
Airflow 3.x:
Airflow 2.x:
You can combine the OS variant with other image type segments. For example, to use a RHEL UBI 9 base image with Python 3.11:
Airflow 3.x:
Airflow 2.x:
UBI 9 is currently the only alternative OS variant. Additional variants may be offered in the future.
For availability details by Runtime version, see Operating system support.
Slim images
Starting with Astro Runtime 11, Astronomer maintains a slim Astro Runtime image. Slim Astro Runtime images only include the dependencies required for the basic functionality of Astro. Providers marked with an asterisk (*) are also installed by default on open source Apache Airflow. The providers installed in the slim image are:
- Common IO
apache-airflow-providers-common-io* - Common SQL
apache-airflow-providers-common-sql* - FAB
apache-airflow-providers-fab* - FTP
apache-airflow-providers-ftp* - HTTP
apache-airflow-providers-http* - IMAP
apache-airflow-providers-imap* - SMTP
apache-airflow-providers-smtp* - SQLite
apache-airflow-providers-sqlite* - Celery
apache-airflow-providers-celery - Elasticsearch
apache-airflow-providers-elasticsearch - MySQL
apache-airflow-providers-mysql - PostgreSQL (Postgres)
apache-airflow-providers-postgres astronomer-kubernetes-executor
Provider packages in Astro Runtime 3.0 and higher
Astro Runtime for 3.0 and higher includes only the provider packages required to run on Astro. All Airflow 3.x images are considered “slim” and do not contain extra providers. If you need additional providers, you must explicitly add them to your requirements.txt file.
Use the slim Astro Runtime image if you want faster local builds and deploys, smaller footprint for security vulnerabilities and dependency conflicts, or you don’t require the packages included in the default Astro Runtime distribution.
The slim Astro Runtime image has the following format:
Airflow 2.x:
Combine image types
Image types are additive, meaning that you can combine multiple type segments in your image tag to specify an image with multiple variations. Segments must appear in the order defined in Image tag conventions.
Airflow 2.x:
You can add or remove any optional segments as needed. For example, to use the RHEL UBI 9 base image with Python 3.11 on Airflow 3.x:
For Airflow 2.x, to use the slim base image with Python 3.11 on Debian:
The -slim segment applies only to Airflow 2.x. All Airflow 3.x images include only the providers required to run on Astro by default.
Operating system support
The following table lists the operating systems and architectures supported by each Astro Runtime version.
¹ RHEL UBI 9 images are officially supported starting with Astro Runtime 12.5.0 in the 12.x series, and available experimentally starting with Astro Runtime 11.13.0 in the 11.x series.
² Astro Runtime 11.13.0 and later support RHEL UBI 9.
³ Astro Runtime 12.3.0 and later support RHEL UBI 9.
Astro Runtime 6.0.4 and later images are multi-arch and support AMD64 and ARM64 processor architectures for local development. Docker automatically uses the correct processor architecture based on the computer you are using.
