Astro includes features that make upgrading to the latest version of Astro Runtime easy and, more importantly, safe. When leveraging these features, follow these best practices to ensure that your Deployments remain stable after you upgrade:
Use this document to learn more about each of these best practices for upgrading Astro Runtime.
Astronomer allows you to control the version of Astro Runtime that you run your Astro deployments on 1, but we do also have an opinionated best practice that we believe most Astro users would benefit from following.
Astronomer recommends:
In your Dockerfile:
In your requirements.txt file:
Astronomer thinks about the Airflow code in two major categories:
The core is the apache-airflow package plus the providers that are required to run successfully on Astro, for example the astronomer-logging-provider. This provider is part of the core because without it, task logs will not work on Astro.
The optional providers are packages like apache-airflow-providers-google that extend Airflow by providing operators that help you interface with services or perform common tasks without having to write a lot of new code. You only need to install a optional provider packages for the specific services and tasks that you actually use in your Astro deployment.
The main concept underpinning Astronomer’s upgrade recommendations is that you should always strictly prefer the latest version of core, but you may not necessarily want the latest provider versions for the operators that you use. Great pains are taken in core to prevent backwards incompatibilities and bugs that could be disruptive to your Airflow pipelines. Because the slim version of Runtime contains only the core parts of Airflow and no optional providers, you never need to change your Dag code to upgrade to the latest Runtime slim version2, unless you are upgrading from Airflow 2 to Airflow 3, which rarely necessitates code level changes.
Fortunately, the optional providers can be both upgraded and downgraded freely, without requiring rollbacks. Which means that the risk introduced by upgrading pre-maturely is low. In many cases it can be a rational choice to simply upgrade your optional providers to the latest version, run your Dags and see if anything breaks. If you get an unexpected Dag import error or task failure, evaluate if that failure could have been caused by the provider upgrade.
However, for greater safety when working with production deployments, it is prudent to check the provider release notes for the major version you are upgrading to and assess them for any breaking changes that might impact your dags.
The Astro CLI has built-in functionality to identify major version upgrades in packages which could introduce breaking changes to your dags. Astronomer recommends making use of this feature prior to every upgrade.
Use the process described in Upgrade Astro Runtime as the basis for all upgrades. It gives step-by-step guidance on completing an upgrade, as well as version-specific advice. In particular, the following steps should be followed for a smooth upgrade:
Run upgrade tests to anticipate and address problems. This involves a single command that:
Compares dependency versions between the current and upgraded environments.
Checks Dags in the project against the new Airflow version for errors.
Runs a Dag parse test with the new Airflow version.
Produces in new project directory upgrade-test-<current_version>--<upgraded_version>:
Dockerfile.pip freeze output for both versions.A Dependency Compare report listing the upgrades, additions and removals coming in the new Astro runtime version and underlying Airflow package. For example, a “Major Updates” section lists the packages receiving new major versions, and a “Minor Updates” lists the packages receiving new minor versions:
For more details about the upgrade-test command, see Testing your Astro project locally.
Astro users on the Enterprise plan can use Organization dashboards to check whether their presently running Astro Runtime versions are currently maintained, as well as how much time is left before they become unmaintained. This is especially helpful in cases where your Organization has many Deployments to track. See Organization dashboards for steps to access these dashboards.
If you are not on the Astro Enterprise plan, you can still reference the Astro Runtime maintenance and lifecycle policy to see when you’ll need to upgrade your Deployments to a new version of Astro Runtime. A Deployment Health Alert will also be displayed if a Deployment is on an unmaintained Astro Runtime version.
In the case of an emergency, you can roll back to your previous deploy to downgrade the Astro Runtime version. Astro’s rollback functionality is similar to the Git revert command. Rollbacks revert project code only, keeping environment variables and other configuration unchanged.
Some rollbacks require reverting the underlying database schema used by Airflow. Sometimes, data cannot be represented in the older schema and so will be dropped by the rollback process. For example, if you upgrade to Runtime 12, new tasks will have the try number associated with Event Logs, but if you roll back to a previous version, these try numbers will be lost. This data is always data that would not have been present to begin with if you had not upgraded your Runtime, so there is not risk of data loss from the overall process of upgrading and subsequently rolling back.
upgrade-test command can also be used to test the current Runtime version against a downgraded version.