This guide explains how to upgrade an Astronomer Software Deployment from Airflow 1.10.15 to 2.3.
As a follow up to Airflow 2, Airflow 2.3 was released in May 2022 with new features like dynamic task mapping and a Grid view in the Airflow UI. Given the significance of this release, Astronomer is providing full support for Airflow 2.3 until October 2023.
Astronomer strongly recommends upgrading any Astronomer Software Deployments currently running Airflow 1.10.15 to Airflow 2.3.
Airflow 2 was built to be fast, reliable, and infinitely scalable. Among the hundreds of new features both large and small, Airflow 2 includes:
Airflow 2.3 subsequently introduced several powerful features, the most notable of which is dynamic task mapping. For more information on Airflow 2.3, see “Apache Airflow 2.3.0 is here” and the Airflow 2.3.0 changelog.
This setup requires:
Not all Airflow 1.10.15 DAGs work in Airflow 2,. The Airflow 2 upgrade check script can check for compatibility issues in your DAG code.
To run the Airflow 2 upgrade check script and install the latest version of the apache-airflow-upgrade-check package at runtime, open your Astro project and run the following command:
This command outputs the results of tests which check the compatibility of your DAGs with Airflow 2.
In the upgrade check output, you can ignore the following entries:
Fernet is enabled by defaultCheck versions of PostgreSQL, MySQL, and SQLite to ease upgrade to Airflow 2Users must set a kubernetes.pod_template_file valueFor more information about upgrade check functionality, see Upgrade Check Script in Apache Airflow documentation.
Review the results from the Airflow upgrade check script and then update your import statements, DAGs, and configurations if necessary.
All Airflow 2 providers supported a backported package version for Airflow 1.10.15. You can use backported provider packages to test your DAGs with Airflow 2’s functionality in a 1.10.15 environment.
requirements.txt file of the Astro project.For more information, see 1.10.15 Backport Providers in Apache Airflow documentation, or see the collection of Backport Providers in PyPi.
Depending on your DAGs, you might need to make the following changes to make sure your code is compatible with Airflow 2:
dag_run_conf_overrides_params.For other compatibility considerations, see Step 5: Upgrade Airflow DAGs in Apache Airflow documentation.
If the upgrade check script didn’t identify any issues with your existing DAGs and configurations, you’re ready to upgrade to Airflow 2.3.
To upgrade to Airflow 2.3,
Initialize the Airflow upgrade process via the Astronomer UI or CLI.
Update your project’s Dockerfile to use Astro Runtime 5.4, which is Astronomer’s distribution of Airflow 2.3:
Modify all backport providers and replace them with fully supported provider packages. For example, if you were using the Mongo backport provider, replace apache-airflow-backport-providers-mongo with apache-airflow-providers-mongo in your requirements.txt file. For more information, see Airflow documentation on provider packages.
Restart your local environment and open the Airflow UI to confirm that your upgrade was successful.
Deploy your project to Astronomer.
Airflow 2.3 includes changes to the schema of the Airflow metadata database. When you first upgrade to Runtime 2.3, consider the following:
Upgrading to Airflow 2.3 can take 10 to 30 minutes or more depending on the number of task instances that have been recorded in the metadata database throughout the lifetime of your Deployment. During the upgrade, scheduled tasks will continue to execute but new tasks will not be scheduled.
Once you upgrade successfully to Airflow 2.3, you might see errors in the Airflow UI that warn you of incompatible data in certain tables of the database. For example:
These warnings have no impact on your tasks or DAGs and can be ignored. If you want to remove these warning messages from the Airflow UI, contact Astronomer Support. If necessary, Astronomer can remove incompatible tables from your metadata database.