This guide provides steps to upgrade your Astro project from Apache Airflow 2 to Apache Airflow 3.
To upgrade to Airflow 3, your Deployment must first run Astro Runtime 12.x or 13.x, and you must upgrade to the latest available patch version of the Airflow 3–based Runtime.
Before upgrading to Airflow 3, you must test your Dags and dependencies locally to identify and resolve any compatibility issues.
Astro Runtime for 3.0 and higher includes only the provider packages required to run on Astro. Previous versions contained additional providers, and if you are using those providers you must explicitly add them to your requirements.txt.
Follow these steps to prepare your local project:
Test your project locally to identify any compatibility issues:
Use the results of this test to identify compatibility issues. Address any compatibility issues by updating your Dags, custom operators, and dependencies. Common changes include:
In your local Astro project, update the Dockerfile to use an Airflow 3 image:
Next, run the following to execute any unit tests:
For more information, see testing your Astro project locally.
<astro-runtime-version> with the specific Astro Runtime version that includes Airflow 3.x that you want to use.For a final check and to test Dags manually, start Airflow locally:
Refer to the docs on local Airflow development to make and test any changes.
There are two supported upgrade strategies for Airflow 3. Choose the approach that fits your operational needs and workflow requirements.
Pros:
Cons:
To continue with this strategy, proceed to Step 3A.
Pros:
Cons:
Astronomer recommends running the metadata database cleanup DAG before you upgrade. In-place upgrades require a database migration that can take several hours for Deployments with large amounts of task history. Cleaning up old metadata before the upgrade significantly reduces migration time and the risk of issues during the upgrade process.
To continue with this strategy, move to Step 3B.
For more information, see Create a new Deployment.
Deploy your updated project:
For additional customization, refer to docs on deploying Astro project images.
Verify that your Dags are running correctly in the new environment.
Airflow 3 to Airflow 2 rollbacks are temporarily unavailable. Astronomer has identified edge cases that prevent the scheduler and triggerer from starting after rollback. Contact Astro support if you need to roll back a Deployment.
When rollbacks are restored, they require that:
Review the Airflow 3 to Airflow 2 rollback support and requirements before proceeding with this upgrade.
Upgrades can take several hours to complete, especially for Deployments that are more than six months old or have a large amount of task history. Astronomer recommends running the metadata database cleanup DAG before you upgrade to reduce migration time and risk. Contact Astro support if you have concerns or questions about the progress of the upgrade.
Deploy your updated project:
This step is applicable for Deployments configured with Customer Managed Workload identity.
With Runtime 3.0, Python package resolution has switched by default from pip to uv, which is intended as a fast drop in replacement for pip. However, uv has a different behavior for packages that exist on multiple indexes, because pip’s behavior is unsafe against dependency confusion attacks. If you rely on the pip behavior, you can add # ASTRO_RUNTIME_USE_PIP to your requirements.txt file to use pip instead of uv. You can also opt to install your dependencies in a separate RUN line on your Dockerfile instead of using Runtime’s built-in pip installation.
When using astro dev start, you can specify a docker-compose.override.yml file. If you specified any overrides for the webserver container in your override file for Airflow 2, these break your ability to use astro dev start because there is no longer a webserver container in Airflow 3. Replace all references to webserver with api-server.
On Airflow 2, the dags/ folder is located at /usr/local/airflow/dags, but with the addition of Dag versioning in Airflow 3, you can no longer be sure of the exact path of the relevant Dags folder. Instead of using this hardcoded path or the AIRFLOW_HOME environment variable, use paths relative to the current file using the Python keyword file. Note also that because Dag bundling is not enabled when using astro dev start, you don’t find errors from this folder location change until your code is deployed to Astro.
For example, if you use the AIRFLOW_HOME variable to open a file in your Dags folder like the following code example, it breaks in Airflow 3:
On Astro with Airflow 3, the dags/ folder isn’t on the PYTHONPATH, and you can’t customize the PYTHONPATH to include the dags/ folder or any other folders. This is a consequence of Dag versioning and Dag bundles in Airflow 3, where the exact path of the Dags folder isn’t fixed at runtime. Setting PYTHONPATH in your Dockerfile or as an environment variable has no effect on Astro.
If your Airflow 2 Dags imported modules directly from the dags/ folder using an import like from my_utils import helper, where my_utils.py lives in dags/, these imports break on Airflow 3. Imports that include the dags. prefix, such as from dags.my_utils import helper, continue to work.
Astronomer recommends that you move shared modules out of the dags/ folder and into the include/ folder, which remains on the PYTHONPATH, then update your imports. For example:
If you want to keep helper files in the dags/ folder, for example, to avoid an image deploy when you change helper code, update your imports to use the dags. prefix:
See Manage Airflow code for the recommended project structure.
Airflow 3 to Airflow 2 rollbacks are temporarily unavailable. Astronomer has identified edge cases that prevent the scheduler and triggerer from starting after rollback. Contact Astro support if you need to roll back a Deployment.
Astro supports rolling back from Airflow 3 to Airflow 2 but not all Airflow 3.x Deployments can be reverted to all Airflow 2.x versions.