Upgrade Astronomer Software

For Astronomer Software customers, new product features are regularly made available in stable and long-term support (LTS) releases as described in Release and lifecycle policy. Patch versions of Astronomer Software with additional bug and security fixes are also released on a regular basis.

Follow this guide to upgrade to any version of Astronomer Software. For information on new features and changes, refer to Software release notes. Find detailed information about the component image versions in each Astronomer Software version.

A few notes before you get started:

• The upgrade process will not affect running Airflow tasks as long as upgradeDeployments.enabled=false is set in your upgrade script.
• Updates will not cause any downtime to Astronomer services, including the Software UI, the Astro CLI, and the Houston API.
• To avoid potential complications, perform Astronomer Software upgrades in order and include all minor versions in your upgrade sequence.

Step 1: Review upgrade considerations

The Upgrade considerations section of this document contains upgrade information for specific Astronomer versions. Review these notes before starting the upgrade process. To avoid potential complications, perform Astronomer Software upgrades in order and include all minor versions in your upgrade sequence.

Step 2: Check permissions

You need Astronomer System Admin permissions to complete version upgrades. To confirm that you’re a System Admin, check that you have access to the System Admin menu in the Software UI:

You also need permissions to create Kubernetes resources. To confirm that you have the required permissions, run the following commands:

1
kubectl auth can-i create pods --namespace <your-astronomer-namespace>
2
kubectl auth can-i create sa --namespace <your-astronomer-namespace>
3
kubectl auth can-i create jobs --namespace <your-astronomer-namespace>

If all commands return yes, then you have the appropriate Kubernetes permissions.

Step 3: Fetch images from Astronomer’s Helm template

1. To gather the list of images, use one of the following methods:

1
CHART_VERSION=<v-prefixed version of the Astronomer Software platform chart>
2
UNPREFIXED_CHART_VERSION=${CHART_VERSION#v}
3
curl -s https://updates.astronomer.io/astronomer-software/releases/astronomer-${UNPREFIXED_CHART_VERSION}.json | jq -r '(.astronomer.images, .airflow.images) | to_entries[] | "\(.value.repository):\(.value.tag)"'| sort

2. Copy the above images to the container registry using the naming scheme you configured when you set up a custom image registry.

Step 4: Back up your database

Before you perform an upgrade, back up your Astronomer database. To create a backup, follow the instructions provided by your cloud provider, or ask your organization’s database administrator for assistance.

Step 5: Check the status of your Kubernetes Pods

Then, ensure that all Kubernetes Pods in your Astronomer namespace are healthy by running:

1
kubectl get pods -n <your-astronomer-namespace>

All Pods should be in either the Running or Completed state. If any of your Pods are in a CrashLoopBackOff state or are otherwise unhealthy, make sure that is the expected behavior before you proceed.

Step 6: Get a copy of your values.yaml file and confirm values

1. Run the following command to retrieve your current platform configuration:

   1
   helm get values <your-platform-release-name> -n <your-platform-namespace>  > values.yaml

2. Review this configuration. If you see the line "USER-SUPPLIED VALUES:", delete it.

3. Create a copy of values.yaml called old_values.yaml. Save this in case you need to roll back your upgrade.

Step 7: Verify your current platform version

To verify your current version of Astronomer, run:

1
helm list --all-namespaces | grep astronomer

Step 8: Run Astronomer’s upgrade script

Review and run the script below to upgrade to the version of your choice.

1
#!/bin/bash
2
set -xe
3
4
RELEASE_NAME=<astronomer-platform-release-name>
5
NAMESPACE=<astronomer-platform-namespace>
6
ASTRO_VERSION=<astronomer-version>
7
8
helm repo add --force-update astronomer https://helm.astronomer.io
9
helm repo update
10
11
# upgradeDeployments false ensures that Airflow charts are not upgraded when this script is ran
12
# If you deployed a config change that is intended to reconfigure something inside Airflow,
13
# then you may set this value to "true" instead. When it is "true", then each Airflow chart will
14
# restart. Note that some stable version upgrades require setting this value to true regardless of your own configuration.
15
# If you are currently on Astronomer Software 0.25, 0.26, or 0.27, you must upgrade to version 0.28 before upgrading to 0.29. A direct upgrade to 0.29 from a version lower than 0.28 is not possible.
16
helm upgrade --namespace $NAMESPACE \
17
            -f ./values.yaml \
18
            --reset-values \
19
            --version $ASTRO_VERSION \
20
            --debug \
21
            --timeout 20m \
22
            --set astronomer.houston.upgradeDeployments.enabled=false \
23
            $RELEASE_NAME \
24
            astronomer/astronomer $@

Make sure to substitute the following 3 variables with your own values:

• <astronomer-platform-release-name>
• <astronomer-platform-namespace>
• <astronomer-patch-version>

To upgrade to Astronomer Software v0.33.0, for example, set ASTRO_VERSION=0.33.0.

Step 9: Confirm that the installation was successful

If the upgrade was successful, you should be able to:

• Log in to Astronomer at https://app.<BASEDOMAIN>.
• See Workspaces and Airflow Deployments in the Software UI.
• Access the Settings tab for each of your Deployments in the Software UI.
• See metrics on the Metrics tab in the Software UI.
• Run $ astro deploy for an existing Deployment using the Astro CLI.
• Open the Airflow UI for each of your Deployments.
• Access task logs for your DAGs in the Airflow UI.
• Create a new Airflow Deployment that is healthy.

If there is a problem creating a new Airflow Deployment, check the commander logs to troubleshoot. Here is an example of what to look for:

2022-04-14T05:10:45 INFO Calling commander method #updateDeployment
2022-04-14T05:10:48 INFO Response from #updateDeployment: {"result":{"message":"values don't meet the specifications of the schema(s) in the following chart(s):\nairflow:\ ... },"deployment":{}}
2022-04-14T05:10:48 INFO Deployment some-deployment successfully updated

Make changes as needed and rerun the upgrade command from Step 7. Do not continue to Step 8 until you have successfully created a new Airflow Deployment.

Step 10: Upgrade the Astro CLI

Each Software version is compatible only with specific versions of the Astro CLI. Ensure that all users in your organization are using the latest compatible version of the Astro CLI for your Software version. See Version compatibility reference.

For standard upgrade steps, see Upgrade the CLI. To upgrade from a pre-1.0 version of the CLI to version 1.0+, see Upgrade to Astro CLI version 1.0+.

Upgrade considerations

This topic contains information about upgrading to specific versions of Astronomer Software. This includes notes on breaking changes, database migrations, and other considerations that might depend on your use case.

To avoid extended service disruptions, Astronomer recommends upgrading Astronomer Software to a compatible version before you upgrade Kubernetes. To view Astronomer Software and Kubernetes compatibility information, see Version compatibility reference for Astronomer Software.

Supported upgrade paths

If you’re upgrading through multiple Astronomer Software versions in a single upgrade process, review the following table to ensure that you’re following the correct upgrade path. If your combination of Current version and Target version isn’t listed, you can upgrade directly from your current version to the target version.

Upgrade to Kubernetes 1.25

Deprecation of PodSecurityPolicy

In Kubernetes 1.25, PodSecurityPolicies (PSPs) were deprecated in favor of Pod Security Standards. If you were using PSPs on your Astronomer Software installation, consider the following options for moving away from PSPs on Kubernetes 1.25:

• Run the following command to set a baseline Pod Security Standard for all Pods in your Astronomer namespace:

  1
  kubectl label --overwrite ns <your-namespace> \
  2
  pod-security.kubernetes.io/enforce=baseline \
  3
  pod-security.kubernetes.io/enforce-version=v1.25

  If you chose this option, complete the following additional setup to satisfy the baseline policy:

  • Export logs using container sidecars
  • Use an external Elasticsearch instance for Airflow task log management

• Implement Pod Security through a third-party Open Policy Agent tool such as GateKeeper.

Upgrade to Kubernetes 1.24 (Azure only)

To use Kubernetes 1.24 and later on Azure, you must set nginx.ingressAnnotations.service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: "/healthz" in your values.yaml file. See Apply a platform config change.

Upgrade to Kubernetes 1.22

If you’re upgrading to Astronomer Software 0.29 or later and Kubernetes 1.22 at the same time, complete your upgrades in the following order:

1. Follow the standard Software upgrade procedure as described in this document.

2. For each Deployment, run the following command to upgrade the Deployment to use the latest version of the Airflow Helm chart:

   1
   kubectl exec -it `kubectl get pods -l component=houston --no-headers -n <deployment-namespace>` -n <deployment-namespace> -- yarn run upgrade-deployments

3. Upgrade Kubernetes to version 1.22.

Upgrade to Astronomer Software 0.37

Astronomer recommends migrating to v0.37.4 or higher if upgrading to v0.37 due to changes in ephemeral storage behavior for versions 0.37.0-0.37.3.

Ephemeral storage behavior fix in 0.37.4

Changes were introduced in versions 0.37.0 to 0.37.3 that resulted in every KubernetesExecutor task getting ephemeral storage, whether explicitly requested in the dag code or not. Prior to 0.37.0, any task Pod created by the KubernetesExecutor without defined resource quotas fell back to default quota limits of 1AU.

The configuration introduced in 0.37.4, houston.deployments.executors.workers.ephemeralstorage.disabled, default True, reverts this behavior so that only tasks that explicitly request ephemeral storage receive it, and there is no fallback behavior if dag code does not explicitly request ephemeral storage.

Upgrades with no extra steps

• If you are upgrading from versions 0.34.x, 0.35.x, or 0.36.x, you can upgrade directly to 0.37.4 without any extra steps required.
• If you do not use the KubernetesExecutor for any dags you can upgrade to 0.37.4+ from any versions without any extra configuration steps required.
• If you are upgrading from versions 0.37.0-0.37.3 and you have not defined values for ephemeral storage at the global level or in your Houston Config, you can upgrade to 0.37.4 without any extra configuration steps required.
• If you are upgrading from versions 0.37.0-0.37.3 and you have defined values for ephemeral storage in your dag-code or through Pod overrides, you can upgrade to 0.37.4 without any extra configuration steps required.

Upgrade steps for 0.37.0-0.37.3

If you are upgrading from versions 0.37.0-0.37.3 and use the KubernetesExecutor, additional configuration is required if you meet any of the following criteria:

• You do not define ephemeral storage in dag code or via Pod override
• Do not define ephemeral storage amounts in the Houston configuration
• Rely on global ephemeral storage definitions

Then, to avoid task failures or scalability issues, you must either:

1. Update your dag code to request ephemeral storage and keep ephemeralstorage.disabled:true.
2. Set houston.deployments.executors.workers.ephemeralstorage.disabled, to False and define ephemeral storage defaults in the Houston config.

Astronomer Certified removal in 0.37.0

Astronomer Certified (AC) is no longer supported by Astronomer Software starting in Software v0.37. You must upgrade all existing Deployments from AC to Runtime to upgrade to Software 0.37.

See Migrate to Runtime for migration instructions.

Upgrade to Postgres 17 in Astronomer Software 0.37.1

Astronomer Software 0.37.1 uses Postgres 17 by default. If you are using Astronomer provided in-cluster postgres database, Astronomer recommends that you configure Astronomer Software to stay on Postgres 15 when you upgrade to 0.37.1. To do so:

• Set global.postgresql.image.tag=<latest-version-15-tag> in your values.yaml file.
• Complete the upgrade as documented.

You may then opt to upgrade to Postgres 17 at a later date. See How to upgrade PostgreSQL in Docker and Kubernetes.

Upgrade to Astronomer Software 0.35

Security issue with DAG-only deploys resolved in 0.35.1

Due to an issue with the DAG-only Deploy feature in Astronomer Software version 0.35.0, there is a risk that users could potentially access DAGs across different namespaces or Airflow deployments without proper permissions. This issue does not affect Deployments that do not use DAG-only deploys.

To address this issue do not use DAG-only deploys if you use versions 0.35. Only enable DAG-only deploys when you are able to upgrade to version 0.35.1.

Upgrade to Astronomer Software 0.34

Security issue with DAG-only deploys resolved in 0.34.3

Due to an issue with the DAG-only Deploy feature in Astronomer Software version 0.34.0-0.34.2, there is a risk that users could potentially access DAGs across different namespaces or Airflow deployments without proper permissions. This issue does not affect Deployments that do not use DAG-only deploys.

To address this issue do not use DAG-only deploys if you use versions 0.34.0-0.34.2. Only enable DAG-only deploys when you are able to upgrade to version 0.34.3 or 0.35.1.

Upgrade to Astronomer Software 0.32

Upgrade to Postgres 15

Astronomer Software uses Postgres 15 by default. If you are using an in-cluster database solution, Astronomer recommends that you configure Astronomer Software to stay on Postgres 11.18.0-1 when you upgrade to 0.32. To do so:

• Set global.postgresql.image.tag=<latest-version-11-tag> in your values.yaml file.
• Complete the upgrade as documented.

You can then upgrade to Postgres 15 after the upgrade is complete. See How to upgrade PostgreSQL in Docker and Kubernetes.

Renamed feature flag in Helm configuration

The astronomer.houston.config.deployments.sysAdminScalabilityImprovementsEnabled flag has been deprecated and replaced with astronomer.houston.config.deployments.performanceOptimizationModeEnabled for improved performance across additional Software UI views.

If you set sysAdminScalabilityImprovementsEnabled in your values.yaml file, replace it with performanceOptimizationModeEnabled before upgrading. If you don’t replace the key, the upgrade will fail.

Upgrade to Astronomer Software 0.31

Upgrade Airflow Helm chart to 1.9.4 for all Deployments

If your Deployments are using version 1.9.3 or earlier of the Airflow Helm chart, their task logs will be missing when you upgrade to Astronomer Software 0.31. To upgrade your Deployments’ Airflow Helm chart versions during your platform upgrade, ensure that you set --set astronomer.houston.upgradeDeployments.enabled=true for your upgrade command like in the following example:

1
helm upgrade --namespace $NAMESPACE \
2
            -f ./values.yaml \
3
            --reset-values \
4
            --version $ASTRO_VERSION \
5
            --debug \
6
            --timeout 20m \
7
            --set astronomer.houston.upgradeDeployments.enabled=true \
8
            $RELEASE_NAME \
9
            astronomer/astronomer

New default resource limits and requests

Astronomer Software 0.31 includes new default resource limits and requests on the following resources:

• Alertmanager
• Elasticsearch
• NATS
• PostgreSQL
• STAN
• Nginx
• Grafana
• Blackbox exporter

You might experience OOMKill errors or unexpected behavior after upgrading if you use resources beyond the new default limits. To minimize disruption, view resource usage for these components in Grafana prior to upgrade and compare this usage to the default resource limits in the Astronomer Helm chart.

If your current usage is expected and higher than the default resource limits, update the limits in your values.yaml file prior to upgrading to Astronomer Software 0.31.

Upgrade to Astronomer Software 0.30

Do not run the 0.30 upgrade script with —no-hooks

Using the --no-hooks flag in Step 7 results in the upgrade script skipping a necessary database migration job. For almost all use cases, you should not specify this flag when upgrading to 0.30.

If you do specify the --no-hooks flag, the upgrade script will return a success message even though it failed, resulting in broken behavior in your upgraded environment.

PostgreSQL requirements for Astronomer Software 0.30

The trgm extension for PosgreSQL was enabled in Astronomer Software 0.30. Superuser permissions are required to run the extension. To apply PostgreSQL superuser permissions to an existing user, run the following SQL command:

1
`ALTER USER <username> WITH SUPERUSER;`

If you’re using Azure Database for PostgreSQL as your database backend, you need to enable the pg_trgm extension with the Azure portal or the Azure CLI before upgrading to Astronomer Software 0.30. To configure PostgreSQL extensions in Azure Database for PostgreSQL, see PostgreSQL extensions in Azure Database for PostgreSQL.

If you don’t enable the pg_trgm extension before upgrading Astronomer Software, the upgrade will fail.

Upgrade to Astronomer Software 0.29

Kubernetes duplicate key removal bug in 0.29.3

There is an unresolved Kubernetes bug that occurs when upgrading Helm charts that include duplicate keys in an env array. If you have a Helm chart with duplicate keys and upgrade to Astronomer Software 0.29.3, all key-value pairs with the duplicate key are removed from your environment.

To preserve duplicate keys in your Helm chart, you can either reapply the values after upgrade or ensure that you use the --reset-values flag when running the upgrade script in Step 7.

Resync Astronomer’s signing certificate

As part of the 0.29 release, Astronomer deprecated its usage of kubed for performance and security reasons. Kubed was responsible for syncing Astronomer’s signing certificate to Deployment namespaces and is now replaced by an in-house utility. While this change does not directly affect users, you need to run a one-time command to reset Astronomer’s signing certificate as part of the upgrade process to 0.29.

When upgrading to 0.29 from any earlier minor version, run the following command between Steps 2 and 3 in the standard procedure to annotate the certificate secret:

$
kubectl -n <astronomer-platform-release-namespace> annotate secret <astronomer-platform-release-name>-houston-jwt-signing-certificate "astronomer.io/commander-sync"="platform=astronomer"

If you upgraded to Astronomer Software 0.29 without annotating this secret, run the following command to complete the synchronization:

$
kubectl create job -n <astronomer-platform-release-namespace> --from=cronjob/<astronomer-platform-release-name>-config-syncer upgrade-config-synchronization

Upgrade to Astronomer Software 0.28

Version compatibility

Before upgrading to Astronomer Software 0.28, ensure that the following tools are running compatible versions:

• Kubernetes: Your version must be 1.19 or greater. If you need to upgrade Kubernetes, contact your cloud provider or your Kubernetes administrator.

• Airflow images: You must be using an Astronomer Certified Airflow image, and the version of your image must be 1.10.15 or greater.

  For example, all of the following images would work for this upgrade:

  • quay.io/astronomer/ap-airflow:1.10.15-7-buster
  • quay.io/astronomer/ap-airflow:2.0.0-3-buster-onbuild
  • quay.io/astronomer/ap-airflow:2.0.2-buster-onbuild
  • quay.io/astronomer/ap-airflow:2.2.2-onbuild

• Helm: Your version must be 3.6 ≤ 3.8.

Modify values.yaml values

During Step 5 of the upgrade process, check your values.yaml file to see if you have any configuration listed under astronomer.houston.config.deployments.helm. You must update all key-value pairs in this section to be under astronomer.houston.config.deployments.helm.airflow instead.

For example, consider an existing values.yaml file that includes an override of webserver.allowPodLogReading:

1
astronomer:
2
  houston:
3
    config:
4
      deployments:
5
        helm:
6
          webserver:
7
            allowPodLogReading: true

In this case, you need to modify the configuration to include an airflow key after helm:

1
astronomer:
2
  houston:
3
    config:
4
      deployments:
5
        helm:
6
          airflow: ## added this key as this config is coming from the subchart
7
            webserver:
8
              allowPodLogReading: true

Once you complete this change, compare any values under the airflow section with the default values from airflow-chart and open source Airflow chart to ensure that they are formatted correctly. Incorrectly formatted values for these configurations might result in an error during upgrade.

Upgrading to Astronomer Software 0.25

Version compatibility

Before upgrading to 0.25, ensure that the following software is updated to the appropriate version:

• Astronomer: Your current Software version must be at least v0.23.

• Kubernetes: Your version must be 1.17 or 1.18. If you need to upgrade Kubernetes, contact your cloud provider’s support or your Kubernetes administrator.

• Airflow images: You must be using an Astronomer Certified Airflow image, and the version of your image must be 1.10.5 or greater. In addition, your image should be in the following format:

  quay.io/astronomer/ap-airflow:<airflow-version>-<build-number>-<distribution>-onbuild

  For example, all of the following images would work for this upgrade:

  1
  quay.io/astronomer/ap-airflow:1.10.10-5-alpine3.10-onbuild
  2
  quay.io/astronomer/ap-airflow:2.0.0-3-buster-onbuild
  3
  quay.io/astronomer/ap-airflow:2.0.2-buster-onbuild
  4
  quay.io/astronomer/ap-airflow:1.10.5-9-buster

  Note: While -onbuild and <build-number> are optional, we recommend including them for most upgrades. If you have your own build, test, and publish workflows that are layered on top of the Astronomer Airflow images, then removing <build-number> is appropriate because images including <build-number> are immutable.

• Helm: Your version must be 3.6 or greater.

Change to upgrade script

Upgrading to 0.25 requires a non-standard upgrade script. Ignore Step 4 in the standard upgrade process and run the following command instead:

1
kubectl apply -f https://raw.githubusercontent.com/astronomer/astronomer/v0.25.15-final/migrations/scripts/lts-to-lts/0.23-to-0.25/manifests/upgrade-0.23-to-0.25.yaml