Deploy Dags to Astro Private Cloud using the Astro CLI

Dag-only deploys are the fastest way to deploy code to Astro Private Cloud. They are recommended if you only need to deploy changes made to the dags directory of your Astro project.

When this feature is configured for a Deployment, you must still do a full project deploy when you make a change to any file in your Astro project that isn’t in the dags directory, or when you upgrade Astro Runtime.

Dag-only deploys have the following benefits:

• Dag-only deploys are faster than project deploys.
• Deployments pick up Dag-only deploys without restarting. This results in a more efficient use of workers and no downtime for your Deployments.
• If you have a CI/CD process that includes both Dag and image-based deploys, you can use your repository’s permissions to control which users can perform which kinds of deploys.
• Dag deploys transmit significantly less data in most cases, which makes them quicker than image deploys when upload speeds to the Deployment are slow.

You can also configure user permissions so that they can roll back just Dags for Dag-only deploys. See Granular rollback permissions.

Prerequisites

• Astro CLI version 1.23 or later

Enable on an Astronomer cluster

By default, Dag-only deploys are disabled for all Deployments on Astro Private Cloud. All types of Dag deploys require astronomer.houston.config.deployments.configureDagDeployment to be set to true. To enable the Dag-only deploys feature, add global.dagOnlyDeployment.enabled to true in your values.yaml file:

1
global:
2
  dagOnlyDeployment:
3
    enabled: true
4
    serviceAccount:
5
      create: true
6
astronomer:
7
  houston:
8
    config:
9
      deployments:
10
        configureDagDeployment: true

Then, push the configuration change. See Apply a config change.

Customize Dag deploy resources

When you enable Dag-only deploys on a given Deployment, Astro Private Cloud spins up a component in the Deployment called the Dag deploy server. The default resources for the Dag deploy server are 100m CPU and 384 Mi of memory which allows you to push up to 15MB of compressed Dags per deploy. To deploy more than 15MB of compressed Dags at a time, increase the CPU and memory in the resources configuration by 1 CPU and 1.5MB for each additional 15MB of Dags you want to upload. For more information, see How Dag-only deploys work.

1
global:
2
  dagOnlyDeployment:
3
    enabled: true
4
    serviceAccount:
5
      create: true
6
    resources:
7
      requests:
8
        # Update values as required for your cluster
9
        cpu: "100m"
10
        memory: "256Mi"
11
      limits:
12
        # Update values as required for your cluster
13
        cpu: "500m"
14
        memory: "1024Mi"

Dag-deploy service account

Astro Private Cloud automatically creates the necessary service account, role, and rolebindings using the Dag-deploy role templates in the airflow-chart when using the default configuration settings:

1
global:
2
  dagOnlyDeployment:
3
    enabled: true
4
    serviceAccount:
5
      create: true

If serviceAccount.create is not set to true, you must provide your own custom service account templates. Otherwise, the Dag-server Pod fails to create. See Bring your own Service Account for steps about how to configure custom service account templates.

Configure Dag-only deploys on a deployment

By default, Deployments are configured only for complete project image deploys. To enable Dag-only deploys:

1. Open your Deployment in the Astro Private Cloud UI.
2. In the Settings tab under the Dag Deployment section, change the Mechanism to Dag Only Deployment.
3. Click Deploy changes
4. Redeploy your Dags using astro deploy --dags. See Trigger a Dag-only deploy. This step is required because all Dags in your deployed image will be deleted from your Deployment when you enable the feature.

Trigger a Dag-only deploy

Run the following command to trigger a Dag-only deploy:

1
astro deploy --dags <deployment-id>

Deploy Dag-only deploys programmatically

Astro Private Cloud Deployment includes a REST API endpoint that you can use to upload Dags programmatically.

Prerequisites

Create a service account that has access to your Deployment and copy its associated API key. See [Create a service account using the Astro Private Cloud UI](ci-cd#create-a-service-account-using-the-Astro Private Cloud-ui). Alternatively, go to https://app.BASEDOMAIN/token and copy the generated token to authenticate with your own user credentials.

Setup

Your automated workflow must include the following two steps:

1. Create a .tgz file that contains only the dags folder of your Astro project. This file should be accessible from the rest of your automated process. For example, you can do this using the following command:

   1
   tar -czf dags.tgz dags

2. Run a POST request to the endpoint https://deployments.basedomain/<deployment-release-name>/dags/upload to upload your .tgz file to your Deployment. For example, making the request using curl would look similar to the following:

   1
   curl --location 'https://deployments.basedomain/<deployment-release-name>/dags/upload' \
   2
   --header 'Authorization: Bearer <your-service-account-token>' \
   3
   --form 'dags.tar.gz=@<your-dags-tgz-location>'

How Deployments handle code deploys

If you deploy Dags to a Deployment that is running a previous version of your code, then tasks that are running continue to run on existing workers and will only be terminated if they exceed the 24-hour timeout after the deploy of the Dag deploy.

These new workers execute downstream tasks of Dag runs that are in progress. For example, if you deploy to Astronomer when Task A of your Dag is running, Task A continues to run on an old Celery worker. If Task B and Task C are downstream of Task A, they are both scheduled on new Celery workers running your latest code.

This means that Dag runs could fail due to downstream tasks running code from a different source than their upstream tasks. Dag runs that fail this way need to be fully restarted from the Airflow UI so that all tasks are executed based on the same source code.