Astro CI/CD templates for AWS CodeBuild

Use the following CI/CD templates to automate deploying Apache Airflow dags from a Git repository to Astro with AWS CodeBuild.

The templates for AWS CodeBuild use image deploy templates. If you have one Deployment and one environment on Astro, use the single branch implementation. If you have multiple Deployments that support development and production environments, use the multiple branch implementation.

If you use the dag-only deploy feature on Astro and are interested in a dag deploy CI/CD template, see Template overview to configure your own. To learn more about CI/CD on Astro, see Choose a CI/CD strategy.

Prerequisites

• An Astro project hosted in a Git repository that AWS CodeBuild can access. See Plan a build in AWS Codebuild.
• An Astro Deployment.
• A Deployment API token, Workspace API token, or Organization API token.
• Access to AWS CodeBuild. See Getting started with CodeBuild.

Each CI/CD template implementation might have additional requirements.

Single branch implementation

To automate code deploys from a single branch to a single Deployment using AWS CodeBuild, complete the following setup in the Git-based repository that hosts your Astro project:

1. In your AWS CodeBuild pipeline configuration, add the following environment variables:

   • ASTRO_API_TOKEN: The value for your Workspace or Organization API token.
   • ASTRO_DEPLOYMENT_ID: The ID for your Deployment.

   Be sure to set the value of your API token as secret.

2. At the root of your Git repository, add a buildspec.yml file that includes the following script:

   1
   version: 0.2
   2
   3
   phases:
   4
     install:
   5
       runtime-versions:
   6
         python: latest
   7
   8
     build:
   9
       commands:
   10
         - echo "${CODEBUILD_WEBHOOK_HEAD_REF}"
   11
         - export ASTRO_API_TOKEN="${ASTRO_API_TOKEN}"
   12
         - curl -sSL install.astronomer.io | sudo bash -s
   13
         - astro deploy "${ASTRO_DEPLOYMENT_ID}" -f

3. In your AWS CodeBuild project, create a webhook event for the Git repository where your Astro project is hosted. If you use GitHub, see GitHub webhook events. When you configure the webhook, select an event type of PUSH.

Your buildspec.yml file now triggers a code push to an Astro Deployment every time a commit or pull request is merged to the main branch of your repository.

Multiple branch implementation

To automate code deploys across multiple Deployments using AWS CodeBuild, complete the following setup.

This setup requires two Deployments on Astro and two branches in your Git repository. The example assumes that one Deployment is a development environment, and that the other Deployment is a production environment. To learn more, see Multiple environments.

1. In your AWS CodeBuild pipeline configuration, add the following environment variables:

   • PROD_ASTRO_API_TOKEN: The value for your production Workspace or Organization API token.
   • PROD_DEPLOYMENT_ID: The Deployment ID of your production Deployment.
   • DEV_ASTRO_API_TOKEN: The value for your development Workspace or Organization API token.
   • DEV_DEPLOYMENT_ID: The Deployment ID of your development Deployment.

   Be sure to set the values for your API tokens as secret.

2. At the root of your Git repository, add a buildspec.yml that includes the following script:

   1
   version: 0.2
   2
   3
   phases:
   4
     install:
   5
       runtime-versions:
   6
         python: latest
   7
   8
     build:
   9
       commands:
   10
         - |
   11
           if expr "${CODEBUILD_WEBHOOK_HEAD_REF}" : "refs/heads/main" >/dev/null; then
   12
             export ASTRO_API_TOKEN="${PROD_ASTRO_API_TOKEN}"
   13
             curl -sSL install.astronomer.io | sudo bash -s
   14
             astro deploy "${PROD_DEPLOYMENT_ID}" -f
   15
           fi
   16
         - |
   17
           if expr "${CODEBUILD_WEBHOOK_HEAD_REF}" : "refs/heads/dev" >/dev/null; then
   18
             export ASTRO_API_TOKEN="${DEV_ASTRO_API_TOKEN}"
   19
             curl -sSL install.astronomer.io | sudo bash -s
   20
             astro deploy "${DEV_DEPLOYMENT_ID}" -f
   21
           fi

3. In your AWS CodeBuild project, create a webhook event for the Git repository where your Astro project is hosted. If you use GitHub, see GitHub webhook events. When you configure the webhook, select an event type of PUSH.

Your buildspec.yml file now triggers a code push to your development Deployment every time a commit or pull request is merged to the dev branch of your repository, and a code push to your production Deployment every time a commit or pull request is merged to the main branch of your repository.