Astro CI/CD templates for CircleCI

CircleCI is a continuous integration and continuous delivery platform that can be used to implement DevOps practices. This document provides sample CI/CD templates to automate deploying Apache Airflow dags from a GitHub repository to Astro using CircleCI.

If you have one Deployment and one environment on Astro, use the single branch implementation. If you have multiple Deployments that support development and production environments, use the multiple branch implementation. If your team builds custom Docker images, use the custom image implementation.

Refer to Template overview to see generic templates expressed as simple shell scripts or configure your own. To learn more about CI/CD on Astro, see Choose a CI/CD strategy.

Prerequisites

• An Astro project hosted in a Git repository that CircleCI can access.
• An Astro Deployment.
• A Deployment API token, Workspace API token, or Organization API token.
• A CircleCI account.

Image deploy templates

Image deploy templates build a Docker image and push it to Astro whenever you update any file in your Astro project.

1
# Use the latest CircleCI pipeline process engine version.
2
# See: https://circleci.com/docs/2.0/configuration-reference
3
version: 2.1
4
5
orbs:
6
  docker: circleci/docker@2.0.1
7
  github-cli: circleci/github-cli@2.0.0
8
9
# Define a job to be invoked later in a workflow.
10
# See: https://circleci.com/docs/2.0/configuration-reference/#jobs
11
jobs:
12
13
  build_image_and_deploy:
14
    docker:
15
      - image: cimg/base:stable
16
    # Add steps to the job
17
    # See: https://circleci.com/docs/2.0/configuration-reference/#steps
18
    steps:
19
      - setup_remote_docker:
20
          version: 20.10.11
21
      - checkout
22
      - run:
23
          name: "Deploy to Astro"
24
          command: |
25
            curl -sSL install.astronomer.io | sudo bash -s
26
            astro deploy ${ASTRO_DEPLOYMENT_ID} -f
27
28
# Invoke jobs with workflows
29
# See: https://circleci.com/docs/2.0/configuration-reference/#workflows
30
workflows:
31
  version: 2.1
32
  wf-build-and-deploy:
33
    jobs:
34
      - build_image_and_deploy:
35
          context:
36
            - <YOUR-CIRCLE-CI-CONTEXT>
37
          filters:
38
            branches:
39
              only:
40
                - <YOUR-BRANCH-NAME>

Dag deploy templates

A dag deploy template uses the --dags flag in the astro deploy command in the Astro CLI to push only dags to your Deployment.

This CI/CD pipeline deploys your dags to Astro when one or more files in your dags folder are modified. It deploys the rest of your Astro project as a Docker image when other files or directories are also modified. For more information about the benefits of this workflow, see Deploy dags only.

Configuration requirements

For each Deployment that you use with dag deploy templates, you must enable dag deploys.

Single branch implementation

To automate code deploys to a Deployment using CircleCI, complete the following setup in a Git-based repository that hosts an Astro project:

1. Set the following environment variables in a CircleCI context:

• ASTRO_API_TOKEN: The value for your Workspace or Organization API token.
• ASTRO_DEPLOYMENT_ID: The ID for your Deployment.

2. In your project repository, create a new YAML file in .circleci/config.yml that includes the following configuration:

1
# Use the latest CircleCI pipeline process engine version.
2
# See: https://circleci.com/docs/2.0/configuration-reference
3
version: 2.1
4
5
orbs:
6
  docker: circleci/docker@2.0.1
7
  github-cli: circleci/github-cli@2.0.0
8
9
# Define a job to be invoked later in a workflow.
10
# See: https://circleci.com/docs/2.0/configuration-reference/#jobs
11
jobs:
12
13
  build_image_and_deploy:
14
    docker:
15
      - image: cimg/base:stable
16
    # Add steps to the job
17
    # See: https://circleci.com/docs/2.0/configuration-reference/#steps
18
    steps:
19
      - setup_remote_docker:
20
          version: 20.10.11
21
      - checkout
22
      - run:
23
          name: "Build image and deploy"
24
          command: |
25
            curl -sSL install.astronomer.io | sudo bash -s
26
            files=($(git diff-tree HEAD --name-only --no-commit-id))
27
            echo ${files}
28
            find="dags"
29
            if [[ ${files[*]} =~ (^|[[:space:]])"$find"($|[[:space:]]) && ${#files[@]} -eq 1 ]]; then
30
              echo "only deploying dags"
31
              astro deploy ${ASTRO_DEPLOYMENT_ID} --dags -f;
32
            else
33
              echo "image deploy"
34
              astro deploy ${ASTRO_DEPLOYMENT_ID} -f;
35
            fi
36
37
# Invoke jobs with workflows
38
# See: https://circleci.com/docs/2.0/configuration-reference/#workflows
39
workflows:
40
  version: 2.1
41
  wf-build-and-deploy:
42
    jobs:
43
      - build_image_and_deploy:
44
          context:
45
            - <YOUR-CIRCLE-CI-CONTEXT>
46
          filters:
47
            branches:
48
              only:
49
                - <YOUR-BRANCH-NAME>

This script checks the diff between your current commit and the HEAD of your branch to which you are pushing the changes to. If the changes are only in dags then it executes a dag-only deploy. Otherwise, it executes an image-based deploy. Make sure to customize the script to use your specific branch and context.

You can customize this script to work for multiple branches as shown in the image-based multi-branch deploy template by creating separate job and workflow for each branch.