Using external registry backends in Astronomer Software

Astronomer Software requires a Docker Registry to store the Docker Images generated every time a user pushes code or makes a configuration change to an Airflow Deployment on Astronomer.

The default storage backend for this Docker Registry is a Kubernetes Persistent Volume. While this may be sufficient for teams just getting started on Astronomer, Astronomer recommends backing the registry with an external storage solution for any team running in production.

The following are the registry backend tools supported by Astronomer:

This document explains only how to set up a registry for hosting your Deployment images in external cloud object storage. To create a custom registry for Deployment images within your Astronomer Software cluster, see Configure a custom image registry for Deployment images. Or, to host all images in a high-security environment with no connections to public networks or internet, see Install Astronomer in an airgapped environment.

Google Cloud Storage

If you’re running Astronomer Software on Google Cloud Platform (GCP) Google Kubernetes Engine (GKE), Astronomer recommends using Google Cloud Storage (GCS) as a registry backend solution.

Prerequisites

To use GCS as a registry backend solution, you’ll need:

An existing GCS Bucket
Your Google Cloud Platform service account JSON Key
Permissions to create a Kubernetes Secret in your cluster

Update your values.yaml file

Download your GCP service account JSON key from the Google Console. Make sure the service account you use has both the Storage Legacy Bucket Owner and Storage Object Admin roles.
Create a Kubernetes Secret using the downloaded key:

kubectl create secret generic astronomer-gcs-keyfile --from-file astronomer-gcs-keyfile=/path/to/key.json -n <your-namespace>

Add the following to your values.yaml file:

1 astronomer:
2   registry:
3     gcs:
4       enabled: true
5       bucket: my-gcs-bucket

Example:

1 #################################
2 ## Astronomer global configuration
3 #################################
4 global:
5   # Base domain for all subdomains exposed through ingress
6   baseDomain: astro.mydomain.com
7 
8   # Name of secret containing TLS certificate
9   tlsSecret: astronomer-tls
10 
11 #################################
12 ## Nginx configuration
13 #################################
14 nginx:
15   # IP address the nginx ingress should bind to
16   loadBalancerIP: 0.0.0.0
17   preserveSourceIP: true
18 
19 #################################
20 ## SMTP configuration
21 #################################
22 astronomer:
23   houston:
24     config:
25       email:
26         enabled: true
27         smtpUrl: YOUR_URI_HERE
28   registry:
29     gcs:
30       enabled: true
31       bucket: my-gcs-bucket

Push the configuration change to your platform as described in Apply a config change.

AWS S3

If you’re running Astronomer Software on the Amazon Elastic Kubernetes Service (EKS), Astronomer recommends using AWS S3 as a registry backend solution.

Prerequisites

To use AWS S3 as a registry backend solution, you’ll need:

An S3 bucket
Your AWS Access Key
Your AWS Secret Key
Ability to create a Kubernetes Secret in your cluster

Create S3 IAM policy and user

Use the following definition to create a new AWS IAM policy, making sure to replace S3_BUCKET_NAME with your own S3 bucket’s name:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads"
      ],
      "Resource": "arn:aws:s3:::S3_BUCKET_NAME"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject",
        "s3:ListMultipartUploadParts",
        "s3:AbortMultipartUpload"
      ],
      "Resource": "arn:aws:s3:::S3_BUCKET_NAME/*"
    }
  ]
}

Create a new IAM User and attach the Policy. Your access key and secret key are generated and displayed after you create the user.

Create Kubernetes secrets for your key credentials in your Astronomer installation:

1 $ kubectl create secret generic astronomer-s3-access-key --from-literal=accesskey=<your-access-key> -n <your-namespace>
2 $ kubectl create secret generic astronomer-s3-secret-key --from-literal=secretkey=<your-secret-key> -n <your-namespace>

Select one of the following options:

To authenticate to AWS with your registry credentials, add this entry to the values.yaml file:

1   astronomer:
2     registry:
3       s3:
4         enabled: true
5         region: us-east-1
6         regionendpoint: <your-region-endpoint>
7         bucket: <your-bucket-name>
8       extraEnvVars:
9         - name: REGISTRY_STORAGE_S3_REGION
10           value: <your-s3-region>
11         - name: REGISTRY_STORAGE_S3_ACCESSKEY
12             valueFrom:
13               secretKeyRef:
14                 name: astronomer-s3-access-key
15                 key: AWS_ACCESS_KEY_ID
16         - name: REGISTRY_STORAGE_S3_SECRETKEY
17             valueFrom:
18               secretKeyRef:
19                 name: astronomer-s3-secret-key
20                 key: AWS_ACCESS_SECRET_ACCESS_KEY

To authenticate to AWS without providing your registry credentials, add this entry to the values.yaml file:

1   astronomer:
2     registry:
3       s3:
4         enabled: true
5         region: us-east-1
6         regionendpoint: <your-region-endpoint>
7         bucket: <your-bucket-name>
8       extraEnvVars:
9         - name: REGISTRY_STORAGE_S3_REGION
10           value: <your-s3-region>

Push the configuration change to your platform. See Apply a config change.

Enable encryption (Optional)

Create a key in AWS Key Management Service (KMS). During the key creation process you’ll be asked to add “key users”. Add the user created above as a “key user”.

Create Kubernetes secrets for your key credentials:

1 $ kubectl create secret generic astronomer-s3-access-key --from-literal=accesskey=<your-access-key> -n <your-namespace>
2 $ kubectl create secret generic astronomer-s3-secret-key --from-literal=secretkey=<your-secret-key> -n <your-namespace>

Add the following values to your values.yaml file to enable encryption:

1 astronomer:
2   registry:
3     s3:
4       enabled: true
5       region: us-east-1
6       bucket: my-s3-bucket
7       encrypt: true
8       keyid: my-kms-key-id
9     extraEnvVars:
10       - name: REGISTRY_STORAGE_S3_REGION
11         value: <your-s3-region>
12       - name: REGISTRY_STORAGE_S3_ACCESSKEY
13           valueFrom:
14             secretKeyRef:
15               name: astronomer-s3-access-key
16               key: accesskey
17       - name: REGISTRY_STORAGE_S3_SECRETKEY
18           valueFrom:
19             secretKeyRef:
20               name: astronomer-s3-secret-key
21               key: secretkey

Push the configuration change to your platform. See Apply a config change.

Authorize Astronomer to a registry backend using IAM roles (Optional)

To avoid hardcoding credentials for your registry backend, add the following configuration to your values.yaml file:

1 registry:
2     serviceAccount:
3       # Specifies whether a service account should be created
4       create: true
5       # Annotations to add to the service account
6       annotations:
7         eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxxxxx:role/<your-iam-role>
8     s3:
9       enabled: true
10       region: <your-region>
11       bucket: <your-registry-backend>

Then, push the configuration change to your platform. See Apply a config change.

Azure Blob Storage

If you’re running Astronomer Software on Azure Kubernetes Service (AKS), Astronomer recommends using Azure Blob Storage as a registry backend solution.

Prerequisites

To use Azure Blog Storage as a registry backend solution, you’ll need:

Azure Storage Account Name
Azure Account Access Key
Azure Container Name

Configure the registry backend

Create Kubernetes secrets for your key credentials:

1 $ kubectl create secret generic astronomer-azure-access-key --from-literal=accountname=<your-account-name> -n <your-namespace>
2 $ kubectl create secret generic astronomer-azure-secret-key --from-literal=accountkey=<your-account-key> -n <your-namespace>

Add the following to your values.yaml file:

1 astronomer:
2   registry:
3     azure:
4       enabled: true
5       accountname: my-account-name
6       accountkey: my-account-key
7       container: my-container-name
8       realm: core.windows.net
9     extraEnvVars:
10       - name: REGISTRY_STORAGE_AZURE_REGION
11         value: <your-azure-region>
12       - name: REGISTRY_STORAGE_AZURE_ACCOUNTNAME
13           valueFrom:
14             secretKeyRef:
15               name: astronomer-azure-access-key
16               key: accountname
17       - name: REGISTRY_STORAGE_AZURE_ACCOUNTKEY
18           valueFrom:
19             secretKeyRef:
20               name: astronomer-azure-secret-key
21               key: accountkey

If you use Software version 0.37.2 and have to force push images to use Registry V3 with Azure blob storage, you can rollback to Registry V2 to resolve the issue with the following configurations in your values.yaml file.

Houston registry configuration:

1 astronomer:
2   houston:
3     config:
4       registry:
5         version: 2

Images registry configuration:

1 astronomer:
2    images:
3        registry:
4           repository: quay.io/astronomer/ap-registry
5           tag: 3.21.3-2

Push the configuration change to your platform as described in Apply a config change.

1	astronomer:
2	registry:
3	gcs:
4	enabled: true
5	bucket: my-gcs-bucket

1	#################################
2	## Astronomer global configuration
3	#################################
4	global:
5	# Base domain for all subdomains exposed through ingress
6	baseDomain: astro.mydomain.com
7
8	# Name of secret containing TLS certificate
9	tlsSecret: astronomer-tls
10
11	#################################
12	## Nginx configuration
13	#################################
14	nginx:
15	# IP address the nginx ingress should bind to
16	loadBalancerIP: 0.0.0.0
17	preserveSourceIP: true
18
19	#################################
20	## SMTP configuration
21	#################################
22	astronomer:
23	houston:
24	config:
25	email:
26	enabled: true
27	smtpUrl: YOUR_URI_HERE
28	registry:
29	gcs:
30	enabled: true
31	bucket: my-gcs-bucket

1	$ kubectl create secret generic astronomer-s3-access-key --from-literal=accesskey=<your-access-key> -n <your-namespace>
2	$ kubectl create secret generic astronomer-s3-secret-key --from-literal=secretkey=<your-secret-key> -n <your-namespace>

1	registry:
2	serviceAccount:
3	# Specifies whether a service account should be created
4	create: true
5	# Annotations to add to the service account
6	annotations:
7	eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxxxxx:role/<your-iam-role>
8	s3:
9	enabled: true
10	region: <your-region>
11	bucket: <your-registry-backend>

1	astronomer:
2	registry:
3	azure:
4	enabled: true
5	accountname: my-account-name
6	accountkey: my-account-key
7	container: my-container-name
8	realm: core.windows.net
9	extraEnvVars:
10	- name: REGISTRY_STORAGE_AZURE_REGION
11	value: <your-azure-region>
12	- name: REGISTRY_STORAGE_AZURE_ACCOUNTNAME
13	valueFrom:
14	secretKeyRef:
15	name: astronomer-azure-access-key
16	key: accountname
17	- name: REGISTRY_STORAGE_AZURE_ACCOUNTKEY
18	valueFrom:
19	secretKeyRef:
20	name: astronomer-azure-secret-key
21	key: accountkey

1	astronomer:
2	images:
3	registry:
4	repository: quay.io/astronomer/ap-registry
5	tag: 3.21.3-2