Configure a Kubernetes namespace pool for Astro Private Cloud

Every Deployment within your Astro Private Cloud installation requires an individual Kubernetes namespace. You can configure a pool of pre-created namespaces to limit Astro Private Cloud access to these namespaces.

When you configure a pool of pre-created namespaces, Astronomer users are required to select a namespace from the pool whenever they create a new Deployment. Once the Deployment is created, Astro Private Cloud marks its corresponding namespace as unavailable. If a Deployment is deleted, its namespace is returned to the pool and is available.

Benefits of pre-created namespaces

A pre-created namespace pool provides the following benefits:

• It limits the cluster-level permissions your organization needs to give to Astro Private Cloud. Astro Private Cloud requires permissions only for the individual namespaces you configure. Pre-created namespaces are recommended when your organization does not want to give Astro Private Cloud cluster-level permissions in a multi-tenant cluster.
• It can reduce costs and resource consumption. By default, Astro Private Cloud allows users to create Deployments until there is no more unreserved space in your cluster. If you use a pool, you can limit the number of active Deployments running at a time. This is especially important if you run other Elastic workloads on the same cluster where Astro Private Cloud runs and need to prevent users from accidentally claiming your entire pool of unallocated resources.

Setup

To create a namespace pool you have the following options:

• Delegate the creation of each namespace, including roles and rolebindings, to the Astronomer Helm chart. This option is suitable for most organizations.
• Create each namespace manually, including roles and rolebindings. This option is suitable if you need to further restrict Astronomer Kubernetes resource permissions. However, using this methodology to limit permissions can prevent Deployments from functioning as expected.

Prerequisites

• Helm.
• kubectl with access to the cluster hosting Astro Private Cloud.

Option 1: Use the Astronomer Helm chart

1. Set the following configuration in your values.yaml file:

1
global:
2
  # set global.disableManageClusterScopedResource=true if not using cluster-roles
3
  # disableManageClusterScopedResources: true
4
  features:
5
    namespacePools:
6
      # if this is false, everything in this section can be ignored. default should be false
7
      enabled: true
8
      namespaces:
9
        # automatically creates namespace, role and rolebinding for commander if set to true
10
        create: true
11
        # this needs to be populated (something other than null) if global.features.namespacePools.enabled is true
12
        names:
13
          - <your-namespace-1>
14
          - <your-namespace-2>

2. Save the changes in your values.yaml file and update your Astro Private Cloud. See Apply a config change.

Based on the namespace names that you specified, Astronomer creates the necessary namespaces and Kubernetes resources. These resources have permissions scoped appropriately for most use-cases.

Once you apply your configuration, you should be able to create new Deployments using a namespace from your pre-created namespace pool. You should also be able to see the namespaces you specified inside your cluster resources.

Option 2: Manually create namespaces, roles, and rolebindings

Complete this setup if you want to further limit the namespace permissions that Astronomer provides by default.

Step 1: Configure namespaces

For every namespace you want to add to a pool, you must create a namespace, role, and rolebinding for Astro Private Cloud to access the namespace with. The rolebinding must be scoped to the astronomer-commander service account and the namespace you are creating.

1. Create a new manifest file for each namespace you want to add to the pool. Replace <your-namespace-name> with the name of the namespace.

   1
   apiVersion: v1
   2
   kind: Namespace
   3
   metadata:
   4
     name: <your-namespace-name>
   5
   ---
   6
   apiVersion: rbac.authorization.k8s.io/v1
   7
   kind: Role
   8
   metadata:
   9
     name: astronomer-commander
   10
     namespace: <your-namespace-name>
   11
   rules:
   12
   - apiGroups: ["*"]
   13
     resources: ["*"]
   14
     verbs: ["list", "watch"]
   15
   - apiGroups: [""]
   16
     resources: ["configmaps"]
   17
     verbs: ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]
   18
   - apiGroups: ["keda.k8s.io"]
   19
     resources: ["scaledobjects"]
   20
     verbs: ["get", "create", "delete"]
   21
   - apiGroups: [""]
   22
     resources: ["secrets"]
   23
     verbs: ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]
   24
   - apiGroups: [""]
   25
     resources: ["namespaces"]
   26
     verbs: ["get", "list", "patch", "update", "watch"]
   27
   - apiGroups: [""]
   28
     resources: ["serviceaccounts"]
   29
     verbs: ["create", "delete", "get", "patch"]
   30
   - apiGroups: ["rbac.authorization.k8s.io"]
   31
     resources: ["roles"]
   32
     verbs: ["*"]
   33
   - apiGroups: [""]
   34
     resources: ["persistentvolumeclaims"]
   35
     verbs: ["create", "delete", "deletecollection", "get", "list", "update", "watch", "patch"]
   36
   - apiGroups: [""]
   37
     resources: ["pods"]
   38
     verbs: ["get", "list", "watch", "delete", "create", "patch"]
   39
   - apiGroups: [""]
   40
     resources: ["endpoints"]
   41
     verbs: ["create", "delete", "get", "list", "update", "watch"]
   42
   - apiGroups: [""]
   43
     resources: ["limitranges"]
   44
     verbs: ["create", "delete", "get", "list", "watch", "patch"]
   45
   - apiGroups: [""]
   46
     resources: ["nodes"]
   47
     verbs: ["get", "list", "watch"]
   48
   - apiGroups: [""]
   49
     resources: ["nodes/proxy"]
   50
     verbs: ["get"]
   51
   - apiGroups: [""]
   52
     resources: ["persistentvolumes"]
   53
     verbs: ["create", "delete", "get", "list", "watch", "patch"]
   54
   - apiGroups: [""]
   55
     resources: ["replicationcontrollers"]
   56
     verbs: ["list", "watch"]
   57
   - apiGroups: [""]
   58
     resources: ["resourcequotas"]
   59
     verbs: ["create", "delete", "get", "list", "patch", "watch"]
   60
   - apiGroups: [""]
   61
     resources: ["services"]
   62
     verbs: ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]
   63
   - apiGroups: ["apps"]
   64
     resources: ["statefulsets"]
   65
     verbs: ["create", "delete", "get", "list", "patch", "watch"]
   66
   - apiGroups: ["apps"]
   67
     resources: ["deployments"]
   68
     verbs: ["create", "delete", "get", "patch","update"]
   69
   - apiGroups: ["autoscaling"]
   70
     resources: ["horizontalpodautoscalers"]
   71
     verbs: ["list", "watch"]
   72
   - apiGroups: ["batch"]
   73
     resources: ["jobs"]
   74
     verbs: ["list", "watch", "create", "delete", "get"]
   75
   - apiGroups: ["batch"]
   76
     resources: ["cronjobs"]
   77
     verbs: ["create", "delete", "get", "list", "patch", "watch"]
   78
   - apiGroups: ["extensions"]
   79
     resources: ["daemonsets", "replicasets"]
   80
     verbs: ["list", "watch"]
   81
   - apiGroups: ["extensions"]
   82
     resources: ["deployments"]
   83
     verbs: ["create", "delete", "get", "list", "patch", "update", "watch"]
   84
   - apiGroups: [""]
   85
     resources: ["events"]
   86
     verbs: ["create", "delete", "patch", "list"]
   87
   - apiGroups: ["extensions"]
   88
     resources: ["ingresses"]
   89
     verbs: ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]
   90
   - apiGroups: ["extensions"]
   91
     resources: ["ingresses/status"]
   92
     verbs: ["update"]
   93
   - apiGroups: ["networking.k8s.io"]
   94
     resources: ["ingresses"]
   95
     verbs: ["get", "create", "delete", "patch"]
   96
   - apiGroups: ["networking.k8s.io"]
   97
     resources: ["ingresses/status"]
   98
     verbs: ["update"]
   99
   - apiGroups: ["networking.k8s.io"]
   100
     resources: ["networkpolicies"]
   101
     verbs: ["create", "delete", "get", "patch"]
   102
   - apiGroups: ["rbac.authorization.k8s.io"]
   103
     resources: ["rolebindings"]
   104
     verbs: ["create", "delete", "get", "patch"]
   105
   - apiGroups: ["authentication.k8s.io"]
   106
     resources: ["tokenreviews"]
   107
     verbs: ["create", "delete"]
   108
   - apiGroups: ["authorization.k8s.io"]
   109
     resources: ["subjectaccessreviews"]
   110
     verbs: ["create", "delete"]
   111
   - apiGroups: ["kubed.appscode.com"]
   112
     resources: ["searchresults"]
   113
     verbs: ["get"]
   114
   - apiGroups: ["policy"]
   115
     resources: ["poddisruptionbudgets"]
   116
     verbs: ["create", "delete", "get"]
   117
   - apiGroups: [""]
   118
     resources: ["pods/log"]
   119
     verbs: ["get", "list"]
   120
   - apiGroups: [""]
   121
     resources: ["pods/exec"]
   122
     verbs: ["get", "create"]
   123
   ---
   124
   apiVersion: rbac.authorization.k8s.io/v1
   125
   kind: RoleBinding
   126
   metadata:
   127
     name: deployment-commander-rolebinding
   128
     namespace: <your-namespace-name>
   129
   roleRef:
   130
     apiGroup: rbac.authorization.k8s.io
   131
     kind: Role
   132
     name: astronomer-commander # Should match name of Role
   133
   subjects:
   134
   - namespace: astronomer # Should match namespace where SA lives
   135
     kind: ServiceAccount
   136
     name: astronomer-commander # Should match service account name

2. Save this file and name it <your-namespace-name>.yaml.

3. For each namespace you will configure in global.features.namespacePools.namespaces.names, run kubectl apply -f <your-namespace-name>.yaml.

Step 2: Configure a namespace pool in Astronomer

1. Set the following values in your values.yaml file, making sure to specify all of the namespaces you created in the namespaces.names object:

   1
   global:
   2
     features:
   3
       namespacePools:
   4
         # if this is false, everything in this section can be ignored. default should be false
   5
         enabled: true
   6
         createRbac: false
   7
         namespaces:
   8
           # automatically creates namespace, role and rolebinding for commander if set to true
   9
           create: false
   10
           # this needs to be populated (something other than null) if global.features.namespacePools.enabled is true
   11
           # add the namespace names that were created in the previous setp
   12
           names:
   13
             - <your-namespace-1>
   14
             - <your-namespace-2>

2. Save the changes in your values.yaml and update Astro Private Cloud. See Apply a config change.

Creating Deployments in pre-created namespaces

After you enable the pre-created namespace pool, the UI shows the namespaces you registered as options when you create a new Deployment.

When you create Deployments with the CLI, you are prompted to select one of the available namespaces for your new Deployment.

If no namespaces are available, the UI and CLI show an error message when you try to create a Deployment. Delete the Deployment associated with the namespace to return the namespace to the pool.

Advanced settings

If your namespace configurations require more granularity, use the following settings in your values.yaml file.

Mixing global and advanced settings might result in unexpected behavior. If you use the standard settings, Astronomer recommends that you set global.features.namespacePools.enabled to false.

When using these settings, Astronomer recommends enabling hard deletion for Deployments.

In the following example values.yaml file, these settings are configured so that you don’t configure namespace pools at a global level:

1
global:
2
  # Make vector gather logs from all available namespaces
3
  manualNamespaceNamesEnabled: true
4
  clusterRoles: false
5
6
astronomer:
7
  houston:
8
    config:
9
      deployments:
10
        # Enable manual namespace names
11
        manualNamespaceNames: true
12
        # Pre-created namespace names
13
        preCreatedNamespaces:
14
          - name: <namespace-name-1>
15
          - name: <namespace-name-2>
16
          - name: <namespace-name-x>
17
18
        # Allows users to immediately reuse a pre-created namespace by hard deleting the associated Deployment
19
        # If set to false, you'll need to wait until a cron job runs before the Deployment record is deleted and the namespace is added back to the pool
20
        hardDeleteDeployment: true
21
22
  commander:
23
    env:
24
      - name: "COMMANDER_MANUAL_NAMESPACE_NAMES"
25
        value: true

Troubleshooting namespace pools

My Deployment is in an unknown state

If a Deployment isn’t active, check the Commander Pods to confirm they executed the Deployment commands successfully. When using a pre-created namespace pool with scoped roles, it’s possible that the astronomer-commander service account doesn’t have the permissions necessary to perform a required action. When Commander succeeds, it shows the following notification:

time="2021-07-21T16:30:23Z" level=info msg="releaseName <release-name>, chart astronomer-ee/airflow,     chartVersion 0.20.0, namespace <your-namespace-name>" function=InstallRelease package=helm
time="2021-07-21T16:30:23Z" level=info msg="CHART PATH: /home/commander/.cache/helm/repository/airflow-    0.20.0.tgz\n" function=InstallRelease package=helm

When Commander fails, it shows messages like the following:

time="2022-02-17T22:52:40Z" level=error msg="serviceaccounts is forbidden: User \"system:serviceaccount:astronomer:astronomer-commander\" cannot create resource \"serviceaccounts\" in API group \"\" in the namespace <your-namespace>" function=InstallDeployment package=kubernetes

This error shows that Commander couldn’t create the service accounts in the pre-created namespaces, so you need to update the roles.

My namespace is not returning to the pool

If you’re not using hard deletion, it can take several days for pre-created namespaces to become available after the associated Deployment is deleted. To enable hard deletes, see Delete a Deployment.

My Deployments using NFS deploys stopped working

NFS deploys don’t work if you both use namespace pools and set global.clusterRoles to false in your values.yaml file.