Read-only root filesystem

Starting in Astro Private Cloud (APC) 1.0, the root filesystem of every platform container and every Airflow Deployment container is read-only. Kubernetes enforces this by setting readOnlyRootFilesystem: true on each container’s securityContext, which reduces the container attack surface.

This document explains the behavior, lists the locations that are writable by default, and shows how to add additional writable directories when an application needs to write outside of those paths.

Default behavior

• Every container in the APC platform Helm chart, the Airflow Helm chart, and the Airflow operator runs with readOnlyRootFilesystem: true.
• This setting is enforced by the chart templates and can’t be disabled through Helm values.
• For Airflow Deployment Pods, /usr/local/airflow and /tmp are mounted as emptyDir volumes so that Airflow, your Dags, and provider packages can continue to write to these standard locations.
• Other paths inside the container image, such as /, /etc, /var, and /home, are read-only. Any process that tries to create or modify a file outside of a mounted writable volume fails with a Read-only file system error.

Add a writable directory with extraVolumes and extraVolumeMounts

When an application must write to a path that isn’t writable by default, mount an emptyDir (or other ephemeral volume) at that path using the extraVolumes and extraVolumeMounts keys on the relevant component.

The Airflow Helm chart exposes extraVolumes and extraVolumeMounts on each Airflow component, including:

• airflow.scheduler
• airflow.workers
• airflow.triggerer
• airflow.dagProcessor
• airflow.apiServer
• airflow.webserver
• airflow.pgbouncer
• airflow.redis
• airflow.statsd
• airflow.flower
• dagDeploy

The following example mounts an emptyDir at /opt/cache on Airflow workers so a provider package can write its cache:

1
airflow:
2
  workers:
3
    extraVolumes:
4
      - name: provider-cache
5
        emptyDir: {}
6
    extraVolumeMounts:
7
      - name: provider-cache
8
        mountPath: /opt/cache

Apply the same pattern to other components by setting extraVolumes and extraVolumeMounts under the appropriate component key.

Size the volume

By default, emptyDir volumes can grow until they exhaust the node’s ephemeral storage. Set sizeLimit to bound the volume:

1
airflow:
2
  workers:
3
    extraVolumes:
4
      - name: provider-cache
5
        emptyDir:
6
          sizeLimit: 1Gi
7
    extraVolumeMounts:
8
      - name: provider-cache
9
        mountPath: /opt/cache

For more guidance on sizing ephemeral storage, see Ephemeral storage configuration.

Seed a writable directory with image contents

extraVolumes mounts an empty directory at the target path, which hides any files that were baked into the container image at that location. If your application reads files from a path it also needs to write to, use an init container to copy the image contents into the writable volume before the main container starts.

The following example makes /opt/app writable on Airflow workers while preserving files from the image:

1
airflow:
2
  workers:
3
    extraVolumes:
4
      - name: app-data
5
        emptyDir: {}
6
    extraVolumeMounts:
7
      - name: app-data
8
        mountPath: /opt/app
9
    extraInitContainers:
10
      - name: seed-app-data
11
        image: "{{ .Values.images.airflow.repository }}:{{ .Values.images.airflow.tag }}"
12
        command:
13
          - sh
14
          - -c
15
          - cp -a /opt/app/. /seed/
16
        volumeMounts:
17
          - name: app-data
18
            mountPath: /seed
19
        securityContext:
20
          readOnlyRootFilesystem: true
21
          allowPrivilegeEscalation: false

The init container shares the emptyDir volume with the main container. When the main container starts, the volume already contains the files from the image, and the main container can also write to it.

Redirect application write paths with environment variables

Some applications can be configured to write to a different location through environment variables. When that option exists, mount a writable volume with extraVolumes and extraVolumeMounts as shown earlier, and then point the application at that mount path.

For example, mount an emptyDir at /tmp on Airflow workers and set TMPDIR so libraries that respect it write there:

1
airflow:
2
  workers:
3
    extraVolumes:
4
      - name: tmp
5
        emptyDir: {}
6
    extraVolumeMounts:
7
      - name: tmp
8
        mountPath: /tmp
9
    env:
10
      - name: TMPDIR
11
        value: /tmp

Check each application’s documentation for the variables it supports, such as TMPDIR, HOME, XDG_CACHE_HOME, or vendor-specific cache directory variables. Combining a single writable volume with an environment variable is often cheaper than mounting volumes at each path the application touches.

Troubleshoot read-only filesystem errors

If a Pod or task fails with an error such as Read-only file system, Permission denied, or Errno 30, the application is attempting to write to a path that isn’t backed by a writable volume.

1. Identify the path the application is writing to from the error message or task logs.
2. If the path should already be writable, such as /usr/local/airflow/logs or a directory you mounted with extraVolumes, verify that the Pod has the expected volumes with kubectl describe pod <pod-name>.
3. If the path isn’t writable, either:
   • Add extraVolumes and extraVolumeMounts for the component that runs the application, as shown in the previous sections.
   • Configure the application with an environment variable to write to a path you have mounted as writable.
4. Apply the change with a Helm upgrade and re-run the failing task.