Set up APC audit log shipping | Astronomer Documentation

This document explains how to enable the Astro Private Cloud (APC) audit log sidecar and ship events to one supported sink. For background on what the feature does and which configurations are supported, see APC audit logging overview.

Exactly one sink can be enabled per installation for this release. Enabling the sidecar with zero or more than one sink causes a Helm validation error.

Prerequisites

An Astro Private Cloud 2.x installation that you can upgrade with Helm.
Access to the Helm values file used by the installation.
kubectl configured against the target cluster.
Permissions to create or modify cloud resources for the sink you choose:
- CloudWatch on EKS: permissions to create IAM policies and roles, and a CloudWatch log group.
- GCP Cloud Logging on GKE: permissions to create Google service accounts and grant IAM bindings in the target project.
- Elasticsearch: a reachable endpoint and, if required, credentials and a CA certificate.

Choose a sink

Use the AWS CloudWatch Logs sink when the Astro Private Cloud control plane runs on Amazon EKS. Use the GCP Cloud Logging sink when it runs on Google Kubernetes Engine (GKE). Use the Elasticsearch sink on Amazon EKS, GKE, or Azure Kubernetes Service (AKS).

AWS CloudWatch Logs

GCP Cloud Logging

Elasticsearch

Use this sink when the Astro Private Cloud control plane runs on Amazon EKS. The recommended authentication method is IAM Roles for Service Accounts (IRSA). Static AWS credentials held in a Kubernetes secret are supported as a fallback when IRSA isn’t in use on the EKS cluster.

Prerequisites

The Astro Private Cloud control plane runs on Amazon EKS.
The AWS CLI is installed and authenticated against the target account.
For the IRSA path, the EKS cluster has, or can be associated with, an OIDC identity provider.
For the static-credentials path, an IAM principal with permission to write to the target CloudWatch log group.

Environment variables

The following variables are referenced throughout this section. Set them to match your installation before running the commands.

$ export EKS_CLUSTER_NAME="<cluster-name>"
$ export AWS_REGION="<region>"
$ export AWS_ACCOUNT_ID="<account-id>"
$ export K8S_NAMESPACE="astronomer"
$ export HELM_RELEASE="astronomer"
$ export K8S_SA="${HELM_RELEASE}-houston-bootstrapper"
$ export IRSA_ROLE_NAME="HoustonCloudWatchRole"
$ export CW_LOG_GROUP="/astronomer/houston/audit"

Configure IRSA (recommended)

Create the CloudWatch log group

$ aws logs create-log-group \
>   --log-group-name "$CW_LOG_GROUP" \
>   --region "$AWS_REGION"

Optionally set a retention policy on the log group:

$ aws logs put-retention-policy \
>   --log-group-name "$CW_LOG_GROUP" \
>   --region "$AWS_REGION" \
>   --retention-in-days 30

Associate the EKS cluster OIDC provider

$ OIDC_URL=$(aws eks describe-cluster \
>   --name "$EKS_CLUSTER_NAME" \
>   --region "$AWS_REGION" \
>   --query "cluster.identity.oidc.issuer" \
>   --output text)
$ OIDC_ID=${OIDC_URL#https://}
$ 
$ aws iam list-open-id-connect-providers \
>   | grep "$(echo "$OIDC_ID" | awk -F/ '{print $NF}')" \
>   || eksctl utils associate-iam-oidc-provider \
>        --cluster "$EKS_CLUSTER_NAME" \
>        --region "$AWS_REGION" \
>        --approve

Create the IAM policy

$ cat > /tmp/houston-cloudwatch-policy.json <<EOF
$ {
>   "Version": "2012-10-17",
>   "Statement": [
>     {
>       "Sid": "CloudWatchDescribe",
>       "Effect": "Allow",
>       "Action": [
>         "logs:DescribeLogGroups",
>         "logs:DescribeLogStreams"
>       ],
>       "Resource": "arn:aws:logs:${AWS_REGION}:${AWS_ACCOUNT_ID}:log-group:*"
>     },
>     {
>       "Sid": "CloudWatchWriteAuditGroup",
>       "Effect": "Allow",
>       "Action": [
>         "logs:CreateLogGroup",
>         "logs:CreateLogStream",
>         "logs:PutLogEvents"
>       ],
>       "Resource": [
>         "arn:aws:logs:${AWS_REGION}:${AWS_ACCOUNT_ID}:log-group:${CW_LOG_GROUP}",
>         "arn:aws:logs:${AWS_REGION}:${AWS_ACCOUNT_ID}:log-group:${CW_LOG_GROUP}:*"
>       ]
>     }
>   ]
> }
$ EOF
$ 
$ aws iam create-policy \
>   --policy-name HoustonCloudWatchLogsPolicy \
>   --policy-document file:///tmp/houston-cloudwatch-policy.json

Create the IRSA role and attach the policy

$ cat > /tmp/trust-policy.json <<EOF
$ {
>   "Version": "2012-10-17",
>   "Statement": [
>     {
>       "Effect": "Allow",
>       "Principal": {
>         "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ID}"
>       },
>       "Action": "sts:AssumeRoleWithWebIdentity",
>       "Condition": {
>         "StringEquals": {
>           "${OIDC_ID}:sub": "system:serviceaccount:${K8S_NAMESPACE}:${K8S_SA}",
>           "${OIDC_ID}:aud": "sts.amazonaws.com"
>         }
>       }
>     }
>   ]
> }
$ EOF
$ 
$ aws iam create-role \
>   --role-name "$IRSA_ROLE_NAME" \
>   --assume-role-policy-document file:///tmp/trust-policy.json
$ 
$ aws iam attach-role-policy \
>   --role-name "$IRSA_ROLE_NAME" \
>   --policy-arn "arn:aws:iam::${AWS_ACCOUNT_ID}:policy/HoustonCloudWatchLogsPolicy"

If the IRSA role already exists and you only need to bind it to a new EKS cluster, use aws iam update-assume-role-policy against the existing role instead of aws iam create-role.

Apply the Helm values override

Add the following to the values file used by the installation and run helm upgrade:

1 astronomer:
2   houston:
3     serviceAccount:
4       annotations:
5         eks.amazonaws.com/role-arn: "arn:aws:iam::<AWS_ACCOUNT_ID>:role/HoustonCloudWatchRole"
6     logging:
7       loggingSidecar:
8         enabled: true
9         cloudwatch:
10           enabled: true
11           region: "<AWS_REGION>"
12           logGroupName: "/astronomer/houston/audit"
13           useIRSA: true

Configure static AWS credentials (fallback)

Use this configuration on EKS when IRSA isn’t in use.

Create the CloudWatch log group

$ aws logs create-log-group \
>   --log-group-name "$CW_LOG_GROUP" \
>   --region "$AWS_REGION"

Create a Kubernetes secret with AWS credentials

$ kubectl create secret generic houston-cloudwatch-creds \
>   --from-literal=aws_access_key_id="<AWS_ACCESS_KEY_ID>" \
>   --from-literal=aws_secret_access_key="<AWS_SECRET_ACCESS_KEY>" \
>   -n "$K8S_NAMESPACE"

The IAM principal whose credentials you use must be allowed to write to the target log group. The policy shown in the IRSA section is a suitable template.

Apply the Helm values override

1 astronomer:
2   houston:
3     logging:
4       loggingSidecar:
5         enabled: true
6         cloudwatch:
7           enabled: true
8           region: "<AWS_REGION>"
9           logGroupName: "/astronomer/houston/audit"
10           useIRSA: false
11           secretName: "houston-cloudwatch-creds"

Verify

After the upgrade completes, confirm that the Vector sidecar is running and that audit events are reaching CloudWatch:

$ HOUSTON_POD=$(kubectl get pods -n "$K8S_NAMESPACE" \
>   -l component=houston \
>   -o jsonpath='{.items[0].metadata.name}')
$ 
$ kubectl logs -n "$K8S_NAMESPACE" "$HOUSTON_POD" -c vector --tail=20
$ 
$ aws logs tail "$CW_LOG_GROUP" --region "$AWS_REGION" --since 5m

Perform any action that the APC API audits, such as creating a Workspace, and confirm a matching event appears in the log group within a few seconds.

Disable audit log shipping

To stop shipping audit events, set the sidecar to disabled and run helm upgrade:

1 astronomer:
2   houston:
3     logging:
4       loggingSidecar:
5         enabled: false

helm upgrade removes the Vector sidecar from the next Pod restart. The APC API continues to emit audit events to standard output, so you can still inspect recent events with kubectl logs on the APC API and APC Worker Pods.

Common issues

Helm reports 'supports exactly one sink at a time'

The sidecar is enabled with more than one sink. Edit the values file so that only one of cloudwatch.enabled, gcpCloudLogging.enabled, or elasticsearch.enabled is true, then run helm upgrade again.

Helm reports 'requires at least one supported sink'

The sidecar is enabled but no sink is selected. Set one of cloudwatch.enabled, gcpCloudLogging.enabled, or elasticsearch.enabled to true, or set loggingSidecar.enabled to false.

Helm reports a required GCP Cloud Logging value is missing

gcpCloudLogging.projectId, gcpCloudLogging.resource.location, and gcpCloudLogging.resource.clusterName are required when GCP Cloud Logging is enabled. Whitespace-only values are also rejected. Set all three to concrete values and run helm upgrade again.

Vector reports AccessDenied errors on CloudWatch

The IAM principal that Vector uses can’t write to the target log group.

For IRSA, check that the IRSA role trust policy names the correct OIDC provider, namespace, and service account, and that HoustonCloudWatchLogsPolicy is attached to the role. Also verify that the eks.amazonaws.com/role-arn annotation on houston.serviceAccount.annotations points to the same role ARN.
For static credentials, check that houston-cloudwatch-creds contains valid aws_access_key_id and aws_secret_access_key values and that the corresponding IAM user or role is allowed to write to the target log group.

GCP Cloud Logging rejects entries with HTTP 400

The monitored resource is invalid. Confirm that resource.type is k8s_container, and that resource.location and resource.clusterName match the GKE cluster as it appears in Cloud Logging.

Vector can't connect to Elasticsearch

Check that endpoint is reachable from within the cluster and that, if TLS is in use, the CA certificate in caSecretName signs the server certificate presented by the endpoint. If basic auth is enabled, verify that the username and password in houston-elasticsearch-creds are correct.

Next steps

For a field-by-field description of each value under houston.logging.loggingSidecar, see Audit logging configuration reference.
For the shape of each audit event and the list of audited operations, see Audit log schema and operations.

$	export EKS_CLUSTER_NAME="<cluster-name>"
$	export AWS_REGION="<region>"
$	export AWS_ACCOUNT_ID="<account-id>"
$	export K8S_NAMESPACE="astronomer"
$	export HELM_RELEASE="astronomer"
$	export K8S_SA="${HELM_RELEASE}-houston-bootstrapper"
$	export IRSA_ROLE_NAME="HoustonCloudWatchRole"
$	export CW_LOG_GROUP="/astronomer/houston/audit"

$	aws logs create-log-group \
>	--log-group-name "$CW_LOG_GROUP" \
>	--region "$AWS_REGION"

$	aws logs put-retention-policy \
>	--log-group-name "$CW_LOG_GROUP" \
>	--region "$AWS_REGION" \
>	--retention-in-days 30

$	OIDC_URL=$(aws eks describe-cluster \
>	--name "$EKS_CLUSTER_NAME" \
>	--region "$AWS_REGION" \
>	--query "cluster.identity.oidc.issuer" \
>	--output text)
$	OIDC_ID=${OIDC_URL#https://}
$
$	aws iam list-open-id-connect-providers \
>	\| grep "$(echo "$OIDC_ID" \| awk -F/ '{print $NF}')" \
>	\|\| eksctl utils associate-iam-oidc-provider \
>	--cluster "$EKS_CLUSTER_NAME" \
>	--region "$AWS_REGION" \
>	--approve

$	cat > /tmp/houston-cloudwatch-policy.json <<EOF
$	{
>	"Version": "2012-10-17",
>	"Statement": [
>	{
>	"Sid": "CloudWatchDescribe",
>	"Effect": "Allow",
>	"Action": [
>	"logs:DescribeLogGroups",
>	"logs:DescribeLogStreams"
>	],
>	"Resource": "arn:aws:logs:${AWS_REGION}:${AWS_ACCOUNT_ID}:log-group:*"
>	},
>	{
>	"Sid": "CloudWatchWriteAuditGroup",
>	"Effect": "Allow",
>	"Action": [
>	"logs:CreateLogGroup",
>	"logs:CreateLogStream",
>	"logs:PutLogEvents"
>	],
>	"Resource": [
>	"arn:aws:logs:${AWS_REGION}:${AWS_ACCOUNT_ID}:log-group:${CW_LOG_GROUP}",
>	"arn:aws:logs:${AWS_REGION}:${AWS_ACCOUNT_ID}:log-group:${CW_LOG_GROUP}:*"
>	]
>	}
>	]
>	}
$	EOF
$
$	aws iam create-policy \
>	--policy-name HoustonCloudWatchLogsPolicy \
>	--policy-document file:///tmp/houston-cloudwatch-policy.json

$	cat > /tmp/trust-policy.json <<EOF
$	{
>	"Version": "2012-10-17",
>	"Statement": [
>	{
>	"Effect": "Allow",
>	"Principal": {
>	"Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ID}"
>	},
>	"Action": "sts:AssumeRoleWithWebIdentity",
>	"Condition": {
>	"StringEquals": {
>	"${OIDC_ID}:sub": "system:serviceaccount:${K8S_NAMESPACE}:${K8S_SA}",
>	"${OIDC_ID}:aud": "sts.amazonaws.com"
>	}
>	}
>	}
>	]
>	}
$	EOF
$
$	aws iam create-role \
>	--role-name "$IRSA_ROLE_NAME" \
>	--assume-role-policy-document file:///tmp/trust-policy.json
$
$	aws iam attach-role-policy \
>	--role-name "$IRSA_ROLE_NAME" \
>	--policy-arn "arn:aws:iam::${AWS_ACCOUNT_ID}:policy/HoustonCloudWatchLogsPolicy"

1	astronomer:
2	houston:
3	serviceAccount:
4	annotations:
5	eks.amazonaws.com/role-arn: "arn:aws:iam::<AWS_ACCOUNT_ID>:role/HoustonCloudWatchRole"
6	logging:
7	loggingSidecar:
8	enabled: true
9	cloudwatch:
10	enabled: true
11	region: "<AWS_REGION>"
12	logGroupName: "/astronomer/houston/audit"
13	useIRSA: true

$	kubectl create secret generic houston-cloudwatch-creds \
>	--from-literal=aws_access_key_id="<AWS_ACCESS_KEY_ID>" \
>	--from-literal=aws_secret_access_key="<AWS_SECRET_ACCESS_KEY>" \
>	-n "$K8S_NAMESPACE"

$	HOUSTON_POD=$(kubectl get pods -n "$K8S_NAMESPACE" \
>	-l component=houston \
>	-o jsonpath='{.items[0].metadata.name}')
$
$	kubectl logs -n "$K8S_NAMESPACE" "$HOUSTON_POD" -c vector --tail=20
$
$	aws logs tail "$CW_LOG_GROUP" --region "$AWS_REGION" --since 5m