Figure out how to run Filebeat using workload identity on AKS

[zmoog]

Context

With elastic/beats#XXXX we're adding support for workload identity to the azure-eventhub input. Workload identity uses OpenID Connect (OIDC) federation to authenticate Kubernetes workloads to Azure without storing credentials.

In this issue thread, I want to set up an AKS cluster with workload identity and run Filebeat using the workload_identity authentication method.

Goal

Document how to set up and test workload identity authentication for the azure-eventhub input running in a Kubernetes pod on AKS.

Impact

Provide a reproducible step-by-step guide for setting up and testing the workload identity scenario.

---

Workload Identity on AKS

With workload identity, Filebeat running in a Kubernetes pod authenticates to Event Hubs and Storage Account using a federated token projected into the pod by the workload identity webhook. No secrets or credentials are stored in the cluster.

Prerequisites

• Azure CLI installed and logged in
• `kubectl` installed
• An Event Hub namespace with an event hub (e.g., `logs`)
• A Storage Account for checkpointing

1. Create an AKS cluster with OIDC and workload identity

```sh

Create resource group

az group create --name filebeat-wi-test --location eastus

Create AKS cluster with OIDC issuer and workload identity enabled

az aks create \
--resource-group filebeat-wi-test \
--name filebeat-wi-cluster \
--enable-oidc-issuer \
--enable-workload-identity \
--generate-ssh-keys \
--node-count 1 \
--node-vm-size Standard_B2s

Get credentials

az aks get-credentials \
--resource-group filebeat-wi-test \
--name filebeat-wi-cluster
```

2. Get the OIDC issuer URL

```sh
export AKS_OIDC_ISSUER=$(az aks show \
--resource-group filebeat-wi-test \
--name filebeat-wi-cluster \
--query "oidcIssuerProfile.issuerUrl" \
--output tsv)

echo $AKS_OIDC_ISSUER
```

3. Create a managed identity

```sh
az identity create \
--name filebeat-wi-identity \
--resource-group filebeat-wi-test \
--location eastus

Capture the client ID

export IDENTITY_CLIENT_ID=$(az identity show \
--name filebeat-wi-identity \
--resource-group filebeat-wi-test \
--query clientId \
--output tsv)

echo $IDENTITY_CLIENT_ID
```

4. Assign roles to the identity

The `azure-eventhub` input requires the following two roles:

• Azure Event Hubs Data Receiver on the Event Hub namespace
• Storage Blob Data Contributor on the Storage Account

```sh

Get identity principal ID

export IDENTITY_PRINCIPAL_ID=$(az identity show \
--name filebeat-wi-identity \
--resource-group filebeat-wi-test \
--query principalId \
--output tsv)

Replace with your Event Hub namespace resource ID

export EVENTHUB_NAMESPACE_ID="/subscriptions//resourceGroups//providers/Microsoft.EventHub/namespaces/"

Replace with your Storage Account resource ID

export STORAGE_ACCOUNT_ID="/subscriptions//resourceGroups//providers/Microsoft.Storage/storageAccounts/"

Assign Event Hubs Data Receiver

az role assignment create \
--assignee-object-id $IDENTITY_PRINCIPAL_ID \
--assignee-principal-type ServicePrincipal \
--role "Azure Event Hubs Data Receiver" \
--scope $EVENTHUB_NAMESPACE_ID

Assign Storage Blob Data Contributor

az role assignment create \
--assignee-object-id $IDENTITY_PRINCIPAL_ID \
--assignee-principal-type ServicePrincipal \
--role "Storage Blob Data Contributor" \
--scope $STORAGE_ACCOUNT_ID
```

5. Create a Kubernetes Service Account

```sh
export SERVICE_ACCOUNT_NAMESPACE="default"
export SERVICE_ACCOUNT_NAME="filebeat-wi"

cat <<SAEOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
name: ${SERVICE_ACCOUNT_NAME}
namespace: ${SERVICE_ACCOUNT_NAMESPACE}
annotations:
azure.workload.identity/client-id: "${IDENTITY_CLIENT_ID}"
labels:
azure.workload.identity/use: "true"
SAEOF
```

6. Establish federated identity credential

This creates the trust relationship between the Kubernetes service account and the Azure managed identity:

```sh
az identity federated-credential create \
--name filebeat-federated-credential \
--identity-name filebeat-wi-identity \
--resource-group filebeat-wi-test \
--issuer $AKS_OIDC_ISSUER \
--subject system:serviceaccount:${SERVICE_ACCOUNT_NAMESPACE}:${SERVICE_ACCOUNT_NAME} \
--audiences api://AzureADTokenExchange
```

7. Deploy Filebeat

Since the PR is not merged yet, build a custom Filebeat image and push it to a registry accessible from AKS. Then deploy using a pod spec:

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: default
data:
filebeat.yml: |
filebeat.inputs:
- type: azure-eventhub
enabled: true
auth_type: workload_identity
eventhub: logs
eventhub_namespace: .servicebus.windows.net
consumer_group: $Default
storage_account:

output.elasticsearch:
  hosts: ["<your-elasticsearch-host>"]
  # ... your ES auth config

---

apiVersion: v1
kind: Pod
metadata:
name: filebeat-wi-test
namespace: default
labels:
azure.workload.identity/use: "true"
spec:
serviceAccountName: filebeat-wi
containers:

• name: filebeat
  image: /filebeat:latest
  args: ["-e", "-v"]
  volumeMounts:
  • name: config
    mountPath: /usr/share/filebeat/filebeat.yml
    subPath: filebeat.yml
    readOnly: true
    volumes:
• name: config
  configMap:
  name: filebeat-config
  ```

```sh
kubectl apply -f filebeat-wi-pod.yaml
```

8. Verify

Check the pod is running and the workload identity webhook injected the environment variables:

```sh

Check pod status

kubectl get pod filebeat-wi-test

Verify the webhook injected the env vars

kubectl describe pod filebeat-wi-test | grep -A3 "AZURE_"

You should see:

AZURE_CLIENT_ID:

AZURE_TENANT_ID:

AZURE_FEDERATED_TOKEN_FILE: /var/run/secrets/azure/tokens/azure-identity-token

Check Filebeat logs

kubectl logs filebeat-wi-test -f
```

Filebeat should start normally, spawning partition consumers every 10s and picking up messages from the event hub.

Cleanup

```sh

Delete the test pod

kubectl delete pod filebeat-wi-test
kubectl delete configmap filebeat-config
kubectl delete serviceaccount filebeat-wi

Delete Azure resources

az group delete --name filebeat-wi-test --yes --no-wait
```