Distrust and caution are the parents of security.
Benjamin Franklin
Azure security can sometimes be really complex to put your head around. Good understanding of fundamentals and deep desire to troubleshoot issues can come to one’s rescue.
Though I am working on Azure Cloud for few years now but the pace at which it is evolving can be mind-boggling at times, especially in the security space.
Recently I came across a very interesting use case while working with AKS. We were using a third party data platform product and to enhance certain functionality we have to mount Azure fileshare over workloads running in AKS cluster.
So came up with a design and blow is a rough design of the below, details have been stripped due to privacy reasons. Please ignore my drawing skills for time being.
As you can see we need to have Azure file share mounted on the AKS pods running this proprietary software.
On paper this seems to be straightforward approach and implementation but still we face few hiccups mostly due to our lack of understanding of certain fine-grained details of how this proprietary software works and partly due to Azure services and their in-built mechanisms.
The problem became even interesting considering that we can’t use the storage account access keys due to compliance reasons.
We decided to use workload identity.
Pod identity is not longer supported by Azure.
For starters, basically it is a simpler, cleaner and safer way to provide authentication and access to your workloads (in our case pods) using mechanisms like managed identities.
For details please read here.
Personally I liked below flow from Azure documentation to get a higher level understanding:
For even detailed look how azure ad token is provided, please look at below flow:
Initially we were tempted to use managed identity (user assigned managed identity, to say precisely) for authentication but by default, it increases the blast radius for security by providing similar access to all pods. Perhaps, this is not the best dig at managing security. Azure RBAC to be utilised for authorisation needs (access to storage account file share).
What roles fits for you depends on what you want to do with file share and storage account.
How we approached the problem?
Step 1: Create necessary environment variables to make running azure cli commands easier.
export RESOURCE_GROUP=”myResourceGroup”
export LOCATION=”westcentralus”
export SERVICE_ACCOUNT_NAMESPACE=”default”
export SERVICE_ACCOUNT_NAME=”workload-identity-sa”
export SUBSCRIPTION=”$(az account show — query id — output tsv)”
export USER_ASSIGNED_IDENTITY_NAME=”myIdentity”
export FEDERATED_IDENTITY_CREDENTIAL_NAME=”myFedIdentity”
Step 2: Enabled workload identity for the cluster, in our case fortunately it was already enabled hence we skipped this step.
az aks update -g “${RESOURCE_GROUP}” -n myAKSCluster --enable-oidc-issuer --enable-workload-identity
Step 3: retrieve the OIDC issuer URL
export AKS_OIDC_ISSUER=”$(az aks show -n myAKSCluster -g “${RESOURCE_GROUP}” --query “oidcIssuerProfile.issuerUrl” -o tsv)”
Step 4: Create a managed identity for this purpose.
az identity create --name “${USER_ASSIGNED_IDENTITY_NAME}” --resource-group “${RESOURCE_GROUP}” --location “${LOCATION}” --subscription “${SUBSCRIPTION}”
Let’s use that also in a env variable to use it for referencing easier.
export USER_ASSIGNED_CLIENT_ID=”$(az identity show --resource-group “${RESOURCE_GROUP}” --name “${USER_ASSIGNED_IDENTITY_NAME}” --query ‘clientId’ -o tsv)”
Step 5: Provide a RBAC role for this UAMI, usually in organisations a separate teams controls access controls like this. So raise request for this access assignment.
The permissions needed in this case was Storage Account SMB FileShare Data Reader and Storage Key Operator Service role.
The second set of permissions was interesting, though we were not using access keys directly. Why it is needed? read here.
Step 6: Now connect to the cluster and create a service account for the namespace running our workload.
az aks get-credentials -n myAKSCluster -g “${RESOURCE_GROUP}”
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
azure.workload.identity/client-id: “${USER_ASSIGNED_CLIENT_ID}”
name: “${SERVICE_ACCOUNT_NAME}”
namespace: “${SERVICE_ACCOUNT_NAMESPACE}”
EOF
Interesting facts about service account and worklad idenitty mapping:
Azure AD Workload Identity supports the following mappings:
- one-to-one (a service account referencing an AAD object)
- many-to-one (multiple service accounts referencing the same AAD object).
- one-to-many (a service account referencing multiple AAD objects by changing the client ID annotation).
Step 7: Create federated credentials mapped to UAMI client ID and service account we created in step 6.
az identity federated-credential create --name ${FEDERATED_IDENTITY_CREDENTIAL_NAME} --identity-name “${USER_ASSIGNED_IDENTITY_NAME}” --resource-group “${RESOURCE_GROUP}” --issuer “${AKS_OIDC_ISSUER}” --subject system:serviceaccount:”${SERVICE_ACCOUNT_NAMESPACE}”:”${SERVICE_ACCOUNT_NAME}” --audience api://AzureADTokenExchange
Step 8: Create a storage class, though not needed and can be skipped. Then create a PV with specification yaml similar to below.
cat <<EOF | kubectl apply -f -
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: azurefile-csi
provisioner: file.csi.azure.com # replace with “kubernetes.io/azure-file” if aks version is less than 1.21
allowVolumeExpansion: true
mountOptions:
— dir_mode=0777
— file_mode=0777
— uid=0
— gid=0
— mfsymlinks
— cache=strict
— actimeo=30
parameters:
skuName: Premium_LRS
EOF
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
pv.kubernetes.io/provisioned-by: file.csi.azure.com
name: azurefile
spec:
capacity:
storage: 5Gi
accessModes:
— ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: azurefile-csi
csi:
driver: file.csi.azure.com
volumeHandle: unique-volumeid # make sure this volumeid is unique for every identical share in the cluster
volumeAttributes:
resourceGroup: "${RESOURCE_GROUP}" # optional, only set this when storage account is not in the same resource group as node
shareName: aksshare
mountOptions:
— dir_mode=0777
— file_mode=0777
— uid=0
— gid=0
— mfsymlinks
— cache=strict
— nosharesock
— nobrl
EOF
Step 9: Create a PVC which gets bound to this PVC.
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: azurefile
namespace: ${SERVICE_ACCOUNT_NAMESPACE}
spec:
accessModes:
— ReadWriteMany
storageClassName: azurefile-csi
volumeName: azurefile
resources:
requests:
storage: 5Gi
EOF
Step 10: Annotate the pod and associate the PVC with the pod.
In our case there are some product config we need to use to use hooks provided by the vendor for additional volumes and volume mounts. Then create the pod/deployment, in our case, this was through some helm charts. You can use something similar to below:
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: your-pod
namespace: “${SERVICE_ACCOUNT_NAMESPACE}”
labels:
azure.workload.identity/use: “true” # Required, only the pods with this label can use workload identity
spec:
serviceAccountName: “${SERVICE_ACCOUNT_NAME}”
containers:
— image: <your image>
name: <containerName>
EOF
Describe the pod to check for identity file mounted to check for authentication mechanism.
And this it…
TBH, it was not that simple for us and took multiple iteration and multiple days due to regulated environment we work in.
One key point though is Microsoft Entra tokens expire in 24 hours after they’re issued, though Kubernetes service account token expiry isn’t correlated with Microsoft Entra tokens.
If you are using pod identity (deprecated) and want to migrate to workload identity there are couple of ways, please check how to migrate section here.
In my personal vanilla sandbox, I was able to run this E2E with 7–8 hours of dedicated effort including setting up cluster etc.
For more such content, and my other blogs please check here and you can connect with me on LinkedIn.
Resources
1. https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview
2. https://azure.github.io/azure-workload-identity/docs/concepts.html
3. https://github.com/kubernetes-sigs/cloud-provider-azure/pull/4996
Credits
Azure documentation and experience 🙂
Disclaimer
The above blog is for educational purpose please avoid using it on prod/live systems without proper design, implementation and testing in lower environments. There are few intricacies, which cant be covered in a small blog like this and sometimes outside the scope of discussion.
This blog is part of Microsoft Azure Week! Find more similar blogs on our Microsoft Azure Landing page here.
About the author:
Neeraj Sharma is a Cloud & DevOps Consultant with extensive experience working with fintech clients at various tech companies, including Amazon, Oracle, and KPMG. He is highly skilled in cloud-native technologies, DevOps/SecOps, and infrastructure management. Neeraj holds multiple technology certifications and loves sharing his knowledge with the wider community through blogs, speaking and community events.
Sharma, N. (2024) Decoding Azure security with Azure File share mount on AKS workloads. Available at: Decoding Azure security with Azure File share mount on AKS workloads | by Neeraj Sharma | Medium [Accessed on 24/06/2024]
