Create and manage persistent volumes (PVs) with Azure Blob storage in Azure Kubernetes Service (AKS)

If multiple pods need concurrent access to the same storage volume, you can use Azure Blob storage to connect using blobfuse or Network File System (NFS).

This article shows you how to dynamically and statically create Azure Blob storage containers for use by multiple pods in an Azure Kubernetes Service (AKS) cluster.

Prerequisites

• The Azure Blob storage CSI driver enabled on your AKS cluster.

• To support an Azure DataLake Gen2 storage account when using blobfuse mount, you need to do the following:

  • To create an ADLS account using the driver in dynamic provisioning, specify isHnsEnabled: "true" in the storage class parameters.
  • To enable blobfuse access to an ADLS account in static provisioning, specify the mount option --use-adls=true in the persistent volume.
  • If you're going to enable a storage account with Hierarchical Namespace, existing persistent volumes should be remounted with --use-adls=true mount option.

• By default, the blobfuse cache is located in the /mnt directory. If the VM SKU provides a temporary disk, the /mnt directory is mounted on the temporary disk. However, if the VM SKU doesn't provide a temporary disk, the /mnt directory is mounted on the OS disk, you can set --tmp-path= mount option to specify a different cache directory.

Use built-in storage classes to create dynamic PVs with Azure Blob storage

A storage class is used to define how an Azure Blob storage container is created. A storage account is automatically created in the node resource group for use with the storage class to hold the Azure Blob storage container. When you use storage CSI drivers on AKS, there are two extra built-in StorageClasses that use the Azure Blob storage CSI driver.

The reclaim policy on both storage classes ensures that the underlying Azure Blob storage is deleted when the respective PV is deleted. The storage classes also configure the container to be expandable by default, as the set allowVolumeExpansion parameter is set to true.

You can select one of the following Azure storage redundancy SKUs for the skuname parameter in the storage class definition:

• Standard_LRS: Standard locally redundant storage
• Premium_LRS: Premium locally redundant storage
• Standard_ZRS: Standard zone redundant storage
• Premium_ZRS: Premium zone redundant storage
• Standard_GRS: Standard geo-redundant storage
• Standard_RAGRS: Standard read-access geo-redundant storage

Create custom storage classes for dynamic PVs with Azure Blob storage

The default storage classes are suitable for most scenarios. In some cases, you might want to have your own storage class customized with your own parameters. In this section, we provide two examples: one using NFS protocol and one using blobfuse.

Custom storage class example using NFS protocol

The manifest in this example mounts a blob storage container using the NFS protocol. You can use it to add the tags parameter.

1. Create a file named blob-nfs-sc.yaml and paste in the following example manifest:

   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: azureblob-nfs-premium
   provisioner: blob.csi.azure.com
   parameters:
     protocol: nfs
     tags: environment=Development
   volumeBindingMode: Immediate
   allowVolumeExpansion: true
   mountOptions:
     - nconnect=4
   

2. Create the storage class using the kubectl apply command:

   kubectl apply -f blob-nfs-sc.yaml
   

   Your output should resemble the following example output:

   storageclass.storage.k8s.io/blob-nfs-premium created
   

Custom storage class example using blobfuse

The manifest in this example uses blobfuse and mounts a Blob storage container. You can use it to update the skuName parameter.

1. Create a file named blobfuse-sc.yaml and paste in the following example manifest:

   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: azureblob-fuse-premium
   provisioner: blob.csi.azure.com
   parameters:
     skuName: Standard_GRS  # available values: Standard_LRS, Premium_LRS, Standard_GRS, Standard_RAGRS
   reclaimPolicy: Delete
   volumeBindingMode: Immediate
   allowVolumeExpansion: true
   mountOptions:
     - -o allow_other
     - --file-cache-timeout-in-seconds=120
     - --use-attr-cache=true
     - --cancel-list-on-mount-seconds=10  # prevent billing charges on mounting
     - -o attr_timeout=120
     - -o entry_timeout=120
     - -o negative_timeout=120
     - --log-level=LOG_WARNING  # LOG_WARNING, LOG_INFO, LOG_DEBUG
     - --cache-size-mb=1000  # Default will be 80% of available memory, eviction will happen beyond that.
   

2. Create the storage class using the kubectl apply command:

   kubectl apply -f blobfuse-sc.yaml
   

   Your output should resemble the following example output:

   storageclass.storage.k8s.io/blob-fuse-premium created
   

Storage class parameters for dynamic PVs with Azure Blob storage

The following table includes parameters you can use to define a custom storage class for your persistent volume claims (PVCs) with Azure Blob storage:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: blob-fuse
provisioner: blob.csi.azure.com
parameters:
  skuName: Premium_LRS  # available values: Standard_LRS, Premium_LRS, Standard_GRS, Standard_RAGRS, Standard_ZRS, Premium_ZRS
  protocol: fuse2
  networkEndpointType: privateEndpoint
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
mountOptions:
  - -o allow_other
  - --file-cache-timeout-in-seconds=120
  - --use-attr-cache=true
  - --cancel-list-on-mount-seconds=10  # prevent billing charges on mounting
  - -o attr_timeout=120
  - -o entry_timeout=120
  - -o negative_timeout=120
  - --log-level=LOG_WARNING  # LOG_WARNING, LOG_INFO, LOG_DEBUG
  - --cache-size-mb=1000  # Default will be 80% of available memory, eviction will happen beyond that.

Create a PVC with Azure Blob storage

A PVC uses the storage class object to dynamically provision an Azure Blob storage. You can use the example YAML manifest in this section to create a PVC that's 5 GB in size with ReadWriteMany access. For more information on access modes, see Kubernetes PV access modes.

1. Create a file named blob-nfs-pvc.yaml and paste in the following YAML manifest:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: azure-blob-storage
   spec:
     accessModes:
     - ReadWriteMany
     storageClassName: azureblob-nfs-premium
     resources:
       requests:
         storage: 5Gi
   

2. Create the PVC using the kubectl create command:

   kubectl create -f blob-nfs-pvc.yaml
   

3. View the status of the PVC with the [kubectl get pvc][kubectl-get-pvc] command:

   kubectl get pvc azure-blob-storage
   

   Your output should resemble the following example output, which shows that the PVC is in a Bound state:

   NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                AGE
   azure-blob-storage   Bound    pvc-aaaaaaaa-0000-1111-2222-bbbbbbbbbbbb   5Gi        RWX            azureblob-nfs-premium       92m
   

Use a PVC in a pod to mount Azure Blob storage

The following YAML creates a pod that uses the persistent volume claim azure-blob-storage to mount the Azure Blob storage at the `/mnt/blob' path.

1. Create a file named blob-nfs-pv and paste in the following YAML manifest. Make sure the claimName matches the PVC created in the previous step.

   kind: Pod
   apiVersion: v1
   metadata:
     name: mypod
   spec:
     containers:
     - name: mypod
       image: mcr.microsoft.com/oss/nginx/nginx:1.17.3-alpine
       resources:
         requests:
           cpu: 100m
           memory: 128Mi
         limits:
           cpu: 250m
           memory: 256Mi
       volumeMounts:
       - mountPath: "/mnt/blob"
         name: volume
         readOnly: false
     volumes:
       - name: volume
         persistentVolumeClaim:
           claimName: azure-blob-storage
   

2. Create the pod using the kubectl apply command:

   kubectl apply -f blob-nfs-pv.yaml
   

3. Once the pod is successfully running, create a new file named test.txt using the following command:

   kubectl exec mypod -- touch /mnt/blob/test.txt
   

4. Validate the disk is correctly mounted using the following command to list the files in the mounted directory:

   kubectl exec mypod -- ls /mnt/blob
   

   Your output should resemble the following example output, which shows the test.txt file you created in the mounted Azure Blob storage:

   test.txt
   

Use a StatefulSet to manage the lifecycle of a volume with Azure Blob storage

To have a storage volume persist for your workload, you can use a StatefulSet. This makes it easier to match existing volumes to new pods that replace any that have failed. The following examples demonstrate how to set up a StatefulSet for Blob storage using the NFS protocol or Blobfuse.

Create a static PV with Azure Blob storage

The following sections provide instructions for creating a static PV with Azure Blob storage. A static PV is a persistent volume that an administrator creates manually. This PV is available for use by pods in the cluster. To use a static PV, you create a PVC that references the PV, and then create a pod that references the PVC.

Storage class parameters for static PVs with Azure Blob storage

The following table includes parameters you can use to define a custom storage class for your static PVCs with Azure Blob storage:

Create a Blob storage container

When you create an Azure Blob storage resource for use with AKS, you can create the resource in the node resource group. This approach allows the AKS cluster to access and manage the blob storage resource.

1. Get the node resource group name of your AKS cluster using the az aks show command with the --query nodeResourceGroup parameter.

   az aks show --resource-group myResourceGroup --name myAKSCluster --query nodeResourceGroup -o tsv
   

   The output of the command resembles the following example:

   MC_myResourceGroup_myAKSCluster_eastus
   

2. Create a container for storing blobs following the steps in the Manage blob storage to authorize access and then create the container.

Mount volume

In this section, you mount the persistent volume using the NFS protocol or Blobfuse.

Use the persistent volume

The following YAML creates a pod that uses the PV or PVC named pvc-blob created earlier to mount the Azure Blob storage at the /mnt/blob path.

1. Create a file named nginx-pod-blob.yaml and paste in the following YAML manifest. Make sure the claimName matches the PVC created in the previous step when creating a PV for NFS or Blobfuse.

   kind: Pod
   apiVersion: v1
   metadata:
     name: nginx-blob
   spec:
     nodeSelector:
       "kubernetes.io/os": linux
     containers:
       - image: mcr.microsoft.com/oss/nginx/nginx:1.17.3-alpine
         name: nginx-blob
         volumeMounts:
           - name: blob01
             mountPath: "/mnt/blob"
             readOnly: false
     volumes:
       - name: blob01
         persistentVolumeClaim:
           claimName: pvc-blob
   

2. Create the pod and mount the PVC using the kubectl create command:

   kubectl create -f nginx-pod-blob.yaml
   

3. Create an interactive shell session with the pod to verify the Blob storage is mounted correctly using the following kubectl exec command:

   kubectl exec -it nginx-blob -- df -h
   

   Your output should resemble the following example output, which shows the Blob storage is mounted at the /mnt/blob path:

   Filesystem      Size  Used Avail Use% Mounted on
   ...
   blobfuse         14G   41M   13G   1% /mnt/blob
   ...
   

Related content

• Use Azure tags in Azure Kubernetes Service (AKS)