Reliability in Azure Kubernetes Service (AKS)

This article describes reliability support in Azure Kubernetes Service (AKS), covering intra-regional resiliency via availability zones and multi-region deployments.

Resiliency is a shared responsibility between you and Microsoft. This article covers ways for you to create a resilient solution that meets your needs.

Production deployment recommendations

For recommendations about how to deploy reliable production workloads in AKS, see the following articles:

• Deployment and cluster reliability best practices for AKS
• High availability (HA) and disaster recovery (DR) overview for AKS
• Zone resiliency considerations for AKS

Reliability architecture overview

When you create an AKS cluster, the Azure platform automatically creates and configures:

• A control plane that has the API server, etcd, the scheduler, and other pods that are required to manage your workload.

• A system node pool to your subscription that hosts your add-ons and other pods that run in the kube-system namespace.

After this initial node pool setup is complete, you can add or delete node pools for your own user workloads. AKS doesn't manage node pools for reliability, and you must ensure that your workloads are resilient to infrastructure failures.

Resiliency is a shared responsibility between you and Microsoft. As a compute service, AKS manages some aspects of your cluster's reliability, but you're responsible for managing other aspects.

• Microsoft manages the control plane and other managed components of AKS.

• It's your responsibility to:

  • Define how components, including node pools and load balancers that attach to services, should be configured to meet your reliability requirements. After you define the components, Microsoft then deploys and manages them on your behalf.

  • Manage any components outside of the AKS cluster, including storage and databases. Verify that these components meet your reliability requirements. When you deploy your workloads, ensure that other Azure components are also configured for resiliency by following the best practices for those services.

Transient faults

Transient faults are short, intermittent failures in components. They occur frequently in a distributed environment like the cloud, and they're a normal part of operations. They correct themselves after a short period of time. It's important that your applications handle transient faults, usually by retrying affected requests.

All cloud-hosted applications should follow the Azure transient fault handling guidance when communicating with any cloud-hosted APIs, databases, and other components. For more information, see Recommendations for handing transient faults.

When you use AKS, transient faults can occur because of various reasons, including application crashes, pod scaling and balancing operations, node patching, and temporary infrastructure failures such as hardware or networking problems.

It's impossible to eliminate all transient faults, so clients that access your AKS-hosted applications should be prepared to retry failed requests and follow other transient fault handling recommendations. You can minimize the likelihood of transient faults and avoid or mitigate the downtime they might cause by following Kubernetes and Azure best practices in your deployment.

• Set pod disruption budgets (PDBs) in your pod YAML to specify how many pods you need to have in a Ready state at a given time. When you set PDBs, AKS ensures a minimum availability of replicas when it runs operations to cordon and drain the nodes. If a PDB can't be satisfied during processes like upgrades, the pod continues to function and the operation might fail. For more information, see PDBs.

• Use maxUnavailable to define the maximum number of replicas that can become unavailable at a given time. For example, when you perform a rolling restart, AKS ensures that no more than the maxUnavailable number of pods are churned at a given time. For more information, see maxUnavailable.

• Follow deployment best practices. Pod replicas can also fail because of application problems. For more information, see Deployment-level best practices for AKS cluster reliability.

Availability zone support

Availability zones are physically separate groups of datacenters within each Azure region. When one zone fails, services can fail over to one of the remaining zones.

For more information on availability zones in Azure, see What are availability zones?

When you deploy an AKS cluster into a region that supports availability zones, different components require different types of configuration.

The AKS control plane is zone resilient by default. If a zone fails, the control plane doesn't require any configuration or management to achieve resiliency. However, control plane resiliency isn't sufficient for your cluster to remain operational when a zone fails. For the system node pool and any user node pools that you deploy, you must enable availability zone support to help ensure that your workloads are resilient to availability zone failures.

Region support

You can deploy zone-resilient AKS clusters into any region that supports availability zones.

Considerations

To enhance the reliability and resiliency of AKS production workloads in a region, you need to configure AKS for zone redundancy by making the following configurations:

• Deploy multiple replicas. Kubernetes spreads your pods across nodes based on node labels. To spread your workload across zones, you need to deploy multiple replicas of your pod. For instance, if you configure the node pool with three zones but only deploy a single replica of your pod, your deployment isn't zone resilient.

• Enable automatic scaling. Kubernetes node pools provide manual and automatic scaling options. By using manual scaling, you can add or delete nodes as needed, and pending pods wait until you scale up a node pool. AKS-managed scaling uses the cluster autoscaler or node autoprovisioning (NAP). AKS scales the node pool up or down based on the pod's requirements within your subscription's SKU quota and capacity. This method helps ensure that your pods are scheduled on available nodes in the availability zones.

• Set pod topology constraints. Use pod topology spread constraints to control how pods are spread across different nodes or zones. Constraints help you achieve HA, resiliency, and efficient resource usage. If you prefer to spread pods strictly across zones, you can set constraints to force a pod into a pending state to maintain the balance of pods across zones. For more information, see Pod topology spread constraints.

• Configure zone-resilient networking. If your pods serve external traffic, configure your cluster network architecture by using services like Azure Application Gateway, Azure Load Balancer, or Azure Front Door.

• Ensure that dependencies are zone resilient. Most AKS applications use other services for storage, security, or networking. Make sure that you review the zone resiliency recommendations for those services.

Cost

There's no extra charge to enable availability zone support in AKS. You pay for the virtual machines (VMs) and other resources that you deploy in the availability zones.

Configure availability zone support

• Create a new AKS cluster that has availability zone support: To configure availability zone support, see Create an Azure Kubernetes Service (AKS) cluster that uses availability zones.
• Migration: You can't enable availability zone support after you create a cluster. Instead, you need to create a new cluster that has availability zone support enabled and delete the existing cluster.
• Disable availability zone support: You can't disable availability zone support after you create a cluster. Instead, you need to create a new cluster that has availability zone support disabled and delete the existing cluster.

Normal operations

• Traffic routing between zones: When you deploy an AKS cluster that uses availability zones, it's important to ensure that your networking components are also zone resilient. Depending on the load balancers and other networking components that you use, you might need to explicitly configure components to route traffic to the correct nodes in the correct zones and to respond to zone outages. For more information, see Zone resiliency considerations for AKS.

• Data replication between zones: If you run a stateless workload, you should use managed Azure services, such Azure databases, Azure Cache for Redis, or Azure Storage to store the application data. You can use these services to help ensure that your traffic can be moved across nodes and zones without risking data loss or affecting the user experience. You can use Kubernetes deployments, services, and health probes to manage stateless pods and ensure even distribution across zones.

  If you need to store state within your cluster by using Azure disks, use Azure zone-redundant storage to help ensure that your data is replicated across multiple availability zones. For more information, see Choose the right disk type based on application needs.

Zone-down experience

• Detection and response: When a zone outage occurs, the control plane automatically fails over. If your node pools use availability zones and follow zone resiliency best practices, you can expect AKS to bring up nodes and replicas in the zones that are operational. AKS does this task automatically when you use managed solutions like cluster autoscaler or NAP. Without autoscaling, nodes and replicas remain in the Pending state and wait for manual intervention to scale up the node pool.

  AKS also attempts to rebalance the pods across the healthy zones. If you choose to manually scale your node pool in a zone-down scenario, your pods might remain in the Pending state when there are no nodes available in the healthy zones. Scaling out in the remaining zones is also subject to the availability of quota and capacity for the VM SKU that you use.

• Notification: AKS doesn't notify you when a zone is down. You can use your node or pod health metrics to monitor the health of your nodes and pods.

• Active requests: Any active requests might experience disruptions. Some requests can fail, and latency might increase while your workload fails over to another zone.

• Expected data loss: If you store state within your cluster by using Azure disks, and you use zone-redundant storage, then a zone failure isn't expected to cause any data loss.

• Expected downtime: If you correctly configure zone resiliency for your cluster and pods, then a zone failure isn't expected to cause downtime for your AKS workload.

• Traffic rerouting: Load balancers reroute new incoming requests to pods that run on healthy nodes.

For more information, see Zone resiliency considerations for AKS.

Failback

When the availability zone recovers, failback behavior depends on the component:

• Control plane: AKS automatically restores control plane operations across all availability zones. No manual intervention is required.

• Node pools and nodes: Immediately after failback, nodes remain in the previously healthy zones and aren't restored in the recovered zone. However, the next time you perform a node-scaling operation, such as when you scale out your node pool, the node pool can create nodes in the recovered zone.

• Pods: Immediately after failback, pods continue to run on the nodes that they currently run on. When new pods are created or existing pods are re-created, they're eligible to use nodes in the recovered zone.

• Storage: Any storage attached to pods recovers based on how zone-redundant storage works.

Testing for zone failures

You can test your resiliency to availability zone failures by using the following methods:

• Cordon and drain nodes in a single availability zone
• Simulate an availability zone failure by using Azure Chaos Studio

Multi-region support

AKS clusters are single-region resources. If the region is unavailable, your AKS cluster is also unavailable.

Alternative multi-region approaches

If you need to deploy your Kubernetes workload to multiple Azure regions, you have two options to manage the orchestration of these clusters.

• Use Azure Kubernetes Fleet Manager if you want a simpler, managed experience. By using Azure Kubernetes Fleet Manager, you can:

  • Manage a set of AKS clusters as a single unit, and those clusters can be distributed across multiple Azure regions.

  • Automate specific aspects of cluster management, such as cluster and node image upgrades.

  • Use traffic distribution capabilities to spread traffic across the clusters and automatically fail over if a region is unavailable.

• Orchestrate failover by using a manual active-active or active-passive deployment model if your workload requires more nuanced control over the different components of interregional failovers. For more information, see HA and DR overview for AKS.

Backups

Azure Backup has an extension that you can use to back up AKS cluster resources and persistent volumes that attach to the cluster. The Backup vault communicates with the AKS cluster through the extension to perform backup and restore operations.

If your AKS cluster is in a paired region, you can configure backups to be stored in geo-redundant storage. You can restore geo-redundant backups into the paired region.

For more information, see the following articles:

• What is Azure Kubernetes Service backup?
• Back up AKS by using Azure Backup

For most solutions, you shouldn't rely exclusively on backups. Instead, use the other capabilities described in this guide to support your resiliency requirements. However, backups protect against some risks that other approaches don't. For more information, see Redundancy, replication, and backup.

Strive to use stateless clusters that minimize the need for backup. Store data in external storage systems and databases instead of within your cluster.

Service-level agreement

The service-level agreement (SLA) for AKS describes the expected availability of the service and the conditions that must be met to achieve that availability expectation. For more information, see SLAs for online services.

AKS provides three pricing tiers for cluster management: Free, Standard, and Premium. The Free tier enables you to use AKS to test your workloads. The Standard and Premium tiers are designed for production workloads. When you deploy an AKS cluster that has availability zones enabled, the uptime percentage defined in the SLA increases. However, the SLA applies only if you deploy a cluster in the Standard or Premium pricing tier.

Related content

• Reliability in Azure