Best practices for cost optimization in Azure Kubernetes Service (AKS)

Cost optimization is about maximizing the value of resources while minimizing unnecessary expenses within your cloud environment. This process involves identifying cost-effective configuration options and implementing best practices to improve operational efficiency. An AKS environment can be optimized to minimize cost while taking into account performance and reliability requirements.

In this article, you learn about:

Start with AKS Automatic for built-in cost optimization

AKS Automatic is a cluster mode that preconfigures many of the cost optimization practices described in this article. If you're creating a new cluster, consider AKS Automatic to reduce engineering effort, configuration risk, and the ongoing operational overhead that leads to unnecessary cloud spend.

AKS Automatic provides the following cost optimization capabilities by default, without any extra configuration:

For workloads that require AKS Standard, the rest of this article describes each of these practices and how to configure them manually. Where AKS Automatic provides a practice by default, a note indicates that no extra steps are needed.

Embrace FinOps to build a cost saving culture

Financial operations (FinOps) is a discipline that combines financial accountability with cloud management and optimization. It focuses on driving alignment between finance, operations, and engineering teams to understand and control cloud costs. The FinOps foundation has several notable projects, such as the FinOps Framework and the FOCUS Specification.

For more information, see What is FinOps?

Select cost-efficient infrastructure and cluster configuration

Evaluate SKU family

It's important to evaluate the resource requirements of your application before deployment. Small development workloads have different infrastructure needs than large production-ready workloads. While a combination of CPU, memory, and networking capacity configurations heavily influences the cost effectiveness of a SKU, consider the following virtual machine (VM) types:

Review storage options

For more information on storage options and related cost considerations, see the following articles:

• Best practices for storage and backups in Azure Kubernetes Service (AKS)
• Storage options for applications in Azure Kubernetes Service (AKS)

Use cluster preset configurations

It can be difficult to pick the right VM SKU, regions, number of nodes, and other configuration options. Cluster preset configurations in the Azure portal offload this initial challenge by providing recommended configurations for different application environments that are cost-conscious and performant. The Dev/Test preset is best for developing new workloads or testing existing workloads. The Production Economy preset is best for serving production traffic in a cost-conscious way if your workloads can tolerate interruptions. Noncritical features are off by default, and you can modify the preset values at any time.

For a more comprehensive approach that goes beyond static presets, consider AKS Automatic. AKS Automatic fully manages node pools using node auto-provisioning (NAP), which continuously right-sizes infrastructure to your actual workload demands. It also enables all workload autoscalers by default and enforces resource governance through deployment safeguards, which are extra benefits that cluster presets don't provide.

Consider multitenancy

AKS offers flexibility in how you run multitenant clusters and isolate resources. For friendly multitenancy, you can share clusters and infrastructure across teams and business units through logical isolation. Kubernetes Namespaces form the logical isolation boundary for workloads and resources. Sharing infrastructure reduces cluster management overhead while also improving resource utilization and pod density within the cluster. To learn more about multitenancy on AKS and to determine if it's right for your organizational needs, see AKS considerations for multitenancy and Design clusters for multitenancy.

Reduce resource waste through application and cluster configuration

Make your container as lean as possible

A lean container refers to optimizing the size and resource footprint of the containerized application. Check that your base image is minimal and only contains the necessary dependencies. Remove any unnecessary libraries and packages. A smaller container image accelerates deployment times and increases the efficiency of scaling operations. Artifact Streaming on AKS lets you stream container images from Azure Container Registry (ACR). It pulls only the necessary layer for initial pod startup, reducing the pull time for larger images from minutes to seconds.

Enforce resource quotas

Resource quotas provide a way to reserve and limit resources across a development team or project. Quotas are defined on a namespace and can be set on compute resources, storage resources, and object counts. When you define resource quotas, individual namespaces are prevented from consuming more resources than allocated. Resource quotas are useful for multitenant clusters where teams are sharing infrastructure.

Use cluster start/stop

When left unattended, small development and test clusters can accrue unnecessary costs. You can turn off clusters that don't need to run at all times using the cluster start and stop feature. This feature shuts down all system and user node pools so you don't pay for extra compute. The state of your cluster and objects is maintained when you start the cluster again.

Use capacity reservations

Capacity reservations let you reserve compute capacity in an Azure region or availability zone for any duration of time. Reserved capacity is available for immediate use until the reservation is deleted. Associating an existing capacity reservation group to a node pool guarantees allocated capacity for your node pool and helps you avoid potential on-demand pricing spikes during periods of high compute demand.

Monitor your environment and spend

Increase visibility with Microsoft Cost Management

Microsoft Cost Management offers a broad set of capabilities to help with cloud budgeting, forecasting, and visibility for costs both inside and outside of the cluster. Proper visibility is essential for deciphering spending trends, identifying optimization opportunities, and increasing accountability among application developers and platform teams. Enable the AKS Cost Analysis add-on for granular cluster cost breakdown by Kubernetes constructs along with Azure Compute, Network, and Storage categories.

Azure Monitor

If you're ingesting metric data via Container insights, we recommend migrating to managed Prometheus, which offers a significant cost reduction. You can disable Container insights metrics using the data collection rule (DCR) and deploy the managed Prometheus add-on, which supports configuration via Azure Resource Manager, Azure CLI, Azure portal, and Terraform.

For more information, see Azure Monitor best practices and managing costs for Container insights.

Log Analytics

For control plane logs, consider disabling the categories you don't need and/or using the Basic Logs API when applicable to reduce Log Analytics costs. For more information, see Azure Kubernetes Service (AKS) control plane/resource logs. For data plane logs, or application logs, consider adjusting the cost optimization settings.

You can also use Transformations in Azure Monitor to filter or modify control plane and data plane logs before they are sent to a Log Analytics workspace. For more information on how to create a transformation see Create a transformation in Azure Monitor.

Azure Advisor cost recommendations

AKS cost recommendations in Azure Advisor provide recommendations to help you achieve cost-efficiency without sacrificing reliability. Advisor analyzes your resource configurations and recommends optimization solutions. For more information, see Get Azure Kubernetes Service (AKS) cost recommendations in Azure Advisor.

Optimize workloads through autoscaling

Establish a baseline

Before configuring your autoscaling settings, you can use Azure Load Testing to establish a baseline for your application. Load testing helps you understand how your application behaves under different traffic conditions and identify performance bottlenecks. Once you have a baseline, you can configure autoscaling settings to ensure your application can handle the expected load.

Enable application autoscaling

Vertical pod autoscaling

Requests and limits that are higher than actual usage can result in overprovisioned workloads and wasted resources. In contrast, requests and limits that are too low can result in throttling and workload issues due to lack of memory. The Vertical Pod Autoscaler (VPA) allows you to fine-tune CPU and memory resources required by your pods. VPA provides recommended values for CPU and memory requests and limits based on historical container usage, which you can set manually or update automatically. Best for applications with fluctuating resource demands. VPA’s recommendation-only off mode allows teams to review resource suggestions without enforcing them automatically. This mode can be enabled during testing, and VPA recommendations can be used to set the CPU and memory request and limits for production environments.

Horizontal pod autoscaling

The Horizontal Pod Autoscaler (HPA) dynamically scales the number of pod replicas based on observed metrics, such as CPU or memory utilization. During periods of high demand, HPA scales out, adding more pod replicas to distribute the workload. During periods of low demand, HPA scales in, reducing the number of replicas to conserve resources. Best for applications with predictable resource demands.

Kubernetes event-driven autoscaling

The Kubernetes Event-driven Autoscaler (KEDA) add-on provides extra flexibility to scale based on various event-driven metrics that align with your application behavior. For example, for a web application, KEDA can monitor incoming HTTP request traffic and adjust the number of pod replicas to ensure the application remains responsive. For processing jobs, KEDA can scale the application based on message queue length. Managed support is provided for all Azure Scalers. KEDA also allows you to scale down to 0 replicas, especially helpful for sporadic event-driven workloads, periodic machine learning (ML) or GPU workloads, and dev/test or low traffic environments.

Enable infrastructure autoscaling

Cluster autoscaling

To keep up with application demand, the Cluster Autoscaler watches for pods that can't be scheduled due to resource constraints and scales the number of nodes in the node pool accordingly. When nodes don't have running pods, the Cluster Autoscaler scales down the number of nodes. The Cluster Autoscaler profile settings apply to all autoscaler-enabled node pools in a cluster. For more information, see Cluster Autoscaler best practices and considerations.

Node auto-provisioning

Complicated workloads might require several node pools with different VM size configurations to accommodate CPU and memory requirements. Accurately selecting and managing several node pool configurations adds complexity and operational overhead. Node auto-provisioning (NAP) simplifies the SKU selection process and decides the optimal VM configuration based on pending pod resource requirements to run workloads in the most efficient and cost effective manner.

Save with Azure discounts

Azure Reservations

If your workload is predictable and exists for an extended period of time, consider purchasing an Azure Reservation to further reduce your resource costs. Azure Reservations operate on a one-year or three-year term, offering up to 72% discount as compared to pay-as-you-go prices for compute. Reservations automatically apply to matching resources. Best for workloads that are committed to running in the same SKUs and regions over an extended period of time.

Azure Savings Plan

If you have consistent spend, but your use of disparate resources across SKUs and regions makes Azure Reservations infeasible, consider purchasing an Azure Savings Plan. Like Azure Reservations, Azure Savings Plans operate on a one-year or three-year term and automatically apply to any resources within benefit scope. You commit to spend a fixed hourly amount on compute resources irrespective of SKU or region. Best for workloads that utilize different resources and/or different data center regions.

Azure Hybrid Benefit

Azure Hybrid Benefit for Azure Kubernetes Service (AKS) allows you to maximize your on-premises licenses at no extra cost. Use any qualifying on-premises licenses that also have an active Software Assurance (SA) or a qualifying subscription to get Windows VMs on Azure at a reduced cost.

Next steps

Cost optimization is an ongoing and iterative effort. Learn more by reviewing the following recommendations and architecture guidance:

• What is AKS Automatic? - Get cost optimization built in by default with a fully managed cluster experience.
• Microsoft Azure Well-Architected Framework for AKS: Cost optimization design principles
• Baseline architecture guide for AKS
• Optimize compute costs on AKS
• AKS cost optimization techniques