Azure Kubernetes Service - Why Are System Pods Being Scheduled On User Node Pools?

Hi devopsfj,

Welcome to the Microsoft Q&A Platform! Thank you for asking your question here.

We understand from your query that you are experiencing an issue where System pods are being scheduling into User Node Pools. I have shared troubleshooting steps that I felt will help resolve the issue you reported.

If the System node pool becomes unhealthy or unavailable, Kubernetes may schedule system pods on available user node pools. For system pods managed by DaemonSets, Kubernetes will schedule one pod per node across all nodes, including user pools.

You can Label System Node Pool Nodes with a unique label and Add Node Affinity to System Pods like CoreDNS and Metric Server. This ensures the system pods are preferred to be scheduled on the System node pool as long as there is capacity.

For example: label: node-role.kubernetes.io/system=true

kubectl label nodes <system-node-name> node-role.kubernetes.io/system=true

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: "node-role.kubernetes.io/system"
              operator: In
              values:
                - "true"

See that User node pools should not tolerate the CriticalAddonsOnly=true:NoSchedule taint to prevent system pods from scheduling there. Review the node tolerations on user node pools to ensure that there are no tolerations for the CriticalAddonsOnly=true:NoSchedule taint that would allow system pods to bypass this restriction.

After checking with the above configurations, redeploy system pods to let Kubernetes reschedule them according to the updated rules.

If you still find any difficulties, please let me know I would like to work closer on this issue.

Thank you.

Hi @anashetty

Currently only the system node pool has the taint CriticalAddonsOnly=true:NoSchedule.

My user nodes do not that have this taint, yet system pods are being scheduled on user nodes?

I understand that some components like Kube Proxy, CSI Provider etc should have one pod on each node, however, core pods like Core DNS which only have two pods are not being scheduled on System Nodes... this doesn't seem correct?

How do you recommend adding these node labels to the Pods, will these not be removed when there is a patch to AKS?

Is there no way out the box without adding custom labels to schedule these system pods on the system node?

I have the same concern.

Seems like some pods that are considered to be system pods do not have affinity to the system nodes. Sometimes all replicas are deployed in system nodes but other times are being scheduled in user nodes as well. We have also noticed that in some cases (e.g., gatekeeper) the MS preconfigured replicas (2) are fully scheduled in user nodes.

In our case we have seen this behaviour with Gatekeeper, ama-logs, istio asm, metrics server and others. Worth mentioning that some of these are AKS add-ons that are just enabled (sometime from the portal) and MS provides the deployment configuration.

I'm interested to understand why MS do not consider affinity for these system pods.

Hi devopsfj,

Thanks for getting back with the detailed explanation. I have provided more troubleshooting steps that might help you.

Node labels added to Pods in Azure Kubernetes Service (AKS) may not persist through patches or upgrades. Using AKS-managed labels like kubernetes.azure.com/mode: system in conjunction with nodeSelector is a best practice for ensuring that system pods are scheduled on the appropriate nodes. This approach helps maintain the stability and performance of your AKS cluster, especially during upgrades and patches.

Try to enforce scheduling system-critical pods on your System node pool using AKS-provided node labels:

List the nodes in your cluster and check their labels: kubectl get nodes --show-labels The nodes in the System node pool should have the label: kubernetes.azure.com/mode=system

Modify the deployment specification of system-critical pods to include a nodeSelector that targets the System node pool. Locate the deployment of a system pod: kubectl get deployments -n kube-system Edit the deployment: kubectl edit deployment coredns -n kube-system This will open the deployment specification in your default editor. Add the nodeSelector: In the spec.template.spec section, add the nodeSelector targeting the kubernetes.azure.com/mode=system label:

spec:
  template:
    spec:
      nodeSelector:
        kubernetes.azure.com/mode: system

After this, save and exit the editor. This will update the deployment.

After updating the deployment, Kubernetes will apply the new nodeSelector, but existing pods may not immediately move to the System node pool. You can force them to reschedule: Delete existing pods for the deployment: kubectl delete pod -n kube-system -l k8s-app=kube-dns Kubernetes will automatically recreate the pods based on the updated deployment configuration. Now verify the pods are running on system node pool: kubectl get pods -n kube-system -o wide

Now Repeat this for Other System Pods to Modify the deployment specification of system-critical pods.

You can try using affinity rules. This method allows you to enforce scheduling preferences while providing fallback options if System nodes become unavailable: Define a preferredDuringSchedulingIgnoredDuringExecution rule: Update the system pod deployments to include a nodeAffinity specification.

spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: "kubernetes.azure.com/mode"
                    operator: "In"
                    values:
                      - system

Please let us know if you have any further queries. I’m happy to assist you further.

Thank you.

Hi Koury, Juan,

Welcome to the Microsoft Q&A Platform! Thank you for asking your question here.

Microsoft's design choice to not enforce pod affinity or anti-affinity for AKS system pods is driven by the need to prioritize flexibility and reliability in system workloads across node pools. By deploying system pods with tolerations for taints applied to system node pools, they can operate seamlessly without being impacted by specific scheduling constraints, allowing Kubernetes to balance workloads effectively during scaling, upgrades, or node failures. While Microsoft provides guidelines for configuring taints and tolerations to isolate workloads, there is no default enforcement for affinity on system pods.

Please let us know if you have any further queries. I’m happy to assist you further.