AKS cannot pull from private ABAC-enabled ACR: 401 persists after correct kubelet roles, reconcile, node-image refresh, and pull secret

Update: controlled ACR diagnostic result

We enabled temporary ACR login and repository-event diagnostics, then created exactly one closed-host retry Job using the same immutable Worker image digest. The Job again failed before the container started with the same ImagePullBackOff/ anonymous-token 401.

The ACR login events now show the following sequence from the AKS node’s containerd/2.3.7-3 runtime:

1. The configured AKS kubelet managed identity authenticated successfully to the registry (200).
2. Roughly 0.1 seconds later, the same runtime made an unauthenticated login/token request, which ACR rejected (401).
3. No ContainerRegistryRepositoryEvents were recorded for estate-planning/worker.

This indicates the request reaches ACR and the kubelet identity itself is accepted. The failure occurs after successful managed-identity authentication but before any repository manifest or layer read.

We removed the temporary Job, diagnostic setting, and namespace-scoped Writer assignment afterward.

Could you advise why AKS containerd falls back to an unauthenticated token request after the managed-identity login succeeds in a private ACR using ABAC repository permissions? Is there a supported AKS/ACR configuration or known issue affecting this credential handoff?

Welcome to Microsoft Q&A

Hello @Gregory DuPont I hope you are doing well,

I;m thinking , this does not appear to be a typical AKS-to-ACR connectivity issue. You've already validated the kubelet identity, ABAC role assignments, private endpoint connectivity, DNS resolution, reconciliation, and node image refresh.

A few observations:

• There is currently no documented limitation that prevents AKS from pulling an image by immutable digest from an ABAC-enabled ACR.
• The message: failed to fetch anonymous token does not necessarily mean AKS intentionally attempted an anonymous pull. It indicates that the registry authentication flow reached a point where ACR returned a 401 during token acquisition. The kubelet event alone doesn't conclusively identify which credential path was evaluated.
  • Since both the managed identity approach and the repository-scoped token/imagePullSecret approach produce the same result, the next step is to determine what ACR actually received rather than relying solely on the pod event text.

I would recommend reviewing ACR diagnostic logs if they are available:

ContainerRegistryLoginEvents

| where TimeGenerated > ago(2h)

| project TimeGenerated, Identity, ResultDescription

| order by TimeGenerated desc

and:

ContainerRegistryRepositoryEvents

| where Repository == "estate-planning/worker"

| order by TimeGenerated desc

These logs can help determine:

• Whether the request reached ACR.
• Which identity was presented.
• Whether the ABAC condition evaluation failed.
• Whether the repository authorization succeeded but manifest retrieval failed.

Also, verify the live pod definition rather than the deployment manifest to confirm the pull secret is actually attached and matches the exact login server hostname:

kubectl get pod <pod> -o jsonpath='{.spec.imagePullSecrets[*].name}'

and decode the stored .dockerconfigjson to verify the registry server entry exactly matches:

<private-registry>.azurecr.io

One additional data point that may help: were ACR diagnostic logs enabled before the failed pull attempts? If not, enabling them and performing a single controlled retry may provide the clearest evidence of whether the failure is occurring during authentication, ABAC authorization, or manifest retrieval.

Useful references:

• https://learn.microsoft.com/azure/container-registry/container-registry-rbac-abac-repository-permissions
• https://learn.microsoft.com/azure/aks/deploy-canipull-application-aks
• https://learn.microsoft.com/azure/container-registry/monitor-container-registry
• https://learn.microsoft.com/azure/azure-kubernetes/connectivity/cannot-pull-image-from-acr-to-aks-cluster

If ACR logs show the correct identity, repository (estate-planning/worker), and successful role evaluation but the registry still returns 401, I would recommend opening a Microsoft support case with the AKS resource ID, ACR resource ID, UTC timestamps, and relevant ACR login/repository log entries, as that may indicate a service-side issue requiring backend investigation.

If my answer helped you resolve your issue, please consider marking it as the correct answer. This helps others in the community find similar solutions.