Image deployment is failing with AKS

Question

Image deployment is failing with AKS

Diptesh Kumar 166

I have below image in ACR, you can see:

User's image

in Deployment file also I gave same image name: but it is failing with following error. I verified POD logs you can see below

User's image

Please see the events of POD below :

Events:

Type Reason Age From Message

Normal Scheduled 43s default-scheduler Successfully assigned default/aks-agent-deployment-7d7c5cc86-25269 to aks-jbk-41249360-vmss000001

Normal Pulling 42s kubelet Pulling image "acrsss.azurecr.io/acrrepo:20250419.39"

Normal Pulled 22s kubelet Successfully pulled image "acrsss.azurecr.io/acrrepo:20250419.39" in 19.928s (19.928s including waiting). Image size: 262301303 bytes.

Normal Created 8s (x3 over 22s) kubelet Created container: azp-agent

Normal Started 8s (x3 over 22s) kubelet Started container azp-agent

Normal Pulled 8s (x2 over 22s) kubelet Container image "acrsss.azurecr.io/acrrepo:20250419.39" already present on machine

Warning BackOff 8s (x3 over 21s) kubelet Back-off restarting failed container azp-agent in pod aks-agent-deployment-7d7c5cc86-25269_default(e3b6be02-05a3-4e1e-bbee-f5ec9e3c9e33)

Please suggest fix. Thank you

Arko 1,895 Reputation points Microsoft External Staff

2025-04-21T08:25:41.7766667+00:00
Hello Diptesh Kumar,

I tried to mimic your setup, and I think I figured out the root cause and found the fix for it.

The issue you're facing where the pod pulls the container image successfully from ACR but fails with repeated restarts and the message Back-off restarting failed container is not due to an image pull error, but due to a runtime crash inside the container.

Why the container crashes?

Ans- Your Docker container expects a required environment variable (in your case, AZP_TOKEN) to authenticate and start a DevOps agent inside the container. If this token is missing or invalid, the container exits immediately, causing the BackOff crash loop you’re seeing.

I built and pushed the image acrsss.azurecr.io/acrrepo:20250419.39 to Azure Container Registry.

Deployed it to AKS using a Deployment.yaml with a secret named azp-secret containing a dummy token (invalid-token-xyz). My container’s start.sh script checked if the token was valid. Since it was not, it exited with code 1 and od logs showed-

[error] Missing or invalid AZP_TOKEN. Agent will not start.

How to fix it?

Ans- You need to make sure that the correct Azure DevOps PAT token is stored in the Kubernetes secret.

First check what is the value?

kubectl get secret azp-secret -o yaml

If incorrect then fix it using-

kubectl create secret generic azp-secret \ --from-literal=AZP_TOKEN=<your-valid-pat-token> \ --dry-run=client -o yaml | kubectl apply -f -

Please note, that the PAT must have at least Agent Pools (read, manage) permissions.

Ensure your Deployment YAML refers to this secret

env: - name: AZP_TOKEN valueFrom: secretKeyRef: name: azp-secret key: AZP_TOKEN

Do a fresh kubectl rollout restart deployment aks-agent-deployment

Now check, it should work

kubectl get pods
Diptesh Kumar 166 Reputation points

2025-04-21T08:45:03.69+00:00

H iArko,

Yes, I figured out that but I am seeing following issue now

can you assist on this one also..

https://learn.microsoft.com/en-us/answers/questions/2259606/fdeploy-image-is-failing-aks?source=docs
Diptesh Kumar 166 Reputation points

2025-04-21T09:27:36.9233333+00:00

Hi Arko

and one more question

below 2 secrets are different right? we have to create secret at ACR level also right?

I mean i need to create below secret also right along with below secret?

kubectl create secret docker-registry acr-secret --docker-server=acrrrrr.azurecr.io --docker-username=acrrrr--docker-password="XXX"
Diptesh Kumar 166 Reputation points

2025-04-22T05:13:00.81+00:00

Hi Arko,

Did you get a chance to look my previous comment?
Arko 1,895 Reputation points Microsoft External Staff

2025-04-22T14:12:49.81+00:00

FDeploy Image is failing question? I suggested a fix on that link. Kindly check once and let me know.

and for the two secrets you pointed out serve different purposes, yes you need to create both secrets separately. One is for the image pull authentication, the other is for passing environment variables securely to your container.
Arko 1,895 Reputation points Microsoft External Staff

2025-04-22T14:35:33.6533333+00:00

Hope I was able to clear your queries Diptesh Kumar.
Diptesh Kumar 166 Reputation points

2025-04-23T04:30:26.6533333+00:00
Hi Arko,

I have 2 quick follow up questions , could you please clarify:

1.

yes for second secret I have executed below command manually before pipeline running ,

kubectl create secret generic azp-secret --from-literal=AZP_TOKEN=XXXX --namespace=default

and in that case using directly deploy stage is enough right? we no need to use again separate secret task right ? before kubernaties maniefest deploy task in the pipeline

2.

We supposted use k8s service connection only right for deploying k8s manifest task? not azure resource manager task, right?
Arko 1,895 Reputation points Microsoft External Staff

2025-04-23T12:03:41.9833333+00:00
Qs- Do you need a separate Kubernetes secret creation task in the pipeline?
Ans- No, not if you've already created the secret manually beforehand. You do not need to create it again from the pipeline as long as you're deploying into the same namespace (default), the secret exists before the deployment runs and as long as you’re not changing the token value dynamically per run. If the secret is static or manually maintained, it’s completely fine to proceed directly with the deployment stage using your manifest.

Qs-Which service connection should you use in the Kubernetes manifest deployment task?
Ans- You can use connectionType and kubernetesServiceConnection parameters within the KubernetesManifest@0 tasks.

Checkout MS docs-

https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/kubernetes-manifest-v0?view=azure-pipelines

https://learn.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops
Diptesh Kumar 166 Reputation points

2025-04-23T15:12:07.4+00:00

But how does deployment task knows secret which we created manually?
Arko 1,895 Reputation points Microsoft External Staff

2025-04-24T07:54:50.83+00:00

Hello Diptesh Kumar, the deployment task doesn’t know about the secret. It just applies the manifest. As long as the secret (azp-secret) exists in the correct namespace, Kubernetes itself takes care of injecting that secret into the pod when it starts.

This is confirmed in the Microsoft documentation. Checkout this line-

This shows that the task assumes the secret is already created inside the cluster. It does not attempt to create or validate it. Kubernetes will use the named secret during pod startup and not during deployment application.

Again, from the same document, it also shows how a secret can be created dynamically if needed. But again, if you already created the secret manually, you can skip this task.
Arko 1,895 Reputation points Microsoft External Staff

2025-04-24T07:57:20.34+00:00

Hope I was able to clear your follow up query. Let me know if I can convert the same as answer for you to accept it for anyone else having these similar queries on QnA forum can refer from here. Thanks.

Your answer

Arko 1,895 Reputation points Microsoft External Staff

2025-04-21T08:25:41.7766667+00:00

Hello Diptesh Kumar,

I tried to mimic your setup, and I think I figured out the root cause and found the fix for it.

The issue you're facing where the pod pulls the container image successfully from ACR but fails with repeated restarts and the message Back-off restarting failed container is not due to an image pull error, but due to a runtime crash inside the container.

Why the container crashes?

Ans- Your Docker container expects a required environment variable (in your case, AZP_TOKEN) to authenticate and start a DevOps agent inside the container. If this token is missing or invalid, the container exits immediately, causing the BackOff crash loop you’re seeing.

I built and pushed the image acrsss.azurecr.io/acrrepo:20250419.39 to Azure Container Registry.

Deployed it to AKS using a Deployment.yaml with a secret named azp-secret containing a dummy token (invalid-token-xyz). My container’s start.sh script checked if the token was valid. Since it was not, it exited with code 1 and od logs showed-

[error] Missing or invalid AZP_TOKEN. Agent will not start.

How to fix it?

Ans- You need to make sure that the correct Azure DevOps PAT token is stored in the Kubernetes secret.

First check what is the value?

kubectl get secret azp-secret -o yaml

If incorrect then fix it using-

kubectl create secret generic azp-secret \ --from-literal=AZP_TOKEN=<your-valid-pat-token> \ --dry-run=client -o yaml | kubectl apply -f -

Please note, that the PAT must have at least Agent Pools (read, manage) permissions.

Ensure your Deployment YAML refers to this secret

env: - name: AZP_TOKEN valueFrom: secretKeyRef: name: azp-secret key: AZP_TOKEN

Do a fresh kubectl rollout restart deployment aks-agent-deployment

Now check, it should work

kubectl get pods
Diptesh Kumar 166 Reputation points

2025-04-21T08:45:03.69+00:00

H iArko,

Yes, I figured out that but I am seeing following issue now

can you assist on this one also..

https://learn.microsoft.com/en-us/answers/questions/2259606/fdeploy-image-is-failing-aks?source=docs
Diptesh Kumar 166 Reputation points

2025-04-21T09:27:36.9233333+00:00

Hi Arko

and one more question

below 2 secrets are different right? we have to create secret at ACR level also right?

I mean i need to create below secret also right along with below secret?

kubectl create secret docker-registry acr-secret --docker-server=acrrrrr.azurecr.io --docker-username=acrrrr--docker-password="XXX"
Diptesh Kumar 166 Reputation points

2025-04-22T05:13:00.81+00:00

Hi Arko,

Did you get a chance to look my previous comment?
Arko 1,895 Reputation points Microsoft External Staff

2025-04-22T14:12:49.81+00:00

FDeploy Image is failing question? I suggested a fix on that link. Kindly check once and let me know.

and for the two secrets you pointed out serve different purposes, yes you need to create both secrets separately. One is for the image pull authentication, the other is for passing environment variables securely to your container.
Arko 1,895 Reputation points Microsoft External Staff

2025-04-22T14:35:33.6533333+00:00

Hope I was able to clear your queries Diptesh Kumar.
Diptesh Kumar 166 Reputation points

2025-04-23T04:30:26.6533333+00:00

Hi Arko,

I have 2 quick follow up questions , could you please clarify:

1.

yes for second secret I have executed below command manually before pipeline running ,

kubectl create secret generic azp-secret --from-literal=AZP_TOKEN=XXXX --namespace=default

and in that case using directly deploy stage is enough right? we no need to use again separate secret task right ? before kubernaties maniefest deploy task in the pipeline

2.

We supposted use k8s service connection only right for deploying k8s manifest task? not azure resource manager task, right?
Arko 1,895 Reputation points Microsoft External Staff

2025-04-23T12:03:41.9833333+00:00

Qs- Do you need a separate Kubernetes secret creation task in the pipeline?
Ans- No, not if you've already created the secret manually beforehand. You do not need to create it again from the pipeline as long as you're deploying into the same namespace (default), the secret exists before the deployment runs and as long as you’re not changing the token value dynamically per run. If the secret is static or manually maintained, it’s completely fine to proceed directly with the deployment stage using your manifest.

Qs-Which service connection should you use in the Kubernetes manifest deployment task?
Ans- You can use connectionType and kubernetesServiceConnection parameters within the KubernetesManifest@0 tasks.

Checkout MS docs-

https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/kubernetes-manifest-v0?view=azure-pipelines

https://learn.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops
Diptesh Kumar 166 Reputation points

2025-04-23T15:12:07.4+00:00

But how does deployment task knows secret which we created manually?
Arko 1,895 Reputation points Microsoft External Staff

2025-04-24T07:54:50.83+00:00

Hello Diptesh Kumar, the deployment task doesn’t know about the secret. It just applies the manifest. As long as the secret (azp-secret) exists in the correct namespace, Kubernetes itself takes care of injecting that secret into the pod when it starts.

This is confirmed in the Microsoft documentation. Checkout this line-

This shows that the task assumes the secret is already created inside the cluster. It does not attempt to create or validate it. Kubernetes will use the named secret during pod startup and not during deployment application.

Again, from the same document, it also shows how a secret can be created dynamically if needed. But again, if you already created the secret manually, you can skip this task.
Arko 1,895 Reputation points Microsoft External Staff

2025-04-24T07:57:20.34+00:00

Hope I was able to clear your follow up query. Let me know if I can convert the same as answer for you to accept it for anyone else having these similar queries on QnA forum can refer from here. Thanks.

Share via

Image deployment is failing with AKS

Your answer