AML AmlCompute Standard-NC6s-v3 provisions stale Azure Batch node image with vulnerable Ubuntu 22.04 host packages

Question

AML AmlCompute Standard-NC6s-v3 provisions stale Azure Batch node image with vulnerable Ubuntu 22.04 host packages

Itamar Latnik 65 Microsoft Employee

We have SFI host/OS vulnerability findings for an Azure ML AmlCompute cluster.

Provider service from SFI InventoryAttributes:
Azure Batch

SFI action:
RedeployHostedOBOResource

The SFI report flags host OS packages on the AML/Azure Batch backing VM, not packages inside our AML environment Docker image.

Flagged host OS build:
Linux:ubuntu-22.04-Jammy Jellyfish:6.8.0-1052-azure

Flagged packages:
- libgnutls30: installed 3.7.3-4ubuntu1.8, required 3.7.3-4ubuntu1.9
- libarchive13: installed 3.6.0-1ubuntu1.6, required 3.6.0-1ubuntu1.7
- rsync: installed 3.2.7-0ubuntu0.22.04.4, required 3.2.7-0ubuntu0.22.04.6
- bind9-dnsutils: installed 1:9.18.39-0ubuntu0.22.04.3, required 1:9.18.39-0ubuntu0.22.04.4
- bind9-host: installed 1:9.18.39-0ubuntu0.22.04.3, required 1:9.18.39-0ubuntu0.22.04.4
- bind9-libs: installed 1:9.18.39-0ubuntu0.22.04.3, required 1:9.18.39-0ubuntu0.22.04.4

We submitted a fresh AML job to the affected compute to force a new node allocation. The newly allocated node still boots the same stale host kernel:

uname -a:
Linux ... 6.8.0-1052-azure #58~22.04.1-Ubuntu SMP Thu Mar 26 05:02:21 UTC 2026 x86_64 GNU/Linux

/proc/version:
Linux version 6.8.0-1052-azure ... #58~22.04.1-Ubuntu SMP Thu Mar 26 05:02:21 UTC 2026

Kernel release:
6.8.0-1052-azure

Kernel build:
#58~22.04.1-Ubuntu SMP Thu Mar 26 05:02:21 UTC 2026

Conclusion:
Redeploying/deleting/recreating the AML compute does not remediate the finding yet, because fresh nodes still provision from the same stale Azure Batch/AML platform node image.

This is not caused by our Dockerfile or AML environment image. AML jobs run inside containers and cannot patch the host OS. The Docker container shares the host kernel, and the SFI LinkedObject points to the AML compute resource:
.../Microsoft.MachineLearningServices/workspaces/dre/computes/standard-nc6s-v3-unded

Request:
Please route this to the Azure Batch / AML AmlCompute node-image owner and rebase/refresh the HOBO node image for this compute family/region so fresh AmlCompute nodes provision with a patched Ubuntu 22.04 Azure kernel/image and updated host packages.

Vinayteja Lellela 95 Reputation points Microsoft External Staff Moderator

2026-06-21T16:36:55.1566667+00:00
Hey! Thanks for the detailed repro/observations—this makes it clear the vulnerability findings are for the Microsoft-managed host OS on the AML-managed Azure Batch nodes, not anything inside your AML job container.

What’s going on (based on the docs)

For Azure Machine Learning managed compute hosts / compute clusters, Microsoft uses Microsoft-managed OS VM images. When you provision a node, it pulls the latest updated VM image at provisioning time, but Microsoft doesn’t actively scan/“heal” vulnerabilities on nodes that are already running/allocated. The vulnerability management doc also calls out that:

Nodes that aren’t idle can’t be updated to the latest VM image.

For compute clusters, nodes may only get upgraded to the latest VM image under certain scaling conditions.

So, if your compute family/region is still using a stale Azure Batch/AML platform node image, simply deleting/redeploying or starting a new job may still land you on the same underlying host image, meaning the host kernel and host package set remain unchanged.

Recommended path for remediation

Given your conclusion (“containers can’t patch the host OS”), the correct owner surface is indeed the AML-managed host VM image / Azure Batch node image pipeline.
From the vulnerability management guidance for compute clusters:

Compute clusters automatically upgrade nodes to the latest VM image when nodes are allowed to cycle (and especially if the cluster can reduce to zero).

If the cluster has continuous workload or minimum nodes > 0, it can’t scale down to zero, so it can’t pick up new VM images as cleanly.

So the immediate operational workaround (to force host-image refresh) is usually to allow the cluster to scale down to zero:

Temporarily set min nodes = 0

Let the cluster reduce to 0 nodes (i.e., ensure there are no continuously scheduled jobs keeping nodes alive)

Then schedule again so new nodes are provisioned from the most current VM imageFrom the vulnerability management guidance for compute clusters:
Compute clusters automatically upgrade nodes to the latest VM image when nodes are allowed to cycle (and especially if the cluster can reduce to zero).

If the cluster has continuous workload or minimum nodes > 0, it can’t scale down to zero, so it can’t pick up new VM images as cleanly.

So the immediate operational workaround (to force host-image refresh) is usually to allow the cluster to scale down to zero:

Temporarily set min nodes = 0

Let the cluster reduce to 0 nodes (i.e., ensure there are no continuously scheduled jobs keeping nodes alive)

Then schedule again so new nodes are provisioned from the most current VM image

Why your new allocation still shows the same kernel/packages

The docs indicate AML managed nodes rely on VM images, and upgrading to a new VM image depends on provisioning/scale-down behavior. If the underlying VM image in that region for that compute family hasn’t been refreshed yet (or is still the current image version used by the platform), new nodes can still boot with the same kernel line and host package versions you’re seeing.
Itamar Latnik 65 Reputation points Microsoft Employee

2026-06-23T13:20:40.64+00:00

Thank you Vinayteja on the quick response.

I already set all min nodes = 0 and validate that there are no job running for half a day, but unfortunately it didn't changed yet.

===== HOST KERNEL (uname) =====

Linux 7451f8c0b3c04a83b1674c79521926a2000000 6.8.0-1052-azure #58~22.04.1-Ubuntu SMP Thu Mar 26 05:02:21 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux

===== HOST KERNEL (/proc/version) =====

Linux version 6.8.0-1052-azure (buildd@lcy02-amd64-017) (x86_64-linux-gnu-gcc-11 (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #58~22.04.1-Ubuntu SMP Thu Mar 26 05:02:21 UTC 2026

===== CONTAINER OS (/etc/os-release) =====

PRETTY_NAME="Ubuntu 22.04.5 LTS"

NAME="Ubuntu"

VERSION_ID="22.04"

VERSION="22.04.5 LTS (Jammy Jellyfish)"

VERSION_CODENAME=jammy

ID=ubuntu

ID_LIKE=debian

HOME_URL="https://www.ubuntu.com/"

SUPPORT_URL="https://help.ubuntu.com/"

BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"

PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"

UBUNTU_CODENAME=jammy

===== KERNEL RELEASE =====

6.8.0-1052-azure

===== KERNEL BUILD (uname -v) =====

#58~22.04.1-Ubuntu SMP Thu Mar 26 05:02:21 UTC 2026
Itamar Latnik 65 Reputation points Microsoft Employee

2026-06-25T11:32:46.6466667+00:00

For future reference, no support on NC6 and need to move to other cluster type https://learn.microsoft.com/en-us/azure/virtual-machines/ncv3-retirement

Answer accepted by question author

0 additional answers

Your answer

Itamar Latnik 65 Reputation points Microsoft Employee

2026-06-23T13:20:40.64+00:00

Thank you Vinayteja on the quick response.

I already set all min nodes = 0 and validate that there are no job running for half a day, but unfortunately it didn't changed yet.

===== HOST KERNEL (uname) =====

Linux 7451f8c0b3c04a83b1674c79521926a2000000 6.8.0-1052-azure #58~22.04.1-Ubuntu SMP Thu Mar 26 05:02:21 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux

===== HOST KERNEL (/proc/version) =====

Linux version 6.8.0-1052-azure (buildd@lcy02-amd64-017) (x86_64-linux-gnu-gcc-11 (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #58~22.04.1-Ubuntu SMP Thu Mar 26 05:02:21 UTC 2026

===== CONTAINER OS (/etc/os-release) =====

PRETTY_NAME="Ubuntu 22.04.5 LTS"

NAME="Ubuntu"

VERSION_ID="22.04"

VERSION="22.04.5 LTS (Jammy Jellyfish)"

VERSION_CODENAME=jammy

ID=ubuntu

ID_LIKE=debian

HOME_URL="https://www.ubuntu.com/"

SUPPORT_URL="https://help.ubuntu.com/"

BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"

PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"

UBUNTU_CODENAME=jammy

===== KERNEL RELEASE =====

6.8.0-1052-azure

===== KERNEL BUILD (uname -v) =====

#58~22.04.1-Ubuntu SMP Thu Mar 26 05:02:21 UTC 2026
Itamar Latnik 65 Reputation points Microsoft Employee

2026-06-25T11:32:46.6466667+00:00

For future reference, no support on NC6 and need to move to other cluster type https://learn.microsoft.com/en-us/azure/virtual-machines/ncv3-retirement

Answer 1

Alex Burlachenko 25,030 MVP Volunteer Moderator

hi Itamar Latnik, thx for sharing urs issue here at Q&A portal,

seems like the issue is on the platform side rather than in your AML environment. The packages in the report are host OS packages on the Azure Batch VM backing AmlCompute. Since AML jobs run inside containers, u can update packages in the container, but not the host kernel or the VM image itself. Microsoft describes AmlCompute as managed infrastructure, so the underlying node image is maintained by the platform https://learn.microsoft.com/en-us/azure/machine-learning/concept-compute-target?view=azureml-api-2

Since recreating the compute still provisions nodes with the same kernel (6.8.0-1052-azure) and package versions, I'd expect new nodes are still being created from the current platform image for that VM family and region.

At this point I'd ask the Azure ML/Azure Batch team to verify which node image version is currently deployed for Standard_NC6s_v3 in your region and whether a newer image containing the required Ubuntu security updates is available or still rolling out. From what u shared, this doesn't look like something that can be fixed from the customer side.

rgds,

Alex

&

If my answer was helpful pls mark it and additional thx if u follow me at Q&A portal

Itamar Latnik 65 Reputation points Microsoft Employee

2026-06-23T14:05:38.7433333+00:00

Thanks @Alex Burlachenko for the quick response.

I've created a ticket with the relevant information. hope it will resolve soon.
Itamar Latnik 65 Reputation points Microsoft Employee

2026-06-25T11:32:26.81+00:00

For future reference, no support on NC6 and need to move to other cluster type
https://learn.microsoft.com/en-us/azure/virtual-machines/ncv3-retirement