VM (NV series) unstable after failed installation of AZURE NVidia driver

Question

VM (NV series) unstable after failed installation of AZURE NVidia driver

Todd Knight 0

VM is NV (NVidia) series but came with MS graphic drivers, when AZURE Nvidia driver was selected, install failed. VM is now unstable. Reboots every 2 minutes.

Todd Knight 0 Reputation points

2025-11-11T23:22:33.35+00:00
This is the error. Point being the VM is now unusable as it automatically reboots every 90 seconds. There appears to not be a reversion. I have also removed all graphic drivers.

The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'. (Code: ResourceDeploymentFailure, Target:** )
Jilakara Hemalatha 4,360 Reputation points Microsoft External Staff Moderator

2025-11-12T00:14:16.22+00:00
Hi @Todd Knight

Thank you for reaching out Q/A. The error "The resource operation completed with terminal provisioning state 'Failed'" can result from various underlying issues.

To identify the root cause, it's important to review the entire error message, including the "details" field. You can access this by navigating to your deployment in the Azure Portal, selecting the failed deployment, and clicking on “(Error details)”. Then, switch to the “Raw Error” tab to view the full error output, which often contains critical information needed for troubleshooting.

To help us investigate the NV-series VM instability and provide the most effective guidance for recovery, could you please provide answers to the following questions:

What is the specific NV-series VM SKU and size you are using?

What is the exact OS image and version running on the VM?

In which region was the VM deployed, and was this a new VM or an existing VM at the time of creation?

Was Secure Boot or Trusted Launch enabled on the VM prior to the driver installation attempt?

Did you use the Azure GPU Driver Extension to install the NVIDIA drivers, or was it a manual installation?

Have you tried stopping and redeploying the VM to clear any cached or temporary states?

What version of the NVIDIA GPU driver was attempted for installation?

Are there specific workloads or applications affected by this driver installation issue?

Additionally, could you please check your private messages and provide the necessary details requested there?
Jilakara Hemalatha 4,360 Reputation points Microsoft External Staff Moderator

2025-11-12T16:56:20.08+00:00

Hi @Todd Knight

Could you please check the full error details and let us know? Also, it would be great if you could share the questions and any additional info via private message.

1 answer

Your answer

Todd Knight 0 Reputation points

2025-11-11T23:22:33.35+00:00

This is the error. Point being the VM is now unusable as it automatically reboots every 90 seconds. There appears to not be a reversion. I have also removed all graphic drivers.

The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'. (Code: ResourceDeploymentFailure, Target:** )
Jilakara Hemalatha 4,360 Reputation points Microsoft External Staff Moderator

2025-11-12T00:14:16.22+00:00

Hi @Todd Knight

Thank you for reaching out Q/A. The error "The resource operation completed with terminal provisioning state 'Failed'" can result from various underlying issues.

To identify the root cause, it's important to review the entire error message, including the "details" field. You can access this by navigating to your deployment in the Azure Portal, selecting the failed deployment, and clicking on “(Error details)”. Then, switch to the “Raw Error” tab to view the full error output, which often contains critical information needed for troubleshooting.

To help us investigate the NV-series VM instability and provide the most effective guidance for recovery, could you please provide answers to the following questions:

What is the specific NV-series VM SKU and size you are using?

What is the exact OS image and version running on the VM?

In which region was the VM deployed, and was this a new VM or an existing VM at the time of creation?

Was Secure Boot or Trusted Launch enabled on the VM prior to the driver installation attempt?

Did you use the Azure GPU Driver Extension to install the NVIDIA drivers, or was it a manual installation?

Have you tried stopping and redeploying the VM to clear any cached or temporary states?

What version of the NVIDIA GPU driver was attempted for installation?

Are there specific workloads or applications affected by this driver installation issue?

Additionally, could you please check your private messages and provide the necessary details requested there?
Jilakara Hemalatha 4,360 Reputation points Microsoft External Staff Moderator

2025-11-12T16:56:20.08+00:00

Hi @Todd Knight

Could you please check the full error details and let us know? Also, it would be great if you could share the questions and any additional info via private message.

Answer 1

If your NV series VM is unstable and rebooting every 2 minutes after a failed installation of the Azure NVIDIA driver, you may want to consider the following troubleshooting steps:

Check Extension Status: Verify the status of the NVIDIA GPU Driver Extension in the Azure portal or by using PowerShell or the Azure CLI. You can run the following command to check the deployment state:
```
   Get-AzVMExtension -ResourceGroupName <myResourceGroup> -VMName <myVM> -Name <myExtensionName>
```
Or using Azure CLI:
```
   az vm extension list --resource-group <myResourceGroup> --vm-name <myVM> -o table
```
Review Output Logs: Check the output logs for the NVIDIA GPU Driver Extension deployment. You can find these logs under C:\WindowsAzure\Logs\Plugins\Microsoft.HpcCompute.NvidiaGpuDriverWindows\.
Respond to Error Codes: If you encounter specific error codes during the installation, refer to the documentation for potential follow-up actions. For example, if the error code indicates an operation timeout, you may need to retry the operation.
Reboot the VM: If the installation process was interrupted, a reboot may be necessary to complete the installation.
Manual Driver Installation: If the extension installation continues to fail, consider manually installing the NVIDIA drivers. You can follow the manual installation instructions provided in the documentation for your specific Linux distribution or Windows version.
Check for Compatibility Issues: Ensure that the driver version you are trying to install is compatible with your VM size and type. For instance, certain GRID driver versions may not be compatible with specific NV series SKUs.

If the problem persists, you may need to remove the NVIDIA driver extension and reinstall it or consider restoring the VM to a previous state if possible.

References:

Share via

VM (NV series) unstable after failed installation of AZURE NVidia driver

1 answer

Your answer