host hypervisor hangs on guest PCIe reset during NVMe and GPU passthrough via DDA

Question

host hypervisor hangs on guest PCIe reset during NVMe and GPU passthrough via DDA

Foil Benny 20

Dear Community,

We currently deploy a high-performance compute cluster designed to run resource-intensive workloads inside dedicated virtual machines, and we have encountered a severe host-level stability bottleneck. Under standard operational conditions, our primary hypervisor nodes demonstrate solid uptime and predictable resource allocation across all virtualized environments.

To support automated benchmarking and intensive data processing for platforms such as tech-gear-review.net and cloud-hosting-compare.org, our infrastructure demands direct, raw hardware access to eliminate virtualized storage and graphics bottlenecks.

Initiating a system reboot or triggering a PCIe reset from within the guest OS causes the physical host hypervisor to hang completely.
Passing through an NVMe controller or GPU to a guest VM using Discrete Device Assignment (DDA) causes immediate host instability during device state re-initialization.
The host becomes entirely unresponsive on the management interface, forcing a hard power cycle to restore service.

We have attempted to manually adjust device assignment policies and execute script-based driver dismounting on the host layer, but the host lockup continuously recurs upon guest system reboot.

what is the best practice for properly configuring driver override parameters (DDA Unbind) and managing or disabling PCIe Access Control Services (ACS) checks to guarantee stable passthrough operations?

I would appreciate any technical insights or configuration guidance. Thank you in advance for your time and help.

0 comments

1 answer

Your answer

Answer 1

Hello,

The instability you’re describing is a known risk when using Discrete Device Assignment (DDA) with PCIe devices such as NVMe controllers and GPUs. The root cause is almost always tied to how the hypervisor handles device state transitions during guest resets. When the guest OS reboots or issues a PCIe reset, the host kernel driver must unbind and rebind the device cleanly. If the override parameters or ACS routing are misconfigured, the host can hang because the PCIe fabric is left in an inconsistent state.

For driver override (DDA Unbind), the best practice is to explicitly dismount the device from the host before assigning it to the VM. On Windows Server Hyper-V, this is done using Dismount-VMHostAssignableDevice followed by Set-VMHostAssignableDevice -LocationPath <device location> -Remove. The device should not have any active driver bound in the host OS. You should confirm in Device Manager that the device shows as “Not started” after unbinding. Avoid script-based hot unbinds during guest reboot; instead, configure the device assignment at VM creation and let Hyper-V manage the binding lifecycle.

Regarding PCIe Access Control Services (ACS), disabling ACS checks is not recommended because it compromises isolation between PCIe endpoints. Microsoft’s guidance is to leave ACS enabled and instead ensure that the hardware platform supports proper ACS routing. If your hardware does not expose ACS correctly, passthrough will remain unstable. Unlike Linux/KVM, Hyper-V does not provide a tunable to bypass ACS enforcement. The only supported path is to use devices and platforms that are ACS-compliant. Attempting to disable ACS at the firmware or driver level will lead to exactly the kind of host lockups you’re seeing.

In short, the stable configuration is: unbind the device from the host using the supported PowerShell cmdlets, verify that no host driver is attached, and only assign ACS-compliant devices to guests. If your NVMe controller or GPU does not advertise proper ACS, passthrough on Hyper-V will not be reliable. At that point, the only safe option is to use SR-IOV or vGPU technologies instead of raw DDA passthrough.

I hope you've found something useful here. If it helps you get more insight into the issue, it's appreciated to accept the answer. Should you have more questions, feel free to leave a message. Have a nice day!

HP.