CRITICAL INCIDENT REPORT HYPER V SMB DIRECT RDMA DEADLOCK LIVE MIGRATION

Hello @Malzaar

The first thing that stands out is the Windows Server 2022 build: 20348.2227. That corresponds to the January 2024 cumulative update (KB5034129), so the hosts are significantly behind current servicing. I would update all cluster nodes to a current supported cumulative update before treating this as an RDMA/SET defect.

For the 99% Live Migration stall, I would also verify whether SMB Multichannel is actually establishing healthy RDMA connections on both hosts:

Get-NetAdapterRDMA
Get-SmbClientNetworkInterface
Get-SmbServerNetworkInterface
Get-SmbMultichannelConnection
Get-SmbMultichannelConnection -IncludeNotSelected

SMB Multichannel is responsible for discovering RDMA interfaces and establishing the RDMA connections used by SMB Direct. Microsoft specifically recommends checking these interfaces and connections when troubleshooting SMB Multichannel/RDMA.

Since you're using RoCEv2, also verify the Mellanox/NVIDIA firmware and driver combination, DCB/PFC configuration, VLANs and QoS configuration end-to-end. Microsoft notes that outdated NIC drivers/firmware can cause RDMA capability and connectivity problems.

As an isolation test, temporarily disable RDMA on the Live Migration interfaces:

Disable-NetAdapterRdma -Name "<LM adapter>"

Then retry the migration. If it completes successfully over SMB/TCP, you've narrowed the issue considerably toward the RDMA/RoCE/driver/fabric path rather than Hyper-V migration itself. Re-enable it afterward with Enable-NetAdapterRdma. Microsoft documents this as a supported method of disabling and enabling RDMA.

I wouldn't conclude that Event 5614/status 0x000000EA proves an SMB Direct "deadlock" by itself. If the problem remains after patching and validating the RDMA fabric, collect the SMBClient/SMBServer, Hyper-V-VMMS and network traces and open a Microsoft support case.

Please "Accept the Answer" if this information helped you. This will help us and others in the community as well.