Cluster Resource VM stuck. I can't remove it

Question

_{Friday, December 15, 2017 1:44 AM}

Here's the thing.

I have a Hyper-V 2016 Cluster with 2 nodes and some VMs.

In a quick migration one of these VMs get stucked!!! It disapeared from both nodes hyper-v manager but it still on the Failover Cluster Manager. 

In the shell, I run Get-ClusterResources and I see it there, but the VM with offline status and the VM settings with OnlinePending status.

I did everything I could to rescue it or remove it, but nothing seems to work.

I run Remove-ClusterResource "resource" -force but the shell give me no answers, it keeps stuck in the command and nothing happens!

I need some help. I just need to remove it. After this VM get stucked everything seems to get bad on the cluster! The FailOver Cluster Manager doesn't show the VMs or disks details and seems keeps always trying to update the status.

It is give some error events too. Like (ID) 19060, saying it can't cancel the VM registry because it is runing a restoring operation.

Forgive my bad english.

All replies (6)

_{Friday, December 15, 2017 5:12 AM}

Hi,

Restart the VMMS service on both the nodes and check. Still not resolved then proceed further 

When you ran Get-ClusterResources, did you get the host where the VM was running earlier (I mean the current owner of that resource)?

If you dont see there, please check in Hyper-V manager on both the nodes if the VM is showing up there. IF the VM is present try to start and then add it to the cluster again (but this is very rare). still it does not show? 

Please check the following path (in both the nodes) for the shortcut file which actually points to the VM's config file 
%systemdrive%\programdata\Microsoft\Windows\Hyper-V\Virtual Machines\VM_GUID> this is the shortcut location and you would know the location of the actual xml file location

(VM_GUID is nothing but the VM ID. You can get it from the .xml file name where the VM is saved)

If you still dont find the shortcut for this problematic VM, then create a symbolic link and then check the Hyper-V manager, it should show there

To do so, open the command prompt in admin privilege

C:\mklink "%systemdrive%\programdata\Microsoft\Windows\Hyper-V\Virtual Machines\VM_GUID.xml>" "V:\manualrecover\Virtual Machines\VM_GUID.xml>"

Regards,
Bala

_{Friday, December 15, 2017 2:58 PM}

Thanks, Bala.

I got it fixed by runing Remove-ClusterResource "resource" -force, but it took all night long.

Today morning I saw it was removed from the cluster and almost everything seems to have been back to normal. Now, the only thing that seems to not be working properly it the vm listing on the second host! It keeps hanged saying loading vms, but never show them. the get-vm command is getting the same result.

the hyperv-node2 is give me some event errors like "16300 Microsoft-Windows-Hyper-V-VMMS" white no content and other event (16300) saying that the previous error event couldn't locate the event description because the component that generates that event is not installed or it's installation is corrupted.

I couldn't find what is generating that error event and why the second node is taking too long to list vms (get-vm) yet.

_{Saturday, December 16, 2017 1:06 PM}

Hi,

Could you please restart the VMMS service in the problematic node and then check whether the VMs are listing?

Regards,
Bala

_{Monday, December 18, 2017 6:48 PM}

I would suggest finding a time to reboot the nodes. One by one and have eventualy any stuck vm's failover to the other node. Are you on latest patches as well on WS 2016? If not id recomend patching up. Is this a 2 node S2D cluster or standard with SAN?

Regards
Jan-Tore

_{Friday, December 22, 2017 6:53 AM}

Hi andr3,

Just to check if you get any progress with the issue, and have you tried the suggestions above, welcome to feedback.

Best Regards,

Anne

Please remember to mark the replies as answers if they help.
If you have feedback for TechNet Subscriber Support, contact [email protected].

_{Wednesday, May 2, 2018 10:31 PM}

I also got a stucked VM with status like "pending" when I run a Cluster Validation.

When I do a Get-ClusterRessource in Powershell it lists all other VM's just fine.

But this previous VM is listed as : 

Get-ClusterResource : The object has been deleted from the cluster.
At line:1 char:1

• Get-ClusterResource

    + CategoryInfo          : ObjectNotFound: (:) [Get-ClusterResource], ClusterCmdletException
    + FullyQualifiedErrorId : ClusterObjectNotFound,Microsoft.FailoverClusters.PowerShell.GetResourceCommand
Get-ClusterResource : The object has been deleted from the cluster.
At line:1 char:1

• Get-ClusterResource

    + CategoryInfo          : ObjectNotFound: (:) [Get-ClusterResource], ClusterCmdletException
    + FullyQualifiedErrorId : ClusterObjectNotFound,Microsoft.FailoverClusters.PowerShell.GetResourceCommand

Any suggestions on how to get this Cluster Role removed?

Right now I cannot Live Migrate any VM's, probably due to this issue - so I cant even put the source host into maintenance mode. (Source when the Cluster Role got stuck)