Hyper-V replication - device does not recognize the command 0x80070016

Question

_{Monday, November 4, 2013 1:58 PM}

Wondering if anyone can shed any light on this.

We have 2 server 2012 (Datacenter) clusters running Hyper-v with replication enabled. The VM's all seems to replicate ok and run quite nicely but then they just stop replicating.

The error logs shows event ID 32086 - Hyper-V suspended replication for virtual machine 'VMNAME' due to a non-recoverable failure. (virtual machine GUID). Resum replication after correcting the failure.

followed closely by

event ID 32022 - Hyper-V could not replicate changes for virtual machine 'VMNAME': the device does not recognize the command. (0x80070016). (virtual machine ID GUID).

Replication is going over port 80 and the firewall is setup to allow this on all servers. The servers aren't in production use and are locally connected (so no latency issues). 

All network communication is fine and the servers are all healthy.

I've seen a similar issue in the forum here but we are not getting the timeout error. This was resolved with a script, but surely we don't have to go back a step and start scripting fixes for whats supposedly core functionality?

Does anyone have any thoughts?

All replies (11)

_{Wednesday, November 20, 2013 8:44 AM ✅Answered}

Hi,

I logged this with Microsoft and they said that they couldn't find the error message or what it meant.

we went back to basics and found that it was some software that was provided to us by our SAN provider (DELL) the host integration toolkit was somehow causing this error. When we removed the software the system worked as normal and no errors were posted.

Thanks for your suggestions though.

_{Thursday, November 7, 2013 6:46 AM}

Hi,

1. This issue is already documented in below given article. As per article, it is suggested to Resync, if it is required on replica VM

http://support.microsoft.com/kb/2889734

This happens when you take backup of the disk on recovery host which has data related to replica VM. Both backup and replication acquire modifying lock on VM to complete their operations. At any point only one process can operate on the VM.

For example: If a user starts backup, it acquires the lock on VM and doing so will make replication fail during the time back-up is going on. Depending on the number of cycles missed, replication health of the secondary machine changes to warning or critical and primary machine replication state goes to error state.

Event IDs 19050, 32022, 32032, 32056, 32315, 32546 and 33680 are logged when Hyper-V replication is in progress

http://support.microsoft.com/kb/2889734

You need following the above steps for the troubleshooting, then do the following action:

1. Turn off Windows ODX

Turn off Windows ODX feature on all nodes (instruction are in the KB and below)

If the storage or network is failing 5120 errors are normal .

ODX Feature:

http://msdn.microsoft.com/en-us/library/windows/desktop/hh848056(v=vs.85).aspx

http://blogs.technet.com/b/keiko_harada/rss.aspx

Disable ODX by changing the FilterSupportedFeaturesMode value for the storage device that does not support ODX to

1.Location: HKLM\System\Control\File System\FilterSupportedFeaturesMode

2. Update the outdated NIC drivers and BIOS as given above

3. As a best practice, it is suggested to install following hotfixes on both the hosts:

KB 2870270 KB 2869923 KB 2838043 KB 2813630 KB 2806542

Hope this helps.

_{Friday, November 8, 2013 9:11 AM}

Hi Alex,

thanks for your suggestions - I have looked over the article provided and it seems to relate to when a backup is taking place. Currently there is no backup of the hypervisor taking place.  Whilst the error code is listed in the article that your provided, the error text is different?

1) I will try disabling ODX and report back (although all of our storage equipment supports ODX)

2) All hypervisors have the latest BIOS and NIC drivers installed

3) I've gone through the hotfixes listed and the hypervisors are not exhibiting any of the error messages or symptoms - therefore should we be applying these?

thanks

I

_{Wednesday, November 20, 2013 1:58 AM}

Hi,

The solution above I replayed include your error event ID, you can apply this hotfixs.

Thanks.

We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time.
Thanks for helping make community forums a great place.

_{Wednesday, November 20, 2013 1:58 AM}

Hi,

I would like to check if you need further assistance.

Thanks.

We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time.
Thanks for helping make community forums a great place.

_{Friday, November 22, 2013 9:26 PM}

Hi Aidan,

              I am experiencing the same issue using Equallogic HitKit. Did you simply remove the HitKit or change a setting?

Regards

_{Monday, November 25, 2013 8:21 AM}

We removed the HITKit and all of the errors went away and replication was working normally.

_{Friday, November 29, 2013 3:55 PM}

I have been bashing my head against the wall for the past few months with this exact issue. Tried everything without success. After about a day replication throws the 32022, 32086 errors. Coincidentally we are too using Equallogic SANS and Hitkit 4.6 on the HV hosts. I am going to give this a go and report back to see if there is continuity and fixes this ongoing issue.

Thanks!!

_{Friday, November 29, 2013 10:11 PM}

Did you remove all the HITKit components? I'm working with Dell to see if they can shed any light.

Thanks

_{Wednesday, December 18, 2013 4:12 PM}

Hi,

I logged this with Microsoft and they said that they couldn't find the error message or what it meant.

we went back to basics and found that it was some software that was provided to us by our SAN provider (DELL) the host integration toolkit was somehow causing this error. When we removed the software the system worked as normal and no errors were posted.

Thanks for your suggestions though.

Thanks very much for putting me on the right track! Just thought I'd give some feedback that the HitKit components was the issue for me as well. It was messing up my HV Replication, FOC stability during backups and randomly messing up HV backups. I de-registered the  "Dell Equallogic Storage Provider"   on all my HV hosts and left the HitKit installed. My replication has no errors and my VM backups now work without any issues.

    To un-register Dell under vssadmin list providers using the “eqlvss” command at a dos command prompt:

     C:\Program Files\EqualLogic\bin>eqlvss /unregserver

    If you ever want to re-register the provider

     C:\Program Files\EqualLogic\bin>eqlvss /regserver

     C:\Program Files\EqualLogic\bin>vssadmin list providers and you see Dell re-registered. 

_{Tuesday, December 16, 2014 6:15 PM}

This was an issue with the Equallogic Hit Kit Version - has been resolved in a subsequent release earlier in 2014