Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Question
Friday, November 18, 2016 7:18 PM | 1 vote
Ok I admit it. This one is driving me crazy ;)
1) I have two Server2016 (Standard) HyperV Clusters - Source01 and DR01. Both part of same forest/domain.
2) Both Source01 and DR01 clusters have a HyperV Replica Broker installed and replication settings configured for Kerberos, Port80 and allow from any server.
3) I can initiate VM Replica successfully in both directions using the FailoverClusterManager UI. It creates the replica, transfers the initial data and everything appears to work.
However....
I need to create the replica using Powershell, but when I try to do it using the Enable-VMReplication powershell command it fails with:
I must be doing something wrong as the UI works. Perhaps some remote permissions not enabled? Remote WMI? something? hopefully some rookie mistake...
1) I've turned the firewall of DR01 off (just in case, just until figure out this issue)
2) The hyperV virtual machine management service is running correctly on both Source01 and DR01
3) the HyperV console on Source01 can correctly connect to DR01 and view/manage the VMs in the console - and vice versa.
Anyone got any ideas? Or seen this behaviour before?
thanks
Joe.
All replies (45)
Friday, November 18, 2016 9:42 PM
the plot thickens...
disabling the replica broker on DR01 (destination cluster), changing the port, or making some other breaking change results in exactly the same error! So it must not be even getting far enough to check its actually there??!
I'm checking SPNs at this point and everything looks correct.
As a sanity check I just pointed towards a standalone Server2016 host (so no cluster replica broker) and the enable-vmreplication etc worked first time. replication starts and works fine.
repoint back to the destination cluster replica broker - back to the same error.
I'm open to anything at this point :) No suggestion too stupid :)
Cheers
Sunday, November 20, 2016 5:04 PM
the plot is now quicksand...
Looks like a fairly massive bug in Server2016 atm. Anyone looking to utilise HyperV replica in Server2016 at the moment should probably hang tight for a little big. I've had a paid support case open with MS since Tuesday but all I'm getting is "we don't know... don't have documentation yet" :| Good job this isn't an RTM product that has been available for a couple of months already I guess.
Basically none of the powershell commands related to interfacing with the a cluster replica broker seem to be able to talk to it. Remove the cluster replica broker and talk directly to a standalone host and everything works fine. I've also tested the same commands on a set of Server2012R2 hyperV cluster and that works fine as well.
To double check everything I've now got three Server2016 environments, each with a source and destination cluster - all three are failing with the exact same error.
long shot, but has *anyone* managed to get the cluster replica broker working with the powershell commands?
Cheers
J.
Monday, November 21, 2016 2:57 AM
Hi Joe,
I am trying to involve someone familiar with this topic to further look at this issue. There might be some time delay. Appreciate your patience.
Best Regards,
Leo
Please remember to mark the replies as answers if they help.
If you have feedback for TechNet Subscriber Support, contact [email protected].
Monday, November 21, 2016 11:19 AM
Thanks Leo :)
I can post/pm the support case number if that helps.
Regards
Joe
Monday, November 28, 2016 9:58 AM | 1 vote
still no news from the product team. I'm just in limbo here :|
Wednesday, May 17, 2017 2:29 AM
No, man. I am also by ur side in limbo...
I am trying to invoke powershell command between clustered replica broker and Hyper-V server 2016, and get the same error. I even try replica broken and MS server 2016 too, and all the windows version is related to 14393, which I think is a very new one. Both not working. So do you have any update?
Monday, September 4, 2017 8:33 PM
No Outcome?
Monday, October 30, 2017 2:18 PM
I'm having the exact same problem, anyone have an update?
Wednesday, November 29, 2017 12:59 PM
Having the exact same issue!
trying to automate the initial replication (enable + start replication) on fresh VM's in our DC (using orchestrator)
but this makes it impossible to do, so..enabling one by one manually atm :(
hope this gets fixed soon!
Thursday, February 22, 2018 8:11 PM
Bump - same problem here. Anyone got a fix for this?
I do it via the GUI, works fine. Do it PS, fails.
enable-VMReplication : Hyper-V encountered an error trying to access an object on computer 'replica_broker' because the object was not found. The object might have been deleted.
Verify that the Virtual Machine Management service on the computer is running.
I have 100's of VM's to replicate, doing by hand would be extremely ANNOYING!
Saturday, February 24, 2018 3:49 PM
Nobody?
Tuesday, February 27, 2018 2:23 PM
Opened a case with Microsoft at the time of my previous post
MS engineer confirmed our findings and that it is marked to be fixed in a future release...
so no, the only way is to do every VM manually, one at a time :( which is the suggested so-called "workaround" told to us by the MS Engineer
from my perspective this is an unacceptable solution/workaround, it is a hell of a job to enable replication for hundreds of VM's as you can imagine.
:(
Tuesday, February 27, 2018 11:45 PM
Same problem here. Infuriating, as that's the only way to enable Hyper-V Replica's with VHD Sets.
Wednesday, February 28, 2018 4:25 PM
I have opened a ticket with Microsoft as manual replication is not an option. Will update with findings.
Wednesday, February 28, 2018 7:27 PM
Thank you, this has been driving me crazy (thought it was my scripting!)
If shortcuts worked, then they wouldn't be shortcuts would they...
Thursday, March 1, 2018 9:32 PM | 1 vote
Thank you for sharing with us this information. If only microsoft would think to inform about known issues with it's products.
And are they kidding me with this ?! 1,5 year from release and still not fixed... Just painfully learned about this limitation.
Server 2012 r2 is end of mainstream support in 7 months and 2016 is still not ready for use...
First you can't take a checkpoint of new VHD Sets (vhds). Still not fixed from release. So no checkpoints and no backups for vhds.
Then promised VHD Set replication. Except it has a lot of asterisks and apparently can't be configured with gui (that's fine) but also not from powershell. And not documented at all, so not really working...
And let's not forget S2D named confusingly "storage spaces" - a new thing completely separate from previous storage space, but let's confuse people and make them think that old storage space will also get S2D new shiny features...
Anyway back to point.
Any idea what they meant by "Future release" ? A hotfix or will we have to wait for server 1803, 1809 or whatever next long term release will be called ?
edit: 2018-03-01
found another "workaround". I was trying Enable-VMReplication from 2016 hyper-v standalone to a 2016 cluster and failed missurebly.
However when I ran Enable-VMReplication on 2012 R2 host with " -ComputerName $srcserver -ReplicaServerName $dstserver" for the same combination of 2016 hosts then it worked.
So just configure replication of hyper-v 2016 hosts from 2012 R2 management machine.
Hope you all still have 2012 r2 available somewhere :)
Sunday, March 4, 2018 11:31 AM | 6 votes
Before invoking any replication commands, as a workaround on 2016 Server, you can use:
Remove-Module Hyper-V
Import-Module -Name Hyper-V -RequiredVersion 1.1
Best Regards,
Leon.
Monday, March 12, 2018 11:51 AM
I will have to try this from a 2012 server, I'm not prepared to run hyper-v module reinstall on a production cluster just yet.
Monday, March 12, 2018 12:11 PM
Hi, I can confirm that using Hyper-V module in version 1.1 works fine for replication - I set up replication from single host to cluster.
You do not need to reinstall module. It's loaded only for current powershell session, and version 1.1 is apparently also avalible in 2016 as I had it already avalible.
Check with:
get-module -Name "hyper-v" -ListAvailable
Tuesday, March 20, 2018 11:35 PM
I still have MS support working on this. 2 weeks now, they will still not accept its a bug.
:-(
When I load the PS module from 1.1, I get this. This is something I AM NOT doing on a PRODUCTION cluster!
PS C:\Windows\system32> install-module hyper-v -RequiredVersion 1.1
NuGet provider is required to continue
PowerShellGet requires NuGet provider version '2.8.5.201' or newer to interact with NuGet-based repositories. The NuGet
provider must be available in 'C:\Program Files\PackageManagement\ProviderAssemblies' or
'C:\Users\richard.smith\AppData\Local\PackageManagement\ProviderAssemblies'. You can also install the NuGet provider by
running 'Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force'. Do you want PowerShellGet to install
and import the NuGet provider now?
[Y] Yes [N] No [S] Suspend [?] Help (default is "Y"):
Wednesday, March 21, 2018 10:50 AM | 1 vote
no wonder, you are installing it, not loading to current session - there is a major difference.
You should:
Remove-Module Hyper-V
Import-Module -Name Hyper-V -RequiredVersion 1.1
Import - not install.
Import is different then installing. If you have doubts read help for both to know what they do.
Tuesday, June 12, 2018 8:22 AM
I have a MS premiere ticket open for this issue since March 20th. Nearly 3 months of investigation and running around in circles with MS still no progress.
They have confirmed its a bug though just trying to get it fixed.
Monday, July 2, 2018 8:01 AM
Finally, MS <g class="gr_ gr_15 gr-alert gr_gramm gr_inline_cards gr_run_anim Grammar multiReplace" data-gr-id="15" id="15">have</g> been able to reproduce the issue internally. Awaiting a fix!
Yay! Wonder which cumulative update it will appear in! 2018-07?
Thursday, July 19, 2018 8:14 AM
Hello,
I just want to add that we have same issue on Windows Server 2016 Standart in Hyper-v cluster.
We need to use Resume-VMReplication cmdlet and it fails:
resume-VMReplication : Hyper-V encountered an error trying to access an object on computer 'server01' because
the object was not found. The object might have been deleted, or you might not have permission to perform the task.
Verify that the Virtual Machine Management service on the computer is running. If the service is running, try to
perform the task again by using Run as Administrator.
At line:1 char:1
+ resume-VMReplication -VMName virtualMachine01
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (:) [Resume-VMReplication], VirtualizationException
+ FullyQualifiedErrorId : ObjectNotFound,Microsoft.HyperV.PowerShell.Commands.ResumeVMReplication
Microsoft support suggested to rebuild WMI:
Open CMD as administrator:
- mofcomp %systemroot%\system32\WindowsVirtualization.v2.mof
- mofcomp %systemroot%\system32\en-US\WindowsVirtualization.v2.mfl
But that do not worked for me.
Tuesday, September 11, 2018 6:44 AM
Hi, we are getting the same thing, no matter if we user PowerShell or the Cluster GUI. Nothing to add to help, just wanted to say the same is happening to us too.
Phil
Phil
Wednesday, January 23, 2019 4:11 PM
I have the same issue at the moment but with WS 2019. I have a ticket open with MS and (as some of you have also experienced) they have acknowledged that it's a bug. They suggested using module version workaround but it made no difference and fails with the same error.
Oddly, and I'm not sure if any of you guys have noticed the next bit. Replication fails for all *new* VMs and VMs replicated from another server/cluster. But if I replicate away from the new cluster, fail over and reverse replication, it works without issue.
When replicating a VM for the first time, it creates the VM on the new cluster (so cert auth isn't to blame), but it's the initial seeding that fails. That had me thinking it was an issue with storage or permissions, but it does create the new VHD also.
Then I tried sending the initial replication of a brand new VM using external media, but it failed with the exact same error. So it looks like the failure occurs at source.
Wednesday, January 23, 2019 4:32 PM
Incidentally, I did get the above wizard to finish on a subsequent attempt and it exported the replication data to the D drive as show. I copied that to the cluster and it was accepted on that side. Great, thinks looked good and the 'last replication' time was up to date.
Unfortunately, each subsequent replication attempt for this VM failed with the same error.
> Hyper-V could not find the virtual machine 'TestVMFromRep' on the Hyper-V Replica Broker. (Virtual machine ID 4FEB777E-8D89-4DBC-B58C-ACDB77A5D3EF)
Monday, February 4, 2019 9:44 AM
Hi all,
same issues here with ws2019 17744 insider build.
anybody know if this error occurs also in the stable version of ws2019?
Workaround with the 1.1 module does not work.
GUI does not work.
regards
Tuesday, February 5, 2019 2:15 PM
Mine was on the stable build. I was asked to fill in a business impact questionnaire in order to help push for a fix from the development team. The more users / bigger the impact, the sooner it will be resolved I suppose.
Friday, February 22, 2019 7:20 PM
Hi JoeGough,
Please try the following procedure:
On primary server run the following commands to reinstall the Hyper-V module.
Remove-Module Hyper-V
Import-Module -Name Hyper-V -RequiredVersion 1.1
There is a error on Hyper-V module that prevents the command to work accordingly. I've tested the command in lab and production environment and its works successfully!
Thanks,
Tuesday, March 19, 2019 11:04 AM
Hi,
has anyone some News with the error, i have the same with two 2019er Cluster.
The Workaround with the hyper-v modules 1.1 does not work for me.
Thanks, Heinrich
Wednesday, March 20, 2019 8:53 AM
Same with Win2019:
> Hyper-V could not find the virtual machine 'TestVMFromRep' on the Hyper-V Replica Broker. (Virtual machine ID 4FEB777E-8D89-4DBC-B58C-ACDB77A5D3EF)
Wednesday, March 20, 2019 8:54 AM
Confirm the same issue with Win2019 Cluster.
1. Win2019 Cluster CAN'T receive inbound replication, but can send replica from Hyper-V Replica Broker to any standalone Hyper-V server.
2. Cluster to cluster replication DOESN'T WORK.
As Keiran Kenny wrote, I get the same error:
> Hyper-V could not find the virtual machine 'TestVMFromRep' on the Hyper-V Replica Broker. (Virtual machine ID 4FEB777E-8D89-4DBC-B58C-ACDB77A5D3EF)
Wednesday, March 20, 2019 12:53 PM
this workaround doesnt work for me. @Bruno H. Secchi
i got the same problem just like Aram Dzhulfalakyan & Heinrich Klassen
Outgoing Replication to a Hyper-V Host does work from the 2019 Cluster.
Old Cluster Server 2012R2 trying to Replicate to the New Cluster Server 2019.
The VM is getting created on the Cluster. The VHD is also created with the standard 4096KB Size..
VM-ID is correct, everything seems fine. but the primary replicationserver tells me:
ErrorCode: 32370
The VM was not found on the Hyper-V-Replicabroker.
Any new Idea??
Wednesday, March 20, 2019 1:21 PM
Not a productive answer, but for us we had to buy a 3rd party replication tool to get over this. We were moving from an old cluster to a new cluster - with new hosts/san etc. I think this is 100% crazy that it is 100% reproducible yet MS do nothing to fix this.
Phil
Wednesday, March 20, 2019 9:17 PM
replying to this thread as I'm also having this issue on a client site that setup up a new 2019 cluster to replace there aging / failing older server 2012R2 cluster
Tuesday, March 26, 2019 9:03 AM
I know it doesn't really help anyone but I have the exact same issue since upgrading from 2012 R2 to a 2019 cluster. Tried everything but always get a 'Hyper V could not find the virtual machine on the replica server' It creates the VM but never starts the initial replication
Tuesday, March 26, 2019 10:20 PM
Right So while we all wait to see if MS comes up with a fix for this issues, maybe their is one and it just hasn't made it to this forum, I did find a work around that seems to be working for me. I have been able to get 2 VM's moved over so far.
In my case I'm just trying to move VM's off old servers to new hardware running 2019, So this may not work for everyone.
Veeam has a free version of it's Backup and replication suite. It does allow you to do replication and permanent fail over. tho it takes a few steps. you can find out more / download this version at the link below.
https://www.veeam.com/virtual-machine-backup-solution-free.html
Like I said, this may not work for everyone, It works for me, I was able to move 2 VM's over with out much issue.
Thursday, April 4, 2019 8:12 AM
With your ticket with Microsoft did they ever get back to you on this. Or give you any ETA on when they would have it fixed?
Friday, April 19, 2019 10:09 PM | 2 votes
This may sound silly . . .but make sure both the source and destination VMs are in the Root of the CSV or Storage you are using. We have had this issue after our upgrade and this one change made all of the replication work. Is this awesome . . not at all . . but working is better than having a structure.
To expand a little. Once your initial Replication fails . . move the vm on the destination to a root level folder, create it and move the new copy. Then restart the replication and it should just go.
Wednesday, April 24, 2019 2:10 PM
Wow thanks that does fix it. Definitely not ideal but it is a work around! Thanks
Thursday, May 2, 2019 2:27 PM
Hi Richard,
i've the same Problem with the replication between two Hyper-V 2019 Cluster.
i've try your solution with move the vm ( i use the function in hyper-v move Storage ) to the root of CSV,
but i've the same issue.
What do i wrong or could you tell me the steps what i must do.
Thanks, Heinrich
Sunday, June 2, 2019 6:30 PM
We have a VM scattered across multiple volumes due to other limitations. Does this solution require the whole VM to be on one volume in the root? I have tried just moving the machine config and this doesn't work. Thanks in advance...
Thursday, July 4, 2019 9:40 PM | 1 vote
A fix for this issue is planned to release in September 2019 update package for Windows Server 2016.
Thursday, September 12, 2019 6:12 AM
Thank you! This worked for me also!
Regards,
Jose Angel Rivera
IT Consultant