Exchange 2016 Mailbox Migration - Stalledtotarget_disklatency

Question

_{Wednesday, June 14, 2017 8:57 PM}

I'm hoping someone might be able to lead me to a viable solution. I've done a ton of research and tried different settings, Disable MRS registry, increasing latency in the config file, pretty much anything you can find to speed up a mailbox migration and I keep getting stuck at this message.  Here's the setup:

VMware ESXi 6.5d

Exchange 2016 VM:

Paravirtual SCSI

64GB Ram (All Reserved)

4x vCPU's (E5-2640 v3) Assigned

Dell EqualLogic PS6610X (all 10K SAS drives in a RAID 6)

VMFS 6 Datastores, Dedicated to Exchange 2016 using VMDK disks (Not RDM)

Right now the Exchange Server 2016 is NOT in production, and I moved only my mailbox (600 MB Primary On dedicated D Drive, 800 MB Archive on Dedicated E Drive) from Exchange 2013 to 2016. I decided I wanted to break out my mailboxes to multiple databases and organize it a little differently. Two days ago I then migrated my mailbox again from 1 database D drive, to another one I added, F drive, brand new, blank database. For 2 days the migration is still running and every time I check it keeps saying "Stalledtotarget_disklatency" but when I check the Disk IO it's at <1ms and the IOPS is averaging 3 IOPS. So basically the server is sitting there doing nothing. I checked the EqualLogic box and the Disk IO/IOPS matches what I'm seeing in the VM too. I cannot figure out what Exchange is doing to THINK the latency is too high. I even tried increasing it for the edb in the config file to the max value of "1000" and it made no difference. It's slowly migrating still, up to 47%, moving about 1% an hour or so. I'm kind of stumped at this point. The VM runs BLAZING fast performance wise, especially since I am the only mailbox on the server, but I am concerned when I start mass migrating from 2013 it's going to take like 5 years to migrate. Any pointers or things to look for would be appreciated. I have a gut feeling something is causing it to falsely think there is a disklatency problem, but there really isn't. Other things of note, power settings are set to High Performance on both the ESX box and the VM. Accessing the mailbox from OWA or Outlook 2016 is blazing fast. E-mail flows fine, not seeing any errors in the event log. Open to any suggestions or items to look for, quite frustrating. Thanks for reading.

All replies (4)

_{Thursday, June 15, 2017 5:18 AM ✅Answered | 1 vote}

Hi,

This might happen if there are any issues in the disk performance causing disk latency, the response time from the source is getting high and the migration batches are getting timed out. This delays the movement of the mailboxes.

• Remove the old move request: Remove-MoveRequest -Identity “[email protected]”
• New a move request with highest priority: New-MoveRequest -Identity Mailbox -TargetDatabase “DB Name” -BatchName Test -Priority Highest

Then check the result.

We can also move the system mailbox from Exchange server 2013 to Exchange server 2016, then move the mailboxes, check if it works or not.

If it still doesn’t work, please check as following steps and post out the results in detail:

• Get-MoveRequestStatistics -Identity [email protected] -IncludeReport | fl report
• Get-MoveRequestStatistics  -Identity [email protected] | Select DisplayName,StatusDetail,PercentComplete

If the Exchange server 2016 VM on another physical server, please check the performance log of the disk as below:

Hope it helps.

Regards,

Jason Chao

Please remember to mark the replies as answers if they help.
If you have feedback for TechNet Subscriber Support, contact [email protected].

_{Thursday, June 15, 2017 12:39 PM}

Jason,

Thanks for the quick reply. I checked the Perfmon logs and they are even lower than the screenshot you provided. The first thing I did was run it with "-Priority Highest" and it moved the mailbox in less than 30 minutes. Much better than the 3 days it was taking and only 52% done. I guess when I move my other 3500 mailboxes from 2013 to 2016 I will just have to stage and script it so each one processes with the "-Priority Highest" flag. Seems silly to have to do this by default, but I'm sure there are reasons for Microsoft's madness if it was a very large enterprise environment. Thanks for the assist.

Don

_{Friday, June 16, 2017 2:20 AM}

Welcome! Glad to hear it works.

Regards,

Jason Chao

Please remember to mark the replies as answers if they help.
If you have feedback for TechNet Subscriber Support, contact [email protected].

_{Friday, June 16, 2017 12:30 PM}

Jason,

So I got it to happen again, however, it only happens now if I try to migrate TWO mailboxes to the SAME database at the same time. It still doesn't run too bad, but my calculations show that if I migrate two mailboxes to the same database it will be about 20% slower, than if I moved 1 mailbox, waited for it to finish, and then immediately processed the next mailbox. I did get around this though by making a CSV file from their existing mailbox location with 4 fields and wrote a looped powershell script that seems to be working great. It's certainly not the most efficient but for a quick whip up it's working for me as a test. Now that I know it will work, I'll just refine it to be more dynamic. I had to add some Start-Sleeps in there because my VM is so fast it seemed some of the variables weren't sticking before the next command processed:

CSV Headers: ID,Identity,TargetDatabase,TargetArchiveDatabase

$database = "<database name>"
$file = "C:\<csv file location>"
$record = 0
$loopstatus = 1
$csv = Import-csv $file
$csvmax = Import-csv $file | Measure-Object | Select -ExpandProperty Count
While($loopstatus -eq 1) {
$status = Get-MoveRequest -TargetDatabase $database | Where {($_.Status -eq "InProgress" -or $_.Status -eq "Queued")} | Select -ExpandProperty Status
Set-MailboxDatabase $database -IndexEnabled $false
IF($status) {
Start-Sleep -s 10
$percent = Get-MoveRequestStatistics -Identity $identity | Select -ExpandProperty PercentComplete
Write-Host "$identity Mailbox Move # $record of $csvmax In Progress, Please Wait... $percent% Complete."
Start-Sleep -s 80
} Else {
}
IF($record -eq $csvmax) {
$record = "9999"
$loopstatus = 2
Set-MailboxDatabase $database -IndexEnabled $true
Write-Host "Mailbox Moves Completed Successfully!"
} Else {
}
IF(($record -gt 0) -and (!$status)) {
Set-MailboxMessageConfiguration $identity -IsReplyAllTheDefaultResponse $false -EA SilentlyContinue
Start-Sleep 10
Get-MoveRequest -TargetDatabase $database | Where {($_.Status -eq "Completed")} | Remove-MoveRequest -Confirm:$false
Start-Sleep -s 20
} Else {
}
IF(($record -le $csvmax) -and (!$status)) {
Write-Host "Checking Mailbox Move Completion Status ..."
Start-Sleep 30
$record = $record + 1
} Else {
}
IF(($record -le $csvmax) -and (!$status)) {
Start-Sleep -s 10
$identity = $csv | where {$_.ID -eq $record} | select -ExpandProperty Identity -EA SilentlyContinue
$tdb = $csv | where {$_.ID -eq $record} | select -ExpandProperty TargetDatabase -EA SilentlyContinue
$tadb = $csv | where {$_.ID -eq $record} | select -ExpandProperty TargetArchiveDatabase -EA SilentlyContinue
$alias = Get-Mailbox -Identity $identity | Select -ExpandProperty Alias -EA SilentlyContinue
Start-Sleep -s 10
New-MoveRequest -Identity $identity -TargetDatabase $tdb -ArchiveTargetDatabase $tadb -BatchName $alias -Priority Highest
Write-Host "Starting Mailbox Move for $alias, Please wait..."
Start-Sleep -s 200
} Else {
}
}