Production Checkpoint Failure

Article
2019-07-30

Question

_{Tuesday, July 30, 2019 4:40 PM}

Hyper-V Host is Server 2016, VM is Server 2016, when attempting a production checkpoint I get the following:

'VM' could not initiate a checkpoint operation: Element not found. (0x80070490). (Virtual machine ID E397EC81-9B8C-435F-95C9-F9950638BF28)

Production Checkpoint for VM 'VM' failed in the guest VSS Integration Component with error Element not found. (0x80070490). If you continue to see this problem please use regular checkpoints instead (Virtual machine ID E397EC81-9B8C-435F-95C9-F9950638BF28)

I have two Server 2016 Hyper-V hosts, and each one Server 2016 VMs, on one host my backup software which requires production checkpoints for application aware processing works just fine, on the other I get this error.

The difference is that prior to installing my backup software, on the host with the error I created a production checkpoint, then deleted the checkpoint subtree. Even after removing my backup software I still cannot create a production checkpoint, I get the same error, on the host that works just fine, I never did the production checkpoint / subtree delete.

The production checkpoint / subtree delete seems to be what causes the issue, after testing on test VMs.

All VSS writers on host and VM are stable and not in error, reboot of the VM and/or host do not correct the issue.

All replies (8)

_{Wednesday, July 31, 2019 2:11 AM | 1 vote}

Hi,

Good Day!

1.Please try to uncheck the “Backup” in VM integration service.

2.Please make sure that the integration service has already been update to the latest version.

3.Can you create standard checkpoint?

4.Please check if there is any error message in Event Viewer.

Please let me know if you have any question.

Cheers,

Daniel

Please remember to mark the replies as answers if they help.
If you have feedback for TechNet Subscriber Support, contact [email protected].

_{Wednesday, July 31, 2019 6:19 AM}

Unchecking Backup in the integration services doesn't change anything (and wouldn't allow for application aware processing which I'll need).

Both the VM and the Host have been updated fully with Windows Update. Since they are both Server 2016 that is the only way to update integration services that I'm aware of.

I can create a standard checkpoint, I can also create a production checkpoint if the VM is powered off. Unfortunately I'm going to require the ability to create production checkpoints while the VM is running for application aware processing for backups.

There are no errors in the Event View on the VM, and the only relevant entries I can find on the host are in the Hyper-V Worker Admin log and they are:

'VM' could not initiate a checkpoint operation: Element not found. (0x80070490). (Virtual machine ID E397EC81-9B8C-435F-95C9-F9950638BF28)

and

All VSS writers on Host and VM are stable and not in an error state.

_{Wednesday, July 31, 2019 7:21 AM}

update：

Please run this PowerShell command on your host and upload your screenshot :

Get-VMIntegrationService -VMName {VMNAME} | where Name -eq VSS

please run cmd command to check your integration service:

REG QUERY "HKLM\Software\Microsoft\Virtual Machine\Auto" /v IntegrationServicesVersion

/en-us/windows-server/virtualization/hyper-v/manage/manage-hyper-v-integration-services

Regards,

Daniel

Please remember to mark the replies as answers if they help.
If you have feedback for TechNet Subscriber Support, contact [email protected].

_{Wednesday, July 31, 2019 4:44 PM}

I can post screenshots when my account gets verified and I'm allowed, but just to get the answer for you:

From the host:

PS C:\Windows\system32> Get-VMIntegrationService -VMName CITG2K16 | where Name -eq VSS

VMName   Name Enabled PrimaryStatusDescription SecondaryStatusDescription

CITG2K16 VSS True    OK

From the VM:

Windows PowerShell
Copyright (C) 2016 Microsoft Corporation. All rights reserved.

PS C:\Windows\system32> REG Query "HKLM\Software\Microsoft\Virtual Machine\Auto" /v IntegrationServicesVersion

HKEY_LOCAL_MACHINE\Software\Microsoft\Virtual Machine\Auto
IntegrationServicesVersion REG_SZ 10.0.14393

PS C:\Windows\system32>

_{Thursday, August 1, 2019 1:40 AM}

It seems that integration services are the latest and VSS of VM is no problem, which is really strange.

Can we use the current VHD to rebuild a VM to see if there are still problems?

Regards,

Daniel

Please remember to mark the replies as answers if they help.
If you have feedback for TechNet Subscriber Support, contact [email protected].

_{Thursday, August 1, 2019 5:56 AM}

I shut down my original VM, create a new VM and attached my VHD to it and booted the VM.

Unfortunately attempting a production checkpoint still produces the same error:

and

'VM' could not initiate a checkpoint operation: Element not found. (0x80070490). (Virtual machine ID F5B0472F-0BF3-4C04-A1FE-C4109C226496)

_{Friday, August 2, 2019 1:21 AM | 1 vote}

Something in here might be helpful:

http://backupchain.com/i/hyper-v-backup-error-could-not-initiate-a-checkpoint-operation-element-not-found-0x80070490

Regards,

Daniel

Please remember to mark the replies as answers if they help.
If you have feedback for TechNet Subscriber Support, contact [email protected].

_{Friday, August 2, 2019 6:41 AM}

I had found that page before I made my thread here, but, I repeated the steps there anyway.

I redid the security settings, unchecked the Backup integration, powered down my VM, restarted my host, brought everything back up and then did a Windows backup, after I did the Windows backup, I rechecked Backup integration and did another Windows backup. The windows backup completed with warnings (which was the backup VSS integration), I also attempted another production checkpoint which failed. The errors I got are:

For the snapshot failure:
'VM' could not initiate a checkpoint operation: Element not found. (0x80070490). (Virtual machine ID E397EC81-9B8C-435F-95C9-F9950638BF28)

However, for the Windows backup with the backup component checked I got:

Log Name:      Microsoft-Windows-Backup
Source:        Microsoft-Windows-Backup
Date:          8/2/2019 1:42:41 AM
Event ID:      7
Task Category: None
Level:         Warning
Keywords:
User:          SYSTEM
Computer:      HOST.FQDN.FQDN.net
Description:
The backup operation that started at '‎2019‎-‎08‎-‎02T05:03:03.432289600Z' has completed with errors. Please review the event details for a solution, and then rerun the backup operation once the issue is resolved.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
    <Provider Name="Microsoft-Windows-Backup" Guid="{1DB28F2E-8F80-4027-8C5A-A11F7F10F62D}" />
    <EventID>7</EventID>
    <Version>3</Version>
    <Level>3</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x4000000000000000</Keywords>
    <TimeCreated SystemTime="2019-08-02T05:42:41.855150200Z" />
    <EventRecordID>9</EventRecordID>
    <Correlation />
    <Execution ProcessID="3128" ThreadID="2920" />
    <Channel>Microsoft-Windows-Backup</Channel>
    <Computer>Host.FQDN.FQDN.net</Computer>
    <Security UserID="S-1-5-18" />
</System>
<EventData>
    <Data Name="BackupTemplateID">{E94040C2-0625-4561-8D68-E2935797283C}</Data>
    <Data Name="HRESULT">0x0</Data>
    <Data Name="DetailedHRESULT">0x0</Data>
    <Data Name="ErrorMessage">%%0</Data>
    <Data Name="BackupState">14</Data>
    <Data Name="BackupTime">2019-08-02T05:03:03.432289600Z</Data>
    <Data Name="BackupTarget">\192.168.16.75\temp</Data>
    <Data Name="NumOfVolumes">5</Data>
    <Data Name="VolumesInfo"><VolumeInfo><VolumeInfoItem Name="Esp" OriginalAccessPath="" State="14" HResult="0" DetailedHResult="0" PreviousState="17" IsCritical="1" IsIncremental="0" BlockLevel="1" HasFiles="0" HasSystemState="0" IsCompacted="1" IsPruned="0" IsRecreateVhd="2" FullBackupReason="3" DataTransferred="100663296" NumUnreadableBytes="0" TotalSize="100663296" TotalNoOfFiles="0" Flags="523" BackupTypeDetermined="1" SSBTotalNoOfFiles="0" SSBTotalSizeOnDisk="0" /><VolumeInfoItem Name="C:" OriginalAccessPath="C:" State="14" HResult="0" DetailedHResult="0" PreviousState="9" IsCritical="1" IsIncremental="0" BlockLevel="1" HasFiles="0" HasSystemState="1" IsCompacted="0" IsPruned="0" IsRecreateVhd="0" FullBackupReason="10" DataTransferred="20100939776" NumUnreadableBytes="0" TotalSize="20100939776" TotalNoOfFiles="0" Flags="5672" BackupTypeDetermined="1" SSBTotalNoOfFiles="0" SSBTotalSizeOnDisk="0" /><VolumeInfoItem Name="Recovery" OriginalAccessPath="" State="14" HResult="0" DetailedHResult="0" PreviousState="9" IsCritical="1" IsIncremental="0" BlockLevel="1" HasFiles="0" HasSystemState="1" IsCompacted="0" IsPruned="0" IsRecreateVhd="0" FullBackupReason="10" DataTransferred="341049344" NumUnreadableBytes="0" TotalSize="341049344" TotalNoOfFiles="0" Flags="552" BackupTypeDetermined="1" SSBTotalNoOfFiles="0" SSBTotalSizeOnDisk="0" /><VolumeInfoItem Name="E:" OriginalAccessPath="E:" State="14" HResult="0" DetailedHResult="0" PreviousState="9" IsCritical="0" IsIncremental="0" BlockLevel="1" HasFiles="0" HasSystemState="0" IsCompacted="0" IsPruned="0" IsRecreateVhd="0" FullBackupReason="10" DataTransferred="212068335616" NumUnreadableBytes="0" TotalSize="212068335616" TotalNoOfFiles="0" Flags="8" BackupTypeDetermined="1" SSBTotalNoOfFiles="0" SSBTotalSizeOnDisk="0" /><VolumeInfoItem Name="F:" OriginalAccessPath="F:" State="14" HResult="0" DetailedHResult="0" PreviousState="9" IsCritical="0" IsIncremental="1" BlockLevel="0" HasFiles="1" HasSystemState="0" IsCompacted="0" IsPruned="0" IsRecreateVhd="0" FullBackupReason="0" DataTransferred="4624396" NumUnreadableBytes="0" TotalSize="4628492" TotalNoOfFiles="15" Flags="6228" BackupTypeDetermined="1" SSBTotalNoOfFiles="0" SSBTotalSizeOnDisk="0" /></VolumeInfo></Data>
    <Data Name="SourceSnapStartTime">2019-08-02T05:03:02.103845600Z</Data>
    <Data Name="SourceSnapEndTime">2019-08-02T05:03:41.423493700Z</Data>
    <Data Name="PrepareBackupStartTime"><TimesList><Time Time="2019-08-02T05:05:12.154Z" /><Time Time="2019-08-02T05:05:27.219Z" /><Time Time="2019-08-02T05:08:38.071Z" /><Time Time="2019-08-02T05:08:42.164Z" /><Time Time="2019-08-02T05:42:21.429Z" /></TimesList></Data>
    <Data Name="PrepareBackupEndTime"><TimesList><Time Time="2019-08-02T05:05:12.154Z" /><Time Time="2019-08-02T05:05:27.422Z" /><Time Time="2019-08-02T05:08:38.071Z" /><Time Time="2019-08-02T05:08:42.195Z" /><Time Time="2019-08-02T05:42:22.273Z" /></TimesList></Data>
    <Data Name="BackupWriteStartTime"><TimesList><Time Time="2019-08-02T05:05:12.154Z" /><Time Time="2019-08-02T05:05:27.422Z" /><Time Time="2019-08-02T05:08:38.071Z" /><Time Time="2019-08-02T05:08:42.195Z" /><Time Time="2019-08-02T05:42:22.273Z" /></TimesList></Data>
    <Data Name="BackupWriteEndTime"><TimesList><Time Time="2019-08-02T05:05:18.960Z" /><Time Time="2019-08-02T05:08:37.400Z" /><Time Time="2019-08-02T05:08:41.742Z" /><Time Time="2019-08-02T05:42:00.701Z" /><Time Time="2019-08-02T05:42:22.617Z" /></TimesList></Data>
    <Data Name="TargetSnapStartTime">1601-01-01T00:00:00.000000000Z</Data>
    <Data Name="TargetSnapEndTime">1601-01-01T00:00:00.000000000Z</Data>
    <Data Name="DVDFormatStartTime"><TimesList></TimesList></Data>
    <Data Name="DVDFormatEndTime"><TimesList></TimesList></Data>
    <Data Name="MediaVerifyStartTime"><TimesList></TimesList></Data>
    <Data Name="MediaVerifyEndTime"><TimesList></TimesList></Data>
    <Data Name="BackupPreviousState">19</Data>
    <Data Name="ComponentStatus"><ComponentStatus><ComponentStatusItem Name="E397EC81-9B8C-435F-95C9-F9950638BF28" LogicalPath="" Caption="Online\VM" AppId="HyperV" WriterId="{66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}" InstanceId="{f7cfb7ba-697d-4598-9375-110844a810e8}" ConsistencyCheckResult="-2139618955" ConsistencyCheckResultDetailed="-2147212300"/></ComponentStatus></Data>
    <Data Name="ComponentInfo"><ComponentInfo><ComponentInfoItem WriterId="{66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}" InstanceId="{f7cfb7ba-697d-4598-9375-110844a810e8}" Name="E397EC81-9B8C-435F-95C9-F9950638BF28" LogicalPath="" Caption="Online\VM" AppName="HyperV" hResult="-2139618955" hDetailedResult="-2147212300" IsFullBackup="1" FullBackupReason="0" Size="0" DataTransferred="0" TotalNoOfFiles="0" /><ComponentInfoItem WriterId="{66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}" InstanceId="{f7cfb7ba-697d-4598-9375-110844a810e8}" Name="Host Component" LogicalPath="" Caption="Host Component" AppName="HyperV" hResult="0" hDetailedResult="0" IsFullBackup="1" FullBackupReason="0" Size="398475" DataTransferred="0" TotalNoOfFiles="0" /></ComponentInfo></Data>
    <Data Name="SSBEnumerateStartTime">1601-01-01T00:00:00.000000000Z</Data>
    <Data Name="SSBEnumerateEndTime">1601-01-01T00:00:00.000000000Z</Data>
    <Data Name="SSBVhdCreationStartTime">1601-01-01T00:00:00.000000000Z</Data>
    <Data Name="SSBVhdCreationEndTime">1601-01-01T00:00:00.000000000Z</Data>
    <Data Name="SSBBackupStartTime">1601-01-01T00:00:00.000000000Z</Data>
    <Data Name="SSBBackupEndTime">1601-01-01T00:00:00.000000000Z</Data>
    <Data Name="SystemStateBackup"><SystemState IsPresent="1" HResult="0" DetailedHResult="0" /></Data>
    <Data Name="BMR"><BMR IsPresent="1" HResult="0" DetailedHResult="0" /></Data>
    <Data Name="VssFullBackup">true</Data>
    <Data Name="UserInputBMR">true</Data>
    <Data Name="UserInputSSB">true</Data>
    <Data Name="BackupSuccessLogPath">C:\Windows\Logs\WindowsServerBackup\Backup-02-08-2019_05-03-03.log</Data>
    <Data Name="BackupFailureLogPath">C:\Windows\Logs\WindowsServerBackup\Backup_Error-02-08-2019_05-03-03.log</Data>
    <Data Name="EnumerateBackupStartTime"><TimesList><Time Time="1601-01-01T00:00:00.000Z" /><Time Time="1601-01-01T00:00:00.000Z" /><Time Time="1601-01-01T00:00:00.000Z" /><Time Time="1601-01-01T00:00:00.000Z" /><Time Time="2019-08-02T05:42:20.414Z" /></TimesList></Data>
    <Data Name="EnumerateBackupEndTime"><TimesList><Time Time="1601-01-01T00:00:00.000Z" /><Time Time="1601-01-01T00:00:00.000Z" /><Time Time="1601-01-01T00:00:00.000Z" /><Time Time="1601-01-01T00:00:00.000Z" /><Time Time="2019-08-02T05:42:21.429Z" /></TimesList></Data>
    <Data Name="PruneBackupStartTime"><TimesList><Time Time="1601-01-01T00:00:00.000Z" /><Time Time="1601-01-01T00:00:00.000Z" /><Time Time="1601-01-01T00:00:00.000Z" /><Time Time="1601-01-01T00:00:00.000Z" /><Time Time="1601-01-01T00:00:00.000Z" /></TimesList></Data>
    <Data Name="PruneBackupEndTime"><TimesList><Time Time="1601-01-01T00:00:00.000Z" /><Time Time="1601-01-01T00:00:00.000Z" /><Time Time="1601-01-01T00:00:00.000Z" /><Time Time="1601-01-01T00:00:00.000Z" /><Time Time="1601-01-01T00:00:00.000Z" /></TimesList></Data>
    <Data Name="BackupFlags">0xc</Data>
    <Data Name="ComponentInfoSummary"><ComponentInfoSummary ComponentInfoArrayPresent="1" TotalComponents="2" SucceededComponents="1" /></Data>
</EventData>
</Event>

Also:
Writer Microsoft Hyper-V VSS Writer experienced some error during snapshot creation. More info: .

Log Name:      Application
Source:        SPP
Date:          8/2/2019 1:03:38 AM
Event ID:      16387
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      Host.FQDN.FQDN.net
Description:
Writer Microsoft Hyper-V VSS Writer experienced some error during snapshot creation. More info: .
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
    <Provider Name="SPP" />
    <EventID Qualifiers="0">16387</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2019-08-02T05:03:38.942365800Z" />
    <EventRecordID>1924</EventRecordID>
    <Channel>Application</Channel>
    <Computer>Host.FQDN.FQDN.net</Computer>
    <Security />
</System>
<EventData>
    <Data>Microsoft Hyper-V VSS Writer</Data>
    <Data>
    </Data>
    <Data>The writer experienced a partial failure. Check the component level error state for more information. (0x80042336)</Data>
    <Data>Unspecified error (0x80004005)</Data>
    <Binary>01000000B8120000991200000000000042BEB7C511CAC619E59C92030000000000000000</Binary>
</EventData>
</Event>

I also found one of my VSS writers in a stable state but with an error:

Writer name: 'Microsoft Hyper-V VSS Writer'
   Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
   Writer Instance Id: {f7cfb7ba-697d-4598-9375-110844a810e8}
   State: [1] Stable
   Last error: Unexpected error

Share via

Production Checkpoint Failure

Question

All replies (8)

Additional resources