Share via

VSS error on backup with two Exchange 2016 in DAG "[VSS_WS_FAILED_AT_BACKUP_COMPLETE]. Error code: [0x800423f3]."

  • Article

Question

Friday, November 29, 2019 8:40 AM

Hello everybody,

this are my two exchange servers:

EX1: Exchange 2016 CU13 Version 15.1 (Build 1779.2) on Windows Server 2016 Version 1607 (Build 14393.3326)
EX2: Exchange 2016 CU13 Version 15.1 (Build 1779.2) on Windows Server 2016 Version 1607 (Build 14393.3326)

Name                                          Status          CopyQueue ReplayQueue LastInspectedLogTime   ContentIndex
                                                              Length    Length                             State
                                                       
EX2_DB1\EX2                                   Mounted         0         0                                  Healthy
EX1_DB1\EX2                                   Healthy         0         0           29.11.2019 09:09:54    Healthy
EX_DB1_10GB\EX2                               Healthy         0         0           29.11.2019 09:07:39    Healthy
EX_PF_DB1\EX2                                 Mounted         0         0                                  Healthy

Name                           Server          Recovery        ReplicationType
                                            
EX1_DB1                        EX1             False           Remote
EX2_DB1                        EX2             False           Remote
EX_DB1_10GB                    EX1             False           Remote
EX_PF_DB1                      EX2             False           Remote

We use Veeam for backups. All servers are in one backup job. The error only occurs  intermittent on ex1.

Backups with warning contain only this message:

25.11.2019 21:19:23 :: VSS: Backup job failed.
Cannot notify writers about the 'BACKUP FINISH' event.
A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{f20775fd-7aba-42f8-92d3-b80a17a8ce74}]. Writer's state: [VSS_WS_FAILED_AT_BACKUP_COMPLETE]. Error code: [0x800423f3]. 

When the backup starts "MSExchangeRepl" event id 2140 and 2026are logged:

"MSExchangeRepl" event id 2140
The Microsoft Exchange Replication service VSS Writer encountered an exception in function Microsoft::Exchange::Cluster::ReplicaVssWriter::CReplicaVssWriterInterop::PrepareSnapshot. HResult -2147467259. Exception Microsoft.Mapi.MapiExceptionBackupInProgress: MapiExceptionBackupInProgress: Unable execute Prepare on snapshot. (hr=0x80004005, ec=1293)
Diagnostic context:
    Lid: 1494    Remote Context Beg
    Lid: 33832   StoreEc: 0x50D     
    Lid: 42621   StoreEc: 0x50D     
    Lid: 10786   dwParam: 0x0        Msg: 15.01.1779.007:EX1:14f782c4-aadf-458e-bb8e-68553d8a0320
    Lid: 51578   Guid: 14f782c4-aadf-458e-bb8e-68553d8a0320
    Lid: 1750    Remote Context End
    Lid: 59216  
    Lid: 34640   StoreEc: 0x50D     
    Lid: 57114  
    Lid: 40730   StoreEc: 0x50D     
   bei Microsoft.Mapi.MapiExceptionHelper.InternalThrowIfErrorOrWarning(String message, Int32 hresult, Boolean allowWarnings, Int32 ec, DiagnosticContext diagCtx, Exception innerException)
   bei Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, IExInterface iUnknown, Exception innerException)
   bei Microsoft.Mapi.ExRpcAdmin.ProcessSnapshotOperation(Guid mdbGuid, SnapshotOperationCode operationCode, UInt32 flags)
   bei Microsoft.Exchange.Cluster.Replay.StoreRpcController.<>c__DisplayClass29_0.<SnapshotPrepare>b__0()
   bei Microsoft.Exchange.Cluster.Shared.SafeRefCountedTimeoutWrapper.<>c__DisplayClass21_0.<ProtectedCallWithTimeout>b__0()
   bei Microsoft.Exchange.Data.HA.InvokeWithTimeout.Invoke(Action invokableAction, Action foregroundAction, TimeSpan invokeTimeout, Boolean sendWatsonReportNoThrow, Object cancelEvent)
   bei Microsoft.Exchange.Cluster.Shared.SafeRefCountedTimeoutWrapper.ProtectedCallWithTimeout(String operationName, TimeSpan timeout, Action operation)
   bei Microsoft.Exchange.Cluster.ReplicaVssWriter.CReplicaVssWriterInterop.SnapshotPrepare(Guid dbGuid, UInt32 flags)
   bei Microsoft.Exchange.Cluster.ReplicaVssWriter.CReplicaVssWriterInterop.PrepareSnapshot().

"MSExchangeRepl" event id 2034
The Microsoft Exchange Replication service VSS Writer (Instance 71904d88-eff5-455b-ac6a-68f07fb8c20b) failed with error 80004005 when preparing for snapshot.

When backup is complete we get this errors:

"MSExchangeRepl" event id 2140
The Microsoft Exchange Replication service VSS Writer encountered an exception in function Microsoft::Exchange::Cluster::ReplicaVssWriter::CReplicaVssWriterInterop::BackupCompleteTruncateLogsComponents. HResult -3. Exception Microsoft.Mapi.MapiExceptionCallFailed: MapiExceptionCallFailed: Unable execute Truncate on snapshot. (hr=0x80004005, ec=1295)
Diagnostic context:
    Lid: 1494    Remote Context Beg
    Lid: 63528   StoreEc: 0x50F     
    Lid: 42621   StoreEc: 0x50F     
    Lid: 10786   dwParam: 0x0        Msg: 15.01.1779.007:EX1:14f782c4-aadf-458e-bb8e-68553d8a0320
    Lid: 51578   Guid: 14f782c4-aadf-458e-bb8e-68553d8a0320
    Lid: 1750    Remote Context End
    Lid: 59216  
    Lid: 34640   StoreEc: 0x50F     
    Lid: 57114  
    Lid: 40730   StoreEc: 0x50F     
   bei Microsoft.Mapi.MapiExceptionHelper.InternalThrowIfErrorOrWarning(String message, Int32 hresult, Boolean allowWarnings, Int32 ec, DiagnosticContext diagCtx, Exception innerException)
   bei Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, IExInterface iUnknown, Exception innerException)
   bei Microsoft.Mapi.ExRpcAdmin.ProcessSnapshotOperation(Guid mdbGuid, SnapshotOperationCode operationCode, UInt32 flags)
   bei Microsoft.Exchange.Cluster.Replay.StoreRpcController.<>c__DisplayClass32_0.<SnapshotTruncateLogInstance>b__0()
   bei Microsoft.Exchange.Cluster.Shared.SafeRefCountedTimeoutWrapper.<>c__DisplayClass21_0.<ProtectedCallWithTimeout>b__0()
   bei Microsoft.Exchange.Data.HA.InvokeWithTimeout.Invoke(Action invokableAction, Action foregroundAction, TimeSpan invokeTimeout, Boolean sendWatsonReportNoThrow, Object cancelEvent)
   bei Microsoft.Exchange.Cluster.Shared.SafeRefCountedTimeoutWrapper.ProtectedCallWithTimeout(String operationName, TimeSpan timeout, Action operation)
   bei Microsoft.Exchange.Cluster.ReplicaVssWriter.CReplicaVssWriterInterop.SnapshotTruncateLogInstance(Guid dbGuid, UInt32 flags)
   bei Microsoft.Exchange.Cluster.ReplicaVssWriter.CReplicaVssWriterInterop.BackupCompleteTruncateLogsReplica(ReplayConfiguration replica, Guid vssIdCurrentSnapshotSetId, Boolean fTruncateLogs)
   bei Microsoft.Exchange.Cluster.ReplicaVssWriter.CReplicaVssWriterInterop.BackupCompleteTruncateLogsComponents(IVssWriterComponents* pComponents, Guid vssIdCurrentSnapshotSetId, Boolean fTruncateLogs).

"MSExchangeRepl" event id 2140
The Microsoft Exchange Replication service VSS Writer (Instance 71904d88-eff5-455b-ac6a-68f07fb8c20b) failed with error FFFFFFFD when processing the backup completion event.

"VSS" 8229

Von einem VSS Writer wurde ein Ereignis mit dem Fehler "0x800423f3, Für den Generator wurde ein vorübergehender Fehler festgestellt. Wenn der Sicherungsvorgang wiederholt wird, tritt der Fehler möglicherweise nicht erneut auf." zurückgewiesen. Änderungen, die während der Verarbeitung des Ereignisses an den Writerkomponenten vorgenommen wurden, sind für den Anforderer nicht verfügbar. Zugehörige Ereignisse der Hostanwendung des VSS Writer finden Sie im Ereignisprotokoll. 

Vorgang:
   BackupComplete-Ereignis

Kontext:
   Ausführungskontext: Writer
   Generatorklassen-ID: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
   Generatorname: Microsoft Exchange Writer
   Generatorinstanzname: Exchange Replication Service
   Generatorinstanz-ID: {f20775fd-7aba-42f8-92d3-b80a17a8ce74}
   Befehlszeile: "C:\Program Files\Microsoft\Exchange Server\V15\Bin\msexchangerepl.exe"
   Prozess-ID: 12984

I hope we can solve this prorblem. Thanks in advance

All replies (6)

Friday, November 29, 2019 8:47 AM | 1 vote

Check following or contact Veeam support,

https://forums.veeam.com/veeam-backup-replication-f2/upgrade-to-v9-and-exchange-2013-dag-backup-warning-t32764.html

You can run below command in cmd and check the status,

vssadmin list writers

Asper the output, check relevant service status. Then you can manually restart those services,

VSS Writer Process (.exe) Service
ASR Writer VSS Volume Shadow Copy
BITS Writer BITS Background Intelligent Transfer Service
COM+ REGDB Writer VSS Volume Shadow Copy
DFS Replication Service Writer DFSR DFS Replication
DHCP Jet Writer DHCPServer DHCP Server
FRS Writer NtFrs File Replication
FRSM Writer srmsvc File Server Resource Manager
IIS Config Writer AppHostSvc Application Host Helper Service
IIS Metabase Writer IISADMIN IIS Admin Service
Microsoft Exchange Replica Writer MSExchangeRepl Microsoft Exchange Replication Service
Microsoft Exchange Writer MSExchangeIS Microsoft Exchange Information Store
Microsoft Hyper-V VSS Writer vmms Hyper-V Virtual Machine Management
MSMQ Writer MSMQ Message Queuing
MSSearch Service Writer WSearch Windows Search
NTDS NTDS Active Directory Domain Services
O_Search VSS Writer OSearch Office SharePoint Server Search
O_Search 14 VSS Writer OSearch44 SharePoint Server Search 14
Registry Writer VSS Volume Shadow Copy
Shadow Copy Optimization Writer VSS Volume Shadow Copy
SMS Writer SMSSITEVSSWRITER SMS_SITE_VSS_WRITER
SPSearch VSS Writer SPSearch Windows SharePoint Services Search
SPSearch4 VSS Writer SPSearch4 SharePoint Foundation Search V4
SqlServerWriter SQLWriter SQL Server VSS Writer
System Writer CryptSvc Cryptographic Services
TermServLicensing TermServLicensing Remote Desktop Licensing
WIDWriter WIDWriter Windows Internal Database VSS Writer
WINS Jet Writer WINS Windows Internet Name Service (WINS)
WMI Writer Winmgmt Windows Management Instrumentation

Friday, November 29, 2019 8:49 AM

I know this post. But since there are no failover errors during backup, this imo not applicable.

You think this is a 100% veeam error?

C:\Windows\system32>vssadmin list writers
vssadmin 1.1 - Verwaltungsbefehlszeilenprogramm des Volumeschattenkopie-Dienstes
(C) Copyright 2001-2013 Microsoft Corp.

Verfassername: "Task Scheduler Writer"
   Verfasserkennung: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
   Verfasserinstanzkennung: {1bddd48e-5052-49db-9b07-b96f96727e6b}
   Status: [1] Stabil
   Letzter Fehler: Kein Fehler

Verfassername: "VSS Metadata Store Writer"
   Verfasserkennung: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
   Verfasserinstanzkennung: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
   Status: [1] Stabil
   Letzter Fehler: Kein Fehler

Verfassername: "Performance Counters Writer"
   Verfasserkennung: {0bada1de-01a9-4625-8278-69e735f39dd2}
   Verfasserinstanzkennung: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
   Status: [1] Stabil
   Letzter Fehler: Kein Fehler

Verfassername: "System Writer"
   Verfasserkennung: {e8132975-6f93-4464-a53e-1050253ae220}
   Verfasserinstanzkennung: {1fc55e1f-12ed-492d-90ba-96358805b022}
   Status: [1] Stabil
   Letzter Fehler: Kein Fehler

Verfassername: "Cluster Shared Volume VSS Writer"
   Verfasserkennung: {1072ae1c-e5a7-4ea1-9e4a-6f7964656570}
   Verfasserinstanzkennung: {d3e45ba6-faae-4044-97ad-9d266bb00699}
   Status: [1] Stabil
   Letzter Fehler: Kein Fehler

Verfassername: "ASR Writer"
   Verfasserkennung: {be000cbe-11fe-4426-9c58-531aa6355fc4}
   Verfasserinstanzkennung: {c8093753-b126-4266-89a4-fd6104d9da2a}
   Status: [1] Stabil
   Letzter Fehler: Kein Fehler

Verfassername: "Shadow Copy Optimization Writer"
   Verfasserkennung: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   Verfasserinstanzkennung: {695cc125-b0ef-4ee2-acb2-c7f8a3975868}
   Status: [1] Stabil
   Letzter Fehler: Kein Fehler

Verfassername: "Microsoft Exchange Writer"
   Verfasserkennung: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
   Verfasserinstanzkennung: {f20775fd-7aba-42f8-92d3-b80a17a8ce74}
   Status: [1] Stabil
   Letzter Fehler: Wiederholbarer Fehler

Verfassername: "Registry Writer"
   Verfasserkennung: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Verfasserinstanzkennung: {90470a2e-88bb-414b-9282-eabd691f96a8}
   Status: [1] Stabil
   Letzter Fehler: Kein Fehler

Verfassername: "COM+ REGDB Writer"
   Verfasserkennung: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Verfasserinstanzkennung: {c19d4967-5491-4c8a-97ad-44040e757ab8}
   Status: [1] Stabil
   Letzter Fehler: Kein Fehler

Verfassername: "BITS Writer"
   Verfasserkennung: {4969d978-be47-48b0-b100-f328f07ac1e0}
   Verfasserinstanzkennung: {c360818c-7434-4471-9edd-f5e7faa623be}
   Status: [1] Stabil
   Letzter Fehler: Kein Fehler

Verfassername: "WMI Writer"
   Verfasserkennung: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Verfasserinstanzkennung: {5904cc4f-3314-4822-9ed3-b7010cb30262}
   Status: [1] Stabil
   Letzter Fehler: Kein Fehler

Verfassername: "MSMQ Writer (MSMQ)"
   Verfasserkennung: {7e47b561-971a-46e6-96b9-696eeaa53b2a}
   Verfasserinstanzkennung: {9009411b-6995-4176-9bef-c6862c9f4696}
   Status: [1] Stabil
   Letzter Fehler: Kein Fehler

Verfassername: "IIS Metabase Writer"
   Verfasserkennung: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
   Verfasserinstanzkennung: {49435770-cccf-426f-99aa-2a40d523f7c9}
   Status: [1] Stabil
   Letzter Fehler: Kein Fehler

Verfassername: "IIS Config Writer"
   Verfasserkennung: {2a40fd15-dfca-4aa8-a654-1f8c654603f6}
   Verfasserinstanzkennung: {a16ba553-459b-476e-aac8-2a051141abbe}
   Status: [1] Stabil
   Letzter Fehler: Kein Fehler

Verfassername: "Cluster Database"
   Verfasserkennung: {41e12264-35d8-479b-8e5c-9b23d1dad37e}
   Verfasserinstanzkennung: {836e6198-20fa-4803-acda-8fe2b38533ff}
   Status: [1] Stabil
   Letzter Fehler: Kein Fehler

Sorry. I forgot this info. 

Monday, December 2, 2019 9:56 AM

Hi kschueler,

The issue may be caused by a conflict between the VMware Tools and certain backup agents. The solution for us is to make sure the VMware Tools are installed *before* the backup agent. This also means when updating the Tools we need to uninstall the backup agent, update Tools, then reinstall the agent.

Here's a possible resolution for you:

Go to Task Manager, locate the task "ShadowCopyVolume(...)" that shows an error at latest status. Right click and choose delete. See if the issue persists.

Regards,

Beverly Gao

Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact [email protected].

Thursday, December 5, 2019 1:22 AM

Hi,
I am writing here to confirm with you how the thing going now?
If you need further help, please provide more detailed information, so that we can give more appropriate suggestions.

Regards,

Beverly Gao

Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact [email protected].

Thursday, December 5, 2019 7:20 AM

Like Udara Kaushalya reccomended, I contacted veeam support for additional help. The supporter told me, that I need to put the exchange backup into an extra job. Since three backups no errors.

The error was caused by a vss timeout. Since my system is during backup under heavy load, vss could not release vss snapshot fast enough, so the backup got a warning. This is the explanation so far.

Tuesday, December 10, 2019 9:53 AM

Hi kschueler,

It seems the issue has been resolved. You can mark your reply as answer, this will make answer searching in the forum easier and be beneficial to other community members as well.

Regards,

Beverly Gao

Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact [email protected].