System state backups using MARS agent fail on VSS copy: The volume C:\Program Files\Microsoft Azure Recovery Services Agent\Scratch\SSBV\ is not found.

Hello Nathan Hasey, Thank you for posting your query on Microsoft Q&A platform.

We are looking into your issue and will keep you posted updates.

Thanks,

Hello Nathan Hasey, Could you please share below details?

1. “When did this start, and what changed around that time?”
2. “Did you recently update the MARS agent, Windows Server Backup components, or apply cumulative updates?”
3. “Did anything change in storage configuration—new volumes, removed disks, iSCSI targets, mount points, or capacity expansion?”
4. “Is the failure happening only for System State, or do file/folder backups fail too?”
5. “Is this on a physical server or a VM? If it’s a VM, what hypervisor (Hyper‑V/VMware/Azure)?”
6. “Is the issue affecting one server only, or multiple servers?”
7. “What’s the output of vssadmin list writers right before and right after the backup?”
8. “What’s the output of vssadmin list providers?”
9. “Are there any other backup/snapshot tools running on this server (Veeam/Backup Exec/other) during the same time window?”
10. “What was CPU/memory utilization during backup time? Any heavy load or parallel jobs?”
11. “Do you have enough free space on the OS/system volumes and on the volume used for shadow copies?”
12. “Please share vssadmin list shadowstorage output (and confirm if shadow storage is configured/limited for C:).
13. “Are any volumes offline, removable, read-only, or non‑NTFS?
14. (VM-specific but high signal) “Is any ISO mounted in the VM’s DVD drive (or any ‘disconnected’ virtual device)?
15. “What is your configured MARS scratch/cache location? Is it the default path or moved to another drive?”
16. “How much free space is available on the volume hosting the scratch folder?
17. “Do you have antivirus/EDR running? Are exclusions configured for <InstallPath>\Scratch\*, <InstallPath>\Bin\*, and cbengine.exe?”
18. “Any file/folder access hardening (Controlled Folder Access / ransomware protection / EDR tamper protection) that could block scratch or VSS?”
19. “What MARS agent version are you running? Is it the latest from Microsoft?”

Thanks,
Suchitra.

1. The last successful System State backup is from 3/10
2. The MARS agent was recently updated from version 2.0.9221.0 to version 2.0.9506.0, as well as March 10, 2026—KB5078752
3. Nothing has recently changed.
4. I believe this is only for system data.
5. This is in Azure
6. Only one server, it is part of a Failover Cluster and the other node backs up system state perfectly fine

PS C:\Program Files\Microsoft Azure Recovery Services Agent\Scratch\SSBV> vssadmin list writers

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

(C) Copyright 2001-2013 Microsoft Corp.

Writer name: 'Task Scheduler Writer'

Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}

Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}

State: [1] Stable

Last error: No error

Writer name: 'VSS Metadata Store Writer'

Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}

Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}

State: [1] Stable

Last error: No error

Writer name: 'Performance Counters Writer'

Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}

Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}

State: [1] Stable

Last error: No error

Writer name: 'System Writer'

Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}

Writer Instance Id: {77219fbb-03e8-467a-9baa-c5d32f8d2c56}

State: [1] Stable

Last error: No error

Writer name: 'Shadow Copy Optimization Writer'

Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}

Writer Instance Id: {8ea5849e-9480-4d29-bb17-5f2044d99044}

State: [1] Stable

Last error: No error

Writer name: 'Cluster Shared Volume VSS Writer'

Writer Id: {1072ae1c-e5a7-4ea1-9e4a-6f7964656570}

Writer Instance Id: {db5c8410-7288-41b8-a27b-21d613fd87f0}

State: [1] Stable

Last error: No error

Writer name: 'ASR Writer'

Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}

Writer Instance Id: {da4ed490-13f9-4a62-a661-ece919c7e131}

State: [1] Stable

Last error: No error

Writer name: 'IIS Config Writer'

Writer Id: {2a40fd15-dfca-4aa8-a654-1f8c654603f6}

Writer Instance Id: {df6c2d3b-645c-4c28-9922-3473fec24405}

State: [1] Stable

Last error: No error

Writer name: 'COM+ REGDB Writer'

Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}

Writer Instance Id: {f6c32093-b23b-47e8-a1c6-6ee133fdbbe0}

State: [1] Stable

Last error: No error

Writer name: 'Registry Writer'

Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}

Writer Instance Id: {8c64d842-6841-42c2-84f5-7d286e8315b1}

State: [1] Stable

Last error: No error

Writer name: 'SqlServerWriter'

Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}

Writer Instance Id: {7d4ca14a-c76c-485c-866e-53be55b173fa}

State: [1] Stable

Last error: No error

Writer name: 'WMI Writer'

Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}

Writer Instance Id: {8feb8e5c-2b8d-45e3-990c-ce91d9cf781a}

State: [1] Stable

Writer name: 'BITS Writer'

Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}

Writer Instance Id: {d134c657-cb61-4128-b0ae-166862e67f33}

State: [1] Stable

Last error: No error

Writer name: 'Cluster Database'

Writer Id: {41e12264-35d8-479b-8e5c-9b23d1dad37e}

Writer Instance Id: {99f93fa3-3536-468d-a4ae-1fe0583f9011}

State: [1] Stable

Last error: No error

This is for before and after

8. PS C:\Program Files\Microsoft Azure Recovery Services Agent\Scratch\SSBV> vssadmin list providers vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool (C) Copyright 2001-2013 Microsoft Corp.

Provider name: 'Microsoft CSV Shadow Copy Helper Provider' Provider type: Software Provider Id: {26d02d81-6aac-4275-8504-b9c6edc5261d} Version: 1.0.0.1

Provider name: 'Microsoft CSV Shadow Copy Provider' Provider type: Software Provider Id: {400a2ff4-5eb1-44b0-8a05-1fcac0bcf9ff} Version: 1.0.0.1

Provider name: 'Microsoft File Share Shadow Copy provider' Provider Id: {89300202-3cec-4981-9171-19f59559e0f2} Version: 1.0.0.1

Provider name: 'Microsoft Software Shadow Copy provider 1.0' Provider type: System Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5} Version: 1.0.0.7

9. No other backup tools are configured.
10. Memory was under heavy load.
11. C: has 132 GB free
12. PS C:\Program Files\Microsoft Azure Recovery Services Agent\Scratch\SSBV> vssadmin list shadowstorage vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool (C) Copyright 2001-2013 Microsoft Corp.

Shadow Copy Storage association For volume: (C:)\?\Volume{8b07c948-0000-0000-0000-501f00000000}
Shadow Copy Storage volume: (C:)\?\Volume{8b07c948-0000-0000-0000-501f00000000}
Used Shadow Copy Storage space: 28.7 MB (0%) Allocated Shadow Copy Storage space: 1.25 GB (0%) Maximum Shadow Copy Storage space: 25.6 GB (10%)

13. Yes, there is a cluster volume.

PS C:\Program Files\Microsoft Azure Recovery Services Agent\Scratch\SSBV> Get-Volume

DriveLetter FriendlyName            FileSystemType DriveType HealthStatus OperationalStatus SizeRemaining      Size

----------- ------------            -------------- --------- ------------ ----------------- -------------      ----


CB-{2D91842A-4616-4D38- NTFS           Fixed     Healthy      OK                     65.38 GB    152 GB
A                                   Unknown        Removable Healthy      Unknown                     0 B       0 B

System Reserved         NTFS           Fixed     Healthy      OK                    465.33 MB    500 MB
C           Windows                 NTFS           Fixed     Healthy      OK                    132.76 GB 255.51 GB

D           Temporary Storage       NTFS           Fixed     Healthy      OK                      2.56 GB     32 GB

F           Backup                  NTFS           Fixed     Healthy      OK                     65.33 GB 255.98 GB

Volume1                 CSVFS_ReFS     Fixed     Healthy      OK                    117.92 GB 505.94 GB

14. No
15. Default
16. 132 GB
17. Yes defender for cloud is running on the VM.
18. Not that I am aware of.
19. I am running version 2.0.6506.0

Hello Nathan Hasey,

Thanks for the response.

• Only System State fails; the other cluster node completes System State backups successfully → this is node-specific (not vault/service-wide).
• All VSS writers are “Stable” before/after the backup → this is not a “writer failed” scenario.
• The failure you captured earlier is VSS Event ID 12289 with GetStorageDependencyInformation / 0x8007048f (“The device is not connected”) (your screenshot/log in the original post).
• The failing node shows a suspicious “volume”: A: DriveType = Removable, Size = 0 B, OperationalStatus = Unknown (from Get-Volume).
• Shadow copy storage on C: is configured (max 10% / 25.6 GB), and C: has plenty of free space.

The backup fails because VSS snapshot finalization is failing on that node due to a “device not connected” storage dependency issue (0x8007048f).

• The primary failure is the VSS snapshot stage (Event 12289 / “Unexpected error… hr=0x8007048f”). When VSS hits a storage dependency it can’t resolve or access, the snapshot operation fails. Event 12289 supports that this is an underlying VSS error that must be diagnosed via correlated events/requester context.
• In clustered/CSV contexts, this error pattern is commonly associated with a node not being able to read/access the relevant storage dependency (CSV or another dependency). A vendor KB describing the same Event 12289 + 0x8007048f pattern notes it can happen when the node cannot read from CSV and that reboot can clear the condition (workaround).
• The MARS “SSBV volume is not found” message is therefore a secondary symptom: after VSS fails, the System State “copy/validation” phase can’t validate the backup-set mount/location and reports the SSBV mount path as “not found” (as seen in your CBEngine excerpt in the original post).

Your Get-Volume output shows A: as Removable / Unknown / 0 bytes. That is exactly the type of “device not connected” artifact that can trip dependency discovery (because VSS calls like GetStorageDependencyInformation walk storage relationships).

I’m calling this the most likely trigger because:

• It exists on the failing node’s volume list.
• The VSS error is literally “device not connected.”
• The other node does not fail (suggesting a node-specific device difference).

1. Fix the underlying “device not connected” storage dependency on the failing node (primary fix) eliminate/repair any disconnected/phantom devices that show up in the OS storage stack on that node.

• Remove/disable the phantom removable device showing as A: (often a legacy virtual floppy/media device in VM settings or a stale device driver entry).
• If the VM has any mounted ISO/DVD media, unmount it
• Validate the node has proper access to cluster storage/CSV at backup time (since the same error is seen when a node cannot read CSV).

2. Workaround: reboot the failing node:

A reboot is frequently used as a practical workaround in this error family (vendor KB calls it out for this exact 12289/0x8007048f pattern). (If the phantom device remains, the issue may return.)

3. Reduce resource pressure during the snapshot window.

You mentioned memory was under heavy load. Heavy system load can contribute to VSS instability/timeouts in general. While your writers are stable, it’s still a good operational mitigation to schedule System State backups in a lower-load window.

4. Ensure MARS scratch + AV/EDR exclusions are correct : Even though VSS is the main failure, backup can fail if the scratch/cache folder is not accessible or is blocked by other processes (including AV).

• Ensure scratch location is valid/accessible and has adequate free space
• Ensure shadow copy storage is sufficient
• Add AV exclusions for <InstallPath>\Scratch\*, <InstallPath>\Bin\*, and cbengine.exe

You have “Defender for Cloud” on the VM, verifying exclusions/policy impact is worthwhile (as an eliminator).

6. Check the agent version inconsistency: Confirm the installed version from MARS console > Help > About or Programs & Features and ensure the intended version is actually installed.

Thanks,
Suchitra.

1. I have removed the A: drive
2. I have rebooted
3. the reboot freed a lot of presssure
4. I have added exclusions After running mountvol and seeing 

          \\?\Volume{6422bcc2-c38a-4a4b-a13e-8b13ecda1c58}\
              C:\Program Files\Microsoft Azure Recovery Services Agent\Scratch_Old\SSBV\
              C:\Program Files\Microsoft Azure Recovery Services Agent\Scratch\SSBV\
   

   The issue was that it had two mount points making the IsValidSSB check fail. I removed the Scratch_Old dir and backup succeed. What's still unknown to me is why backups failed in the first place as the new scratch dir was created in response to the failing backups. Most likely case is that something in the scratch dir either was corrupted or inconsistent and had to be recreated. which I did, but not wanting to delete the old stuff I mistakenly left around Scratch_Old thinking the rename would lose any association with WSB.