Azure Backup: Transfer Data to Vault Phase Duration Increased Since October 13 — Performance Regression

Matt 0 Reputation points
2026-10-20T16:34:18.6466667+00:00

Problem description Since 2026-10-13 the "Transfer data to vault" phase of Azure VM backup has gone from under 1 hour to 6 to 8 hours on every Central US VM in our subscription: 7 VMs across 7 separate Recovery Services vaults, all on the same day. 11 VMs in East US under the same kind of vaults and policies are unaffected. Backups complete and recovery points are valid. No errors. Example VM, total job time (snapshot phase stays at about 19 minutes): 10/12: 2:08 10/13: 6:10 10/14: 7:11 10/15: 7:28 10/16: 7:11 10/17: 8:15 10/18: 7:41 10/19: 7:11 10/20: 6:56 (transfer phase 6:19 for 6.3 GB of changed data)

Environment Azure Backup, Recovery Services vaults (Standard tier, Geo-redundant, no private endpoints), Enhanced policy, daily. Central US. Example VM has an OS disk and a Premium SSD v2 data disk; the other affected VMs use a mix of disk types.

What I've already tried Compared the affected and unaffected vaults: identical redundancy and configuration, Resource Health Available on all of them. Service Health shows no Azure Backup events in any region since 10/13. No changes to VM size, disks, policies, or extensions around that date (activity log is empty for them). Changed data per day is flat at a few GB. VMs are idle during the backup window, disk IOPS and throughput are at or under 1 percent of limits. Support case 2610200040005510 is open; the AI responder's own conclusion was "a Central US-specific slowdown in the Azure Backup transfer path after the snapshot is created", and the Developer plan routes here for anything further.

Current status Still 6 to 8 hours daily. Looking for anyone from the Azure Backup team who can confirm a Central US transfer-path regression starting 2026-10-13, or tell me what else on the customer side could make 7 vaults slow down on the same day.

Azure Backup
Azure Backup

An Azure backup service that provides built-in management at scale.


4 answers

Sort by: Most helpful
  1. Matt 0 Reputation points
    2026-10-25T16:13:01.5366667+00:00

    Thanks Bharath, glad it is with engineering. Final result for the on-demand backup on the Premium v1 VM, job f5eff1bf-862a-4094-9d56-b7154ed70a4d (10/23, all UTC): Backup Size 444 MB. Take Snapshot 12:55:16 to 13:04:56 (9m40s). Transfer data to vault 13:04:57 to 17:09:03 (4h04m06s). Validate 30s. Total 4h16m. So 444 MB at midday took 4h04m, versus 2h19m for 1.1 to 2.2 GB on the same VM's 01:30 schedule and 19m for up to 5.9 GB before 10/13. The midday slot is the slowest we have measured for this VM, so the wait is not confined to busy hours. The 15-minute pattern holds here too: 19m04s, 2h19m04s and 4h04m06s are all 4m plus a whole number of 15-minute ticks (1, 9 and 16). On the vault resource IDs: I will not post subscription and resource group identifiers on a public thread. Support case 2610200040005510 is already open on the same subscription for this exact issue, and I have no outbound channel on it. If engineering needs the vault IDs, please have them reach out directly through that case and I will provide them there.

    Was this answer helpful?

    0 comments No comments

  2. Matt 0 Reputation points
    2026-10-23T16:28:28.05+00:00

    Thanks Bharath. Answers to the four questions, plus two things in the job data that I think change the picture. All times UTC, all figures straight from az backup job show task lists.

    1. Premium SSD v1 VM, data transferred before 10/13

    Date Backup Size Take Snapshot Transfer data to vault Validate
    10/09 5,928 MB 8m38s 19m03s 32s
    10/10 2,416 MB 9m39s 19m04s 30s
    10/11 1,296 MB 9m08s 19m04s 30s
    10/12 1,227 MB 9m10s 19m03s 30s
    10/13 1,654 MB 9m09s 1h49m15s 15m30s
    10/14 2,064 MB 9m08s 2h19m04s 30s
    10/15 2,192 MB 9m08s 2h19m06s 35s
    10/16 1,473 MB 8m38s 2h19m04s 30s
    10/17 1,117 MB 9m41s 2h19m05s 30s
    10/20 1,097 MB 9m39s 2h19m08s 30s
    10/21 1,581 MB 8m40s 2h19m04s 30s
    10/23 2,142 MB 8m38s 2h20m22s 30s

    So a rate comparison does not describe what is happening. Before 10/13 the transfer phase took 19m03s to 19m04s whether the backup was 1.2 GB or 5.9 GB. Since 10/14 it takes 2h19m04s to 2h19m08s whether it is 1.1 GB or 2.2 GB. The phase has a fixed duration that does not scale with data, and on 10/13 exactly 2h00m was added to it. That looks like queue or scheduling wait inside the transfer phase, not slow throughput.

    Every phase duration in both vaults is quantized to 15 minutes. Transfer on this VM is always 19m or 2h19m (or 34m on 9/28, 6h04m on 9/27). On the Pv2 VM it is 33m, 49m, 1h04m, 1h19m, 1h49m before 10/13 and 5h34m, 6h19m, 6h34m, 6h49m, 7h04m, 7h19m since. Validate is always either 30s or 15m30s. Whatever is pacing these jobs runs on a 15-minute tick, and the slowdown is 8 extra ticks on the 00:30-01:30 schedules and 20+ extra ticks on the 07:00-08:15 schedules.

    2. Pv2 VM changed data since 10/13

    Date Backup Size Transfer
    10/13 4,133 MB 5h34m
    10/14 12,361 MB 6h49m
    10/15 10,694 MB 7h04m
    10/16 4,175 MB 6h49m
    10/17 4,118 MB 7h23m
    10/18 2,213 MB 7h19m
    10/19 1,547 MB 6h49m
    10/20 6,349 MB 6h19m
    10/21 20,087 MB 6h19m
    10/22 5,923 MB 6h34m

    It has not been a steady 6 GB. The variation is expected and explained: this VM was being staged as a replacement file server, with a nightly incremental copy of an 80 GB application dataset from an on-premises server through 10/19, then it went into production on 10/20 (that is the 20 GB on 10/21). No database, logging or antivirus change. The 10/12 figure (2,318 MB) was a low day, not a baseline. Note that 1.5 GB on 10/19 took 6h49m and 20 GB on 10/21 took 6h19m, so on this VM too the transfer time is independent of data.

    On Validate: the 15m30s value is not Pv2-specific. It shows up on the Pv2 VM on 10/6, 10/7, 10/9 and 10/10 (before the slowdown) and on the Premium v1 VM on 10/13, 10/18, 10/19 and 10/22. It is the same 15-minute tick, one missed poll, and it predates 10/13.

    3. Provisioned disks, 7 Central US VMs

    VM OS disk Data disk
    Pv2 VM (example) 256 GB Premium SSD 256 GB Premium SSD v2
    Premium v1 VM (job 73aae2d1) 64 GB Premium SSD 128 GB Premium SSD
    VM 3 30 GB Premium SSD 256 GB Premium SSD
    VM 4 64 GB Premium SSD 256 GB Premium SSD
    VM 5 64 GB Premium SSD 256 GB Premium SSD
    VM 6 64 GB Premium SSD 128 GB Premium SSD
    VM 7 256 GB Standard SSD none

    The 64+128 GB VM and the 64+256 GB VMs on the same 00:30-01:30 schedule all gained the same 2h00m, so I do not see a disk-size dependency either.

    4. Did this happen before 10/13? Yes, and it is the most useful thing I found.

    Looking back to August on the Premium v1 VM (01:30 schedule) and on a 07:00-schedule VM:

    Period Premium v1 VM, total job 07:00 VM, total job
    Aug 1 - Sep 10 1h00m to 2h15m, varying day to day 1h15m to 4h00m, varying
    Sep 11 - Sep 26 2h15m to 2h30m every day 5h15m to 6h00m every day
    Sep 27 6h15m 6h31m
    Sep 28 - Oct 12 0h30m every day 1h00m to 2h00m
    Oct 13 - today 2h30m every day 5h00m to 6h45m

    The state changed for every Central US VM on the same days: 9/11-9/12 (slower), 9/28 (much faster), 10/13 (slower again). The 9/28-10/12 window is the only sustained period in the record where the transfer phase ran without the added wait. The East US vault did not move on any of those dates. Three simultaneous regime changes across all vaults in one region, with the delay arriving in exact 15-minute multiples and independent of data volume, points at the regional transfer pipeline's scheduling or capacity rather than anything in these subscriptions.

    On-demand backup on the Premium v1 VM (your item 2). Job f5eff1bf-862a-4094-9d56-b7154ed70a4d, started 12:55 UTC today (10/23), retain-until 10/26. Take Snapshot 9m40s. "Transfer data to vault" began at 13:05 UTC and was still InProgress at 16:25 UTC, 3h20m in, with Validate not started. That already exceeds the 2h19m the same VM gets on its 01:30 UTC schedule, so the added wait is not confined to the early-morning slots; midday UTC is worse. I will add the final duration and Backup Size when it completes. VM was idle and nothing else was running in the subscription.

    Today's scheduled Central US jobs for comparison, all 10/23:

    VM Schedule Backup Size Transfer data to vault
    Premium v1 VM (above) 01:37 2,142 MB 2h20m22s
    Pv2 VM 08:13 4,004 MB 6h21m37s
    07:00 VM 07:05 6,555 MB 5h34m07s
    07:30 VM 07:38 643 MB 6h48m05s

    643 MB taking 6h48m while 6,555 MB on the neighbouring schedule took 5h34m is the clearest case yet that this phase is waiting, not transferring. Given that, I would rather not move the 07:00-08:00 schedules until engineering says what the wait is; the on-demand result suggests 12:00-15:00 UTC would not help. Nothing on our side has been changed.

    Was this answer helpful?

    0 comments No comments

  3. Matt 0 Reputation points
    2026-10-21T14:18:05.83+00:00

    Thanks Bharath. Answers to your five questions, all times UTC.

    1. Premium SSD v2 in East US? No. Only one of the seven Central US VMs (the example VM) has a Premium SSD v2 data disk. The other six Central US VMs are Premium SSD (v1), one with Standard SSD. All ten East US VMs are Premium SSD v1, Standard SSD or Standard HDD. The six non-Pv2 Central US VMs stepped up on the same day as the Pv2 one, so Pv2 is not the common factor. The only Pv2-specific difference I see is the "Validate Backup" phase, which went from 30 s to about 15 min on that VM only.

    2. Scheduled times and any change around 10/13. Nothing changed. Every policy was created months ago (the newest on 2026-10-07) and none has been edited. Scheduled start times, Central US: 00:30 (two vaults), 01:30, 07:00, 07:30, 08:00, 16:00. East US: 01:00 (all ten VMs, one vault). Jobs still kick off 10 to 20 minutes after the scheduled time, same as before 10/13.

    One pattern that may help your backend team: the size of the slowdown depends on the time of day the job runs.

    Central US schedule (UTC) Before 10/13 Since 10/13
    00:30 and 01:30 (3 VMs) ~30 min 120 to 165 min
    07:00 to 08:15 (3 VMs) 60 to 85 min 330 to 495 min
    16:00 (1 VM) ~40 min 130 to 175 min

    East US (01:00, 10 VMs) is unchanged at 100 to 150 min per VM, with the same two RDS hosts always at the top of that range. "Take Snapshot" is unchanged everywhere; only "Transfer data to vault" grew.

    3. On-demand backup at a different time. Not run yet. I can run one on an idle Central US VM if the backend team wants that data point, and will post the result here.

    4. Other Central US operations. We do not run snapshot copies or disk exports in Central US, so I have no comparison data. The VMs themselves show no change in disk latency or throughput.

    5. Vault resource IDs. Sent through support case 2610200040005510 rather than posting them publicly. Job IDs for the backend team:

    • Central US, affected, Pv2 VM, 2026-10-20: job 093d9f06-fd3d-459f-a939-17057f231d2d. Start 08:11:34. Take Snapshot 19 min. Transfer data to vault 08:31:03 to 14:50:16 (6 h 19 min) for 6,349 MB. Validate 15 min.
    • Central US, affected, Premium SSD v1 VM, 2026-10-20: job 73aae2d1-3484-4581-a57c-e499fa16505b. Start 01:35:59. Take Snapshot 10 min. Transfer 01:45:40 to 04:04:48 (2 h 19 min) for 1,097 MB. Same VM was 30 min end to end on 10/11 and 10/12.
    • Central US, same Pv2 VM before the change, 2026-10-12: job 3f4e2668-c63d-430c-9271-54dc12e94035. Start 08:18:00. Transfer 1 h 49 min for 2,318 MB. On 10/11 the whole job took 85 min.
    • East US, unaffected control, 2026-10-20: job 0ac73a85-f4e2-444f-8a6d-066800f1ecf8. Start 01:19:31. Take Snapshot 8 min. Transfer 01:27:09 to 03:30:49 (2 h 04 min) for 3,096 MB. Flat since before 10/13.

    Nothing was recreated or re-protected and I do not plan to change anything on our side. Happy to supply more job IDs if the backend team wants the full set.

    Was this answer helpful?

    0 comments No comments

  4. Alex Burlachenko 26,275 Reputation points MVP Volunteer Moderator
    2026-10-21T14:06:23.0633333+00:00

    Hi Matt & thx for join me at Q&A platform,

    pattern here is pretty convincing. Seven VMs across seven separate vaults in Central US all slowed down on October 13, while 11 comparable VMs in East US remain unaffected. With unchanged backup sizes, normal snapshot times, and no configuration changes, this looks much more like a regional Azure Backup service-side issue than something happening inside ur VMs.

    One important detail is that "Transfer data to vault" isn't simply a file upload from the guest OS. Azure VM backup uses a platform-managed snapshot transfer process, so VM network bandwidth, guest CPU usage, and disk IOPS don't necessarily explain delays in this phase. Your example of 6.3 GB taking over six hours is particularly unusual.

    I'd want Microsoft to distinguish between time spent waiting for the transfer to start and time spent actually transferring data. Those can point to very different backend problems, such as service-side queueing, throttling, or a degraded transfer worker. Unfortunately, that level of detail isn't fully exposed through the Azure portal.

    The fact that Resource Health is Available and Service Health shows no incident doesn't rule out a localized service degradation. Not every backend performance issue results in a published Service Health advisory.

    Given everything u've already checked, I wouldn't waste time recreating vaults, changing backup policies, or troubleshooting individual VM extensions. Those actions are unlikely to explain seven independent vaults changing behavior on exactly the same date.

    For support case 2610200040005510, I'd specifically request an Azure Backup engineering investigation into the Central US snapshot-to-vault transfer pipeline, including backend queue times, transfer throughput, and any service deployment or infrastructure changes around October 13. Provide a few affected backup job IDs with UTC timestamps and one unaffected East US job for comparison.

    The Developer support plan may limit direct technical support options, but the case number and detailed evidence should still help a Microsoft moderator route this to the appropriate Azure Backup team.

    I can't confirm a Central US regression without Microsoft's internal telemetry, but ur evidence is strong enough that I'd expect a service-side investigation rather than another round of generic customer-side troubleshooting. If anyone from the Azure Backup team is following this thread, confirmation of whether the Central US transfer pipeline experienced a change on October 13 would be particularly useful.

    rgds,

    Alex

    p.s. if my answer is help - pls mark it as an answer

    Was this answer helpful?

    0 comments No comments

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.