Thanks Bharath. Answers to the four questions, plus two things in the job data that I think change the picture. All times UTC, all figures straight from az backup job show task lists.
1. Premium SSD v1 VM, data transferred before 10/13
| Date |
Backup Size |
Take Snapshot |
Transfer data to vault |
Validate |
| 10/09 |
5,928 MB |
8m38s |
19m03s |
32s |
| 10/10 |
2,416 MB |
9m39s |
19m04s |
30s |
| 10/11 |
1,296 MB |
9m08s |
19m04s |
30s |
| 10/12 |
1,227 MB |
9m10s |
19m03s |
30s |
| 10/13 |
1,654 MB |
9m09s |
1h49m15s |
15m30s |
| 10/14 |
2,064 MB |
9m08s |
2h19m04s |
30s |
| 10/15 |
2,192 MB |
9m08s |
2h19m06s |
35s |
| 10/16 |
1,473 MB |
8m38s |
2h19m04s |
30s |
| 10/17 |
1,117 MB |
9m41s |
2h19m05s |
30s |
| 10/20 |
1,097 MB |
9m39s |
2h19m08s |
30s |
| 10/21 |
1,581 MB |
8m40s |
2h19m04s |
30s |
| 10/23 |
2,142 MB |
8m38s |
2h20m22s |
30s |
So a rate comparison does not describe what is happening. Before 10/13 the transfer phase took 19m03s to 19m04s whether the backup was 1.2 GB or 5.9 GB. Since 10/14 it takes 2h19m04s to 2h19m08s whether it is 1.1 GB or 2.2 GB. The phase has a fixed duration that does not scale with data, and on 10/13 exactly 2h00m was added to it. That looks like queue or scheduling wait inside the transfer phase, not slow throughput.
Every phase duration in both vaults is quantized to 15 minutes. Transfer on this VM is always 19m or 2h19m (or 34m on 9/28, 6h04m on 9/27). On the Pv2 VM it is 33m, 49m, 1h04m, 1h19m, 1h49m before 10/13 and 5h34m, 6h19m, 6h34m, 6h49m, 7h04m, 7h19m since. Validate is always either 30s or 15m30s. Whatever is pacing these jobs runs on a 15-minute tick, and the slowdown is 8 extra ticks on the 00:30-01:30 schedules and 20+ extra ticks on the 07:00-08:15 schedules.
2. Pv2 VM changed data since 10/13
| Date |
Backup Size |
Transfer |
| 10/13 |
4,133 MB |
5h34m |
| 10/14 |
12,361 MB |
6h49m |
| 10/15 |
10,694 MB |
7h04m |
| 10/16 |
4,175 MB |
6h49m |
| 10/17 |
4,118 MB |
7h23m |
| 10/18 |
2,213 MB |
7h19m |
| 10/19 |
1,547 MB |
6h49m |
| 10/20 |
6,349 MB |
6h19m |
| 10/21 |
20,087 MB |
6h19m |
| 10/22 |
5,923 MB |
6h34m |
It has not been a steady 6 GB. The variation is expected and explained: this VM was being staged as a replacement file server, with a nightly incremental copy of an 80 GB application dataset from an on-premises server through 10/19, then it went into production on 10/20 (that is the 20 GB on 10/21). No database, logging or antivirus change. The 10/12 figure (2,318 MB) was a low day, not a baseline. Note that 1.5 GB on 10/19 took 6h49m and 20 GB on 10/21 took 6h19m, so on this VM too the transfer time is independent of data.
On Validate: the 15m30s value is not Pv2-specific. It shows up on the Pv2 VM on 10/6, 10/7, 10/9 and 10/10 (before the slowdown) and on the Premium v1 VM on 10/13, 10/18, 10/19 and 10/22. It is the same 15-minute tick, one missed poll, and it predates 10/13.
3. Provisioned disks, 7 Central US VMs
| VM |
OS disk |
Data disk |
| Pv2 VM (example) |
256 GB Premium SSD |
256 GB Premium SSD v2 |
| Premium v1 VM (job 73aae2d1) |
64 GB Premium SSD |
128 GB Premium SSD |
| VM 3 |
30 GB Premium SSD |
256 GB Premium SSD |
| VM 4 |
64 GB Premium SSD |
256 GB Premium SSD |
| VM 5 |
64 GB Premium SSD |
256 GB Premium SSD |
| VM 6 |
64 GB Premium SSD |
128 GB Premium SSD |
| VM 7 |
256 GB Standard SSD |
none |
The 64+128 GB VM and the 64+256 GB VMs on the same 00:30-01:30 schedule all gained the same 2h00m, so I do not see a disk-size dependency either.
4. Did this happen before 10/13? Yes, and it is the most useful thing I found.
Looking back to August on the Premium v1 VM (01:30 schedule) and on a 07:00-schedule VM:
| Period |
Premium v1 VM, total job |
07:00 VM, total job |
| Aug 1 - Sep 10 |
1h00m to 2h15m, varying day to day |
1h15m to 4h00m, varying |
| Sep 11 - Sep 26 |
2h15m to 2h30m every day |
5h15m to 6h00m every day |
| Sep 27 |
6h15m |
6h31m |
| Sep 28 - Oct 12 |
0h30m every day |
1h00m to 2h00m |
| Oct 13 - today |
2h30m every day |
5h00m to 6h45m |
The state changed for every Central US VM on the same days: 9/11-9/12 (slower), 9/28 (much faster), 10/13 (slower again). The 9/28-10/12 window is the only sustained period in the record where the transfer phase ran without the added wait. The East US vault did not move on any of those dates. Three simultaneous regime changes across all vaults in one region, with the delay arriving in exact 15-minute multiples and independent of data volume, points at the regional transfer pipeline's scheduling or capacity rather than anything in these subscriptions.
On-demand backup on the Premium v1 VM (your item 2). Job f5eff1bf-862a-4094-9d56-b7154ed70a4d, started 12:55 UTC today (10/23), retain-until 10/26. Take Snapshot 9m40s. "Transfer data to vault" began at 13:05 UTC and was still InProgress at 16:25 UTC, 3h20m in, with Validate not started. That already exceeds the 2h19m the same VM gets on its 01:30 UTC schedule, so the added wait is not confined to the early-morning slots; midday UTC is worse. I will add the final duration and Backup Size when it completes. VM was idle and nothing else was running in the subscription.
Today's scheduled Central US jobs for comparison, all 10/23:
| VM |
Schedule |
Backup Size |
Transfer data to vault |
| Premium v1 VM (above) |
01:37 |
2,142 MB |
2h20m22s |
| Pv2 VM |
08:13 |
4,004 MB |
6h21m37s |
| 07:00 VM |
07:05 |
6,555 MB |
5h34m07s |
| 07:30 VM |
07:38 |
643 MB |
6h48m05s |
643 MB taking 6h48m while 6,555 MB on the neighbouring schedule took 5h34m is the clearest case yet that this phase is waiting, not transferring. Given that, I would rather not move the 07:00-08:00 schedules until engineering says what the wait is; the on-demand result suggests 12:00-15:00 UTC would not help. Nothing on our side has been changed.