Very slow startup of an app slot (high CPU)

Question

Very slow startup of an app slot (high CPU)

Josh Levett 20

We've recently been having issues with slot deployments to our UAT environment timing out.
I'm not aware of any configuration changes that were made before the issue started occurring, but we do see a significant increase in the CPU usage from the starting slot.

Our CD pipeline is effectively broken as no deployments can take place, we have to wait for hours before then manually swapping the slot.

In this graph you can see the difference between the running slot (pink) and the staging slot (blue) User's image

I'm not seeing any errors in the log stream during this time either, and the resource the slots use doesn't exceed 70% CPU

Praneeth Maddali 2,265 Reputation points Microsoft External Staff Moderator

2025-11-24T12:35:01.7466667+00:00
Hi @Josh Levett

Thank you for your detailed post and the clear CPU graph—it really helps illustrate the issue.

What you're experiencing is quite common when a newly deployed staging slot takes a long time to complete application startup and initialization. Although the CPU usage doesn't exceed around 70%, the warm-up phase is so CPU-intensive that the slot isn't ready in time for the automatic swap, causing the deployment to eventually time out in your pipeline.

These steps typically resolve the issue, listed by their effectiveness:

Enable “Preload enabled” and set up a custom warm-up This instructs App Service to begin warming up the staging slot right after deployment, prior to starting the swap.

Go to the Azure portal >Your App Service > Configuration>General settings, and set Preload enabled to On.

Next, add these two app settings to the staging slot (and to production as needed).
WEBSITE_SWAP_WARMUP_PING_PATH = / WEBSITE_WARMUP_PATH = /healthz (or any endpoint that triggers your startup logic)

Please enable the Always On feature by navigating to Configuration > General settings. This will ensure the worker does not become idle immediately after starting.

Consider giving the deployment a bit more time. While the previous steps address the root cause, it's also helpful to increase the timeout in your pipeline to prevent failures during warm-up testing. In Azure DevOps, you can adjust the “Deployment timeout (in seconds)” field—try setting it to 3600 (1 hour) temporarily.

Reference :

https://learn.microsoft.com/en-us/azure/app-service/deploy-staging-slots?tabs=portal#warm-up-the-slot-before-swap

https://learn.microsoft.com/en-us/azure/app-service/configure-common?tabs=portal#configure-general-settings

Kindly let us know if the above comment helps or you need further assistance on this issue.
Josh Levett 20 Reputation points

2025-11-25T12:27:37.02+00:00
@Praneeth Maddali

I can't see a preload setting in that page

I have turned this on, but I am still seeing the issue

like I said, it does eventually work, but it can take an hour before the service comes online which breaks the CD pipeline as the app will never automatically deploy

It looks like the application restarts several times before it eventually works, as it will flip between returning timeout and service unavailable errors.
I even saw an error very similar to the one described here https://www.rickvandenbosch.net/blog/cannot-createshadow-copy-your-assembly-info-here-when-that-file-already-exists/

I have enabled crash dumps to try and see if that identifies any issues, but the last deployment there was this error.
CrashDumpAnalyzer.exe crashed with Exit Code - 0xE0434352

you said 'Although the CPU usage doesn't exceed around 70%' I didn't give any percentages, and in fact it consistently sits at 100% CPU during deployments, we have already scaled up the instance size to try and accommodate this, but it seemingly hasn't helped.
Again, this was working fine until around the 7th of this month (on the reduced instance size).

(your response also feels rather AI generated, or at least formulaic)
Josh Levett 20 Reputation points

2025-11-25T12:43:46.3133333+00:00

Also, the CPU drill down shows a total CPU usage of 96.66% but the apps in the graph only add up to 56.29%

This is a consistent gap for each deployment
Praneeth Maddali 2,265 Reputation points Microsoft External Staff Moderator

2025-11-26T01:56:07.1966667+00:00

HI @Josh Levett

Thank you for your feedback and for sharing additional details so we can move forward. I’ve sent you a private message requesting the few details for Reviewing the backend logs will help determine if this is the “shadow-copy + Run-From-Package file-lock” issue that’s causing the high CPU usage and restarts. When you have a chance, please check private message and provide the requested information. Once I receive it, I’ll review and either confirm the fix or apply a temporary solution as needed.
Praneeth Maddali 2,265 Reputation points Microsoft External Staff Moderator

2025-12-02T06:16:25.7833333+00:00

HI @Josh Levett

I'm just reaching out to see if your issue has been resolved or if you've had a chance to review my previous comment?
Josh Levett 20 Reputation points

2025-12-02T14:29:45.5666667+00:00

Hi @Praneeth Maddali

It has definitely helped, I'm not seeing the failure rate that there was before.

There was one failure with the log saying 'operation 'POST/SUBSCRIPTIONS/RESOURCEGROUPS/PROVIDERS/MICROSOFT.WEB/SITES/SLOTS/SLOTSSWAP' exceeded the backend storage limit. Please try again after '6' seconds. (CODE: 429)'
but the next run did work correctly.

Thanks for your help
Praneeth Maddali 2,265 Reputation points Microsoft External Staff Moderator

2025-12-03T01:01:30.25+00:00

HI @Josh Levett

Thank you for confirming the improvement. I'm glad to hear that the failure rate has dropped significantly compared to previous runs.

About the single failure, the log shows: operation 'POST/SUBSCRIPTIONS/RESOURCEGROUPS/PROVIDERS/MICROSOFT.WEB/SITES/SLOTS/SLOTSSWAP' exceeded the backend storage limit. Please try again after '6' seconds. (CODE: 429)

This error usually means the backend storage was temporarily throttled. Since the next run succeeded, it appears the issue was brief and resolved on its own.

No further action is needed now, but please let us know if failures happen again so we can look into it.
Josh Levett 20 Reputation points

2025-12-04T16:16:15.8233333+00:00

Hi @Praneeth Maddali

I have a strange development.

The staging slot builds only appear to work when WEBSITE_RUN_FROM_PACKAGE = 1
But the production slot only works when WEBSITE_RUN_FROM_PACKAGE = 0

This would be fine except we can't swap the two because it will first try to build the staging slot with WEBSITE_RUN_FROM_PACKAGE = 0 which will time out & fail.

We are in the process of trying one of the other methods you mentioned in order to circumvent this, but I wasn't sure if this was something you might have seen elsewhere

Answer accepted by question author

0 additional answers

Your answer

Praneeth Maddali 2,265 Reputation points Microsoft External Staff Moderator

2025-11-24T12:35:01.7466667+00:00

Hi @Josh Levett

Thank you for your detailed post and the clear CPU graph—it really helps illustrate the issue.

What you're experiencing is quite common when a newly deployed staging slot takes a long time to complete application startup and initialization. Although the CPU usage doesn't exceed around 70%, the warm-up phase is so CPU-intensive that the slot isn't ready in time for the automatic swap, causing the deployment to eventually time out in your pipeline.

These steps typically resolve the issue, listed by their effectiveness:

Enable “Preload enabled” and set up a custom warm-up This instructs App Service to begin warming up the staging slot right after deployment, prior to starting the swap.

Go to the Azure portal >Your App Service > Configuration>General settings, and set Preload enabled to On.

Next, add these two app settings to the staging slot (and to production as needed).
WEBSITE_SWAP_WARMUP_PING_PATH = / WEBSITE_WARMUP_PATH = /healthz (or any endpoint that triggers your startup logic)

Please enable the Always On feature by navigating to Configuration > General settings. This will ensure the worker does not become idle immediately after starting.

Consider giving the deployment a bit more time. While the previous steps address the root cause, it's also helpful to increase the timeout in your pipeline to prevent failures during warm-up testing. In Azure DevOps, you can adjust the “Deployment timeout (in seconds)” field—try setting it to 3600 (1 hour) temporarily.

Reference :

https://learn.microsoft.com/en-us/azure/app-service/deploy-staging-slots?tabs=portal#warm-up-the-slot-before-swap

https://learn.microsoft.com/en-us/azure/app-service/configure-common?tabs=portal#configure-general-settings

Kindly let us know if the above comment helps or you need further assistance on this issue.
Josh Levett 20 Reputation points

2025-11-25T12:27:37.02+00:00

@Praneeth Maddali

I can't see a preload setting in that page

I have turned this on, but I am still seeing the issue

like I said, it does eventually work, but it can take an hour before the service comes online which breaks the CD pipeline as the app will never automatically deploy

It looks like the application restarts several times before it eventually works, as it will flip between returning timeout and service unavailable errors.
I even saw an error very similar to the one described here https://www.rickvandenbosch.net/blog/cannot-createshadow-copy-your-assembly-info-here-when-that-file-already-exists/

I have enabled crash dumps to try and see if that identifies any issues, but the last deployment there was this error.
CrashDumpAnalyzer.exe crashed with Exit Code - 0xE0434352

you said 'Although the CPU usage doesn't exceed around 70%' I didn't give any percentages, and in fact it consistently sits at 100% CPU during deployments, we have already scaled up the instance size to try and accommodate this, but it seemingly hasn't helped.
Again, this was working fine until around the 7th of this month (on the reduced instance size).

(your response also feels rather AI generated, or at least formulaic)
Josh Levett 20 Reputation points

2025-11-25T12:43:46.3133333+00:00

Also, the CPU drill down shows a total CPU usage of 96.66% but the apps in the graph only add up to 56.29%

This is a consistent gap for each deployment
Praneeth Maddali 2,265 Reputation points Microsoft External Staff Moderator

2025-11-26T01:56:07.1966667+00:00

HI @Josh Levett

Thank you for your feedback and for sharing additional details so we can move forward. I’ve sent you a private message requesting the few details for Reviewing the backend logs will help determine if this is the “shadow-copy + Run-From-Package file-lock” issue that’s causing the high CPU usage and restarts. When you have a chance, please check private message and provide the requested information. Once I receive it, I’ll review and either confirm the fix or apply a temporary solution as needed.
Praneeth Maddali 2,265 Reputation points Microsoft External Staff Moderator

2025-12-02T06:16:25.7833333+00:00

HI @Josh Levett

I'm just reaching out to see if your issue has been resolved or if you've had a chance to review my previous comment?
Josh Levett 20 Reputation points

2025-12-02T14:29:45.5666667+00:00

Hi @Praneeth Maddali

It has definitely helped, I'm not seeing the failure rate that there was before.

There was one failure with the log saying 'operation 'POST/SUBSCRIPTIONS/RESOURCEGROUPS/PROVIDERS/MICROSOFT.WEB/SITES/SLOTS/SLOTSSWAP' exceeded the backend storage limit. Please try again after '6' seconds. (CODE: 429)'
but the next run did work correctly.

Thanks for your help
Praneeth Maddali 2,265 Reputation points Microsoft External Staff Moderator

2025-12-03T01:01:30.25+00:00

HI @Josh Levett

Thank you for confirming the improvement. I'm glad to hear that the failure rate has dropped significantly compared to previous runs.

About the single failure, the log shows: operation 'POST/SUBSCRIPTIONS/RESOURCEGROUPS/PROVIDERS/MICROSOFT.WEB/SITES/SLOTS/SLOTSSWAP' exceeded the backend storage limit. Please try again after '6' seconds. (CODE: 429)

This error usually means the backend storage was temporarily throttled. Since the next run succeeded, it appears the issue was brief and resolved on its own.

No further action is needed now, but please let us know if failures happen again so we can look into it.
Josh Levett 20 Reputation points

2025-12-04T16:16:15.8233333+00:00

Hi @Praneeth Maddali

I have a strange development.

The staging slot builds only appear to work when WEBSITE_RUN_FROM_PACKAGE = 1
But the production slot only works when WEBSITE_RUN_FROM_PACKAGE = 0

This would be fine except we can't swap the two because it will first try to build the staging slot with WEBSITE_RUN_FROM_PACKAGE = 0 which will time out & fail.

We are in the process of trying one of the other methods you mentioned in order to circumvent this, but I wasn't sure if this was something you might have seen elsewhere

Answer 1

HI @Josh Levett

Thank you for sharing additional information in the private chat. Based on your input, now have a clear understanding of the situation from our deep investigation.

What’s happening

Your staging slot is experiencing a known .NET and App Service file-locking issue that occurs when the site runs from a package (ZIP deploy / Run-From-Package). During startup, IIS attempts to shadow-copy assemblies, but if DLLs are locked, shadow-copy fails. As a result, IIS restarts the w3wp process, causing the slot to remain in a “Stopped” state for several minutes, which leads to repeated auto-swap failures, 100% CPU spikes, and long delays.

The symptoms you mentioned match this issue:

The 40% “missing” CPU is due to the staging w3wp not being fully registered
The slot frequently switches between Stopped and running states
CrashDumpAnalyzer crashes because the dump file is locked by the hung worker process

To resolve the issue promptly, implementing any of these three options will provide an immediate solution

The recommended and simplest approach is to disable Run-From-Package on the staging slot, and on the production slot if desired. Please add the following app setting to both slots.

WEBSITE_RUN_FROM_PACKAGE = 0

This adjustment allows the package to extract as expected, rather than remaining read-only, and fully resolves the locking issue.
If you wish to retain Run-From-Package for its advantages, such as supporting immutable deployments, please include the following:

MSDEPLOY_RENAME_LOCKED_FILES = 1

MSDeploy will rename any locked files rather than causing the deployment to fail.
To completely disable shadow-copying, regardless of whether Run-From-Package is used, please add the following setting to your web.config file:
```
   <hostingEnvironment shadowCopyBinAssemblies="false" />
```

Reference :

https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/iis/?view=aspnetcore-8.0#shadow-copy

https://learn.microsoft.com/en-us/azure/app-service/deploy-run-package

https://learn.microsoft.com/en-us/azure/app-service/deploy-best-practices#handle-locked-files-during-deployment

https://learn.microsoft.com/en-us/azure/app-service/troubleshoot-http-502-http-503

Kindly let us know if the above comment helps or you need further assistance on this issue.

Please "upvote" if the information helped you. This will help us and others in the community as well.

Share via

Very slow startup of an app slot (high CPU)

0 additional answers

Your answer