Request for Investigation: Azure Application Gateway – Backend Traffic Distribution and 502 Errors in PROD VMs

Configure a custom health endpoint that returns a nonhealthy result when a VM cannot accept traffic, and disable cookie-based affinity if the application does not require server-local session state.

Application Gateway does not use VM CPU or memory load when selecting a backend. It distributes requests round-robin among servers whose configured probes report healthy; therefore, Prod_VM1 remains eligible while its probe returns HTTP 200, even when the application is overloaded. ^1^

1. Make the health probe overload-aware

Create a lightweight application endpoint such as /health/ready. It should:

• Return 200 only when the instance can accept new requests.
• Return 503 when critical worker queues, thread pools, connection pools, or required dependencies exceed application-defined thresholds.
• Avoid relying solely on whether the web process is running.
• Recover automatically to 200 after capacity returns.

Application Gateway removes a backend after the configured number of consecutive failed probes and adds it back when probes succeed again. Custom probes support a path, interval, timeout, unhealthy threshold, status matching, and response-body matching. ^2^

In Azure portal → Application Gateway → Health probes → Add:

1. Select the same HTTP/HTTPS protocol used by the backend.
2. Set the correct host name, particularly when HTTPS/SNI or virtual hosting is used.
3. Set Path to /health/ready.
4. Start with:
   • Interval: 30 seconds
   • Timeout: less than or equal to both the interval and backend request timeout
   • Unhealthy threshold: 2 or 3
5. Configure matching so only the readiness response is accepted—for example, status 200 and optionally a body token such as Ready.
6. Open Backend settings, select the setting used by the production rule, enable Use custom probe, and attach this probe.
7. Generate controlled load and verify Backend health changes Prod_VM1 to Unhealthy, while new traffic moves to Prod_VM2.

The portal supports these probe properties, but thresholds should be selected to avoid removing a VM because of a brief transient spike. ^3^

Do not configure MinServers above zero for this scenario. That setting can force a minimum number of backends to remain healthy regardless of probe failures and can result in traffic reaching an unhealthy server. ^2^

2. Check and, where possible, remove session affinity

Cookie-based affinity can explain why subsequent mobile requests continue reaching Prod_VM1. Application Gateway issues an affinity cookie and routes later requests carrying that cookie to the same backend. ^4^

Check the Application Gateway Backend setting used by the production routing rule:

• If Cookie-based affinity = Enabled, capture a mobile request and response and look for ApplicationGatewayAffinity on v2 or ARRAffinity on v1.
• Also inspect the mobile HTTP client's cookie jar. Some mobile libraries retain cookies even though no browser UI is involved.
• If sessions are stateless or stored in a shared service accessible to both VMs, set Cookie-based affinity = Disabled, save, and retest with a new session.
• If session state exists only on each VM, do not disable affinity until that state is moved to a shared store. Otherwise, users can lose session continuity.

Affinity can also continue directing requests to an instance being deregistered, so it should not be used as the overload-management mechanism. ^4^

3. Classify the 502 before changing timeouts

Since backend configuration and basic health are already reviewed, correlate each 502 with the access-log fields serverResponseLatency, serverStatus, backend address, URI, and transaction/trace identifier:

• Latency equals the configured backend timeout: the VM accepted the connection but did not respond in time; investigate application saturation on that VM.
• serverStatus contains a backend 5xx: the application generated the error and Application Gateway relayed it.
• Latency is 0 with no serverStatus: the gateway rejected the request before reaching a usable backend.
• Errors coincide with CapacityUnits reaching the autoscale maximum: Application Gateway itself may be saturated; increase its autoscale ceiling or instance capacity.
• ERRORINFO_UPSTREAM_CLOSED_CONNECTION: examine the backend web-server/application logs for connections closed before a complete response was returned. A similar case was resolved at the backend rather than by changing gateway distribution. ^5^

The official 502 decision guide documents these access-log fingerprints and the gateway-capacity check. ^6^

If logs are not already sent to Log Analytics, use Application Gateway → Diagnostic settings → Add diagnostic setting, select ApplicationGatewayAccessLog, and send it to a Log Analytics workspace using resource-specific tables. AGWAccessLogs is the corresponding table; WAF deployments should also collect firewall logs. ^7^

A starting query is:

AGWAccessLogs
| where TimeGenerated > ago(2h)
| where HttpStatus == 502
| project TimeGenerated, ClientIp, RequestUri,
          BackendPoolName, BackendSettingName,
          ServerStatus, ServerResponseLatency, ErrorInfo
| order by TimeGenerated desc

Adjust column capitalization to the schema displayed by the workspace. Correlate the timestamps with CPU, memory, request queue, connection pool, thread-pool, and application logs on each VM. Increasing the request timeout alone may hide overload and prolong user-visible waits; use it only when measured healthy requests legitimately require more time.

References

1. How an application gateway works
2. Application Gateway health probes overview
3. Create a custom probe for Application Gateway by using the portal
4. Application Gateway backend settings configuration
5. How to troubleshoot 502 issue on Application gateway - Microsoft Q&A
6. Troubleshoot HTTP 502 errors in Azure Application Gateway
7. Diagnostic logs for Application Gateway