429s through API Management in front of Azure OpenAI, but not when calling the deployment directly

Question

429s through API Management in front of Azure OpenAI, but not when calling the deployment directly

Aria Rahujani 20

I’m running a batch job that sends sequential chat-completion requests to a gpt-5.1 deployment, roughly 2 requests per minute at about 4,700 prompt tokens and about 110 completion tokens each.

Calling the Azure OpenAI endpoint directly: no 429s.
Calling the same workload through an Azure API Management gateway (API imported for Azure OpenAI, attached to a product, subscription key required): nearly every request returns 429 on the first attempt and succeeds after the Retry-After delay.
Smaller requests (about 1,600 tokens each, 3 per minute) through the same gateway return no 429s.

What I’ve checked so far:

The API’s own policy has no llm-token-limit, azure-openai-token-limit, rate-limit or rate-limit-by-key element.

Questions:

Besides API-level policies, where else can a token or rate limit be applied (product, global, policy fragments, something configured through the AI Foundry gateway integration)? Is there a way to see the effective policy for one operation?
How can I tell whether a 429 was returned by API Management itself or passed through from the Azure OpenAI backend? Application Insights is enabled on the API.
If the API Management backend points at the same deployment as my direct endpoint, what could make the gateway’s effective throughput lower?I’m running a batch job that sends sequential chat-completion requests to a gpt-5.1 deployment, roughly 2 requests per minute at about 4,700 prompt tokens and about 110 completion tokens each.
- Calling the Azure OpenAI endpoint directly: no 429s.
- Calling the same workload through an Azure API Management gateway (API imported for Azure OpenAI, attached to a product, subscription key required): nearly every request returns 429 on the first attempt and succeeds after the Retry-After delay.
- Smaller requests (about 1,600 tokens each, 3 per minute) through the same gateway return no 429s.
What I’ve checked so far:
- The API’s own policy has no llm-token-limit, azure-openai-token-limit, rate-limit or rate-limit-by-key element.
Questions:
1. Besides API-level policies, where else can a token or rate limit be applied (product, global, policy fragments, something configured through the AI Foundry gateway integration)? Is there a way to see the effective policy for one operation?
2. How can I tell whether a 429 was returned by API Management itself or passed through from the Azure OpenAI backend? Application Insights is enabled on the API.
3. If the API Management backend points at the same deployment as my direct endpoint, what could make the gateway’s effective throughput lower?

REDDIMALLA VENKATESH 0 Reputation points Microsoft External Staff

2026-10-22T11:31:23.6833333+00:00
Hello @Aria Rahujani ,

Thank you for providing the detailed information and for sharing the comparison between direct Azure OpenAI requests and requests routed through Azure API Management.

I understand that the direct calls to the same deployment are completing successfully, while the larger requests routed through API Management are receiving HTTP 429 responses on the initial attempt and succeeding after the Retry-After interval. I also noted that smaller requests through the same gateway are not experiencing the issue.

To help isolate whether the 429 response is being generated at the API Management layer or returned by the Azure OpenAI backend, could you please share the following from an affected request:

Approximate UTC timestamp of the request

APIM request/correlation ID, if available

Complete 429 response headers and error response, with any sensitive information removed

The relevant APIM policy configuration, including inherited/global/product-level policies or policy fragments applicable to this API

Azure OpenAI deployment TPM configuration

Since Application Insights is already enabled, the telemetry from an affected request should also help us correlate the gateway request with the backend call.

Please ensure that subscription keys, API keys, access tokens, or other credentials are removed before sharing the information.

Thank you for your cooperation. We’ll use these details to narrow down where the throttling is occurring and determine the appropriate next steps.

1 answer

Your answer

REDDIMALLA VENKATESH 0 Reputation points Microsoft External Staff

2026-10-22T11:31:23.6833333+00:00

Hello @Aria Rahujani ,

Thank you for providing the detailed information and for sharing the comparison between direct Azure OpenAI requests and requests routed through Azure API Management.

I understand that the direct calls to the same deployment are completing successfully, while the larger requests routed through API Management are receiving HTTP 429 responses on the initial attempt and succeeding after the Retry-After interval. I also noted that smaller requests through the same gateway are not experiencing the issue.

To help isolate whether the 429 response is being generated at the API Management layer or returned by the Azure OpenAI backend, could you please share the following from an affected request:

Approximate UTC timestamp of the request

APIM request/correlation ID, if available

Complete 429 response headers and error response, with any sensitive information removed

The relevant APIM policy configuration, including inherited/global/product-level policies or policy fragments applicable to this API

Azure OpenAI deployment TPM configuration

Since Application Insights is already enabled, the telemetry from an affected request should also help us correlate the gateway request with the backend call.

Please ensure that subscription keys, API keys, access tokens, or other credentials are removed before sharing the information.

Thank you for your cooperation. We’ll use these details to narrow down where the throttling is occurring and determine the appropriate next steps.

Answer 1

Hi @Aria Rahujani

Your results match a token limit, not a request limit. The smaller 1,600 token job succeeds even at a higher request rate, and the larger 4,700 token job fails at a lower rate. If it were counting requests, the faster one would fail first.

1. Where else a limit can live

The llm-token-limit policy can be scoped at “global, workspace, product, API, operation”, and it can also appear inside a policy fragment. If you imported this API using the Microsoft Foundry wizard, the Manage token consumption tab lets you “optionally enter settings, or accept defaults that define” this policy. So a token limit can exist even if you did not add it manually.

2. Checking the effective policy for the operation

From Set or edit policies: “To view the effective policies at the current scope, select Calculate effective policy in the policy editor.”

Run this at the operation level. It resolves everything inherited above it.

3. Telling an APIM 429 from a backend 429

llm-token-limit runs inbound, so if it blocks, APIM returns 429 without calling the backend. forward-request is one of the policies that produces dependency telemetry. Compare End-to-end transaction details for a 429 and a 200 from the same minute. If the 200 shows a backend dependency and the 429 shows none, APIM generated the 429.

4. Why the gateway is slower than direct calls

The gateway enforces a counter your direct call never touches. The attribute table in the policy reference also notes: “For each key value, a single counter is used for all scopes at which the policy is configured”, so if the counter key is shared with another API or product, that other traffic consumes your token budget.

If step 2 shows no token policy anywhere, post the effective policy output and we can look at the deployment side.

Help make this community better for everyone: if this answer resolved your issue, please accept it or leave an upvote. If not, share more details in a comment so we can continue the discussion and find the right solution.