Azure Foundry subcription & latency

Question

Azure Foundry subcription & latency

Rollo S 0

Im using Azure Foundry with basic global subscription. And i'm noticing that my LLM calls are very slow. I dont use it much at this time, but definitely needed to boost a bit the speed to make it work for the users. What can I do to speed it up?

0 comments

2 answers

Your answer

Answer 1

Hi @Rollo S ,

If you're using Azure Foundry with a Standard/Global Standard deployment, I'd first check whether the latency is coming from the model itself or from the amount of input/output tokens you're sending.

A few things that usually help:

Try a smaller/faster model if the workload doesn't require the larger model.
Reduce unnecessary context and limit the maximum output tokens.
Enable streaming so users start seeing the response immediately instead of waiting for the complete response.
Check the Foundry monitoring metrics such as Time to First Token and token processing time to see where the delay is happening.
If the latency is mainly due to shared capacity and you need more predictable performance, you can look at Provisioned Throughput (PTU).

Microsoft has a good guide on this here: Azure OpenAI latency

And for predictable throughput/latency: Provisioned Throughput

If you're only using the service occasionally at the moment, I wouldn't move to PTU immediately. I'd first check the model, prompt size, output tokens, and streaming. Those can make a noticeable difference without changing your deployment.

Rollo S 0 Reputation points

2026-09-22T16:49:14.79+00:00

thanks for the response Aditya. Whether the prompt (input tokens) is small or large, it seems it is still the same latency. So maybe it's model - currently using 5.4. Maybe I will try the mini on some of the less demanding/complex calls.

The ave latency is 19-20s for a simple chat turn making it really noticeable for the users. Maybe the foundry layer adds to the latency. Is there any config in the foundry that adds to the runtime?

Answer 2

To reduce Azure AI Foundry / Azure OpenAI latency, focus on the factors that most directly affect per-call response time:

Use a lower-latency model If lowest latency is the priority, the recommended model is the latest GPT-4o mini.
Reduce max_tokens A higher max_tokens value increases latency because the service reserves compute based on the requested generation size. Set max_tokens as low as possible for each call.
Reduce total generated tokens Response generation is typically the slowest step. To reduce latency:
- keep prompts shorter when possible
- add stop sequences to prevent extra output
- avoid multiple outputs per request by leaving n unset or setting n: 1
Enable streaming Set stream: true so tokens are returned as soon as they are available. This does not reduce total completion time, but it improves time to first response and usually makes the app feel much faster to users.
Separate different workloads If different request types share one endpoint, latency can increase because short requests may wait behind longer ones and cache efficiency can drop. Use separate deployments for different workloads when possible.
Test batching carefully If multiple requests go to the same endpoint, batching them into a single call can reduce request overhead and may improve overall response time depending on the scenario.
Review content filtering impact Content filtering improves safety but adds latency. For lower-risk workloads, evaluate whether modified content filtering policies are appropriate.
Measure before changing settings Track these Azure Monitor metrics on the deployment:
- Requests metric, split by ModelDeploymentName, to measure calls per minute
- Processed Inference tokens metric, to measure total tokens per minute

A practical optimization path is:

Lower max_tokens.
Set n: 1.
Enable stream: true.
Shorten prompts and add stop sequences.
If latency is still too high, test a lower-latency model such as GPT-4o mini.
Then review workload separation and monitoring metrics.