Azure Foundry subcription & latency

To reduce Azure AI Foundry / Azure OpenAI latency, focus on the factors that most directly affect per-call response time:

1. Use a lower-latency model If lowest latency is the priority, the recommended model is the latest GPT-4o mini.
2. Reduce max_tokens A higher max_tokens value increases latency because the service reserves compute based on the requested generation size. Set max_tokens as low as possible for each call.
3. Reduce total generated tokens Response generation is typically the slowest step. To reduce latency:
   • keep prompts shorter when possible
   • add stop sequences to prevent extra output
   • avoid multiple outputs per request by leaving n unset or setting n: 1
4. Enable streaming Set stream: true so tokens are returned as soon as they are available. This does not reduce total completion time, but it improves time to first response and usually makes the app feel much faster to users.
5. Separate different workloads If different request types share one endpoint, latency can increase because short requests may wait behind longer ones and cache efficiency can drop. Use separate deployments for different workloads when possible.
6. Test batching carefully If multiple requests go to the same endpoint, batching them into a single call can reduce request overhead and may improve overall response time depending on the scenario.
7. Review content filtering impact Content filtering improves safety but adds latency. For lower-risk workloads, evaluate whether modified content filtering policies are appropriate.
8. Measure before changing settings Track these Azure Monitor metrics on the deployment:
   • Requests metric, split by ModelDeploymentName, to measure calls per minute
   • Processed Inference tokens metric, to measure total tokens per minute

A practical optimization path is:

1. Lower max_tokens.
2. Set n: 1.
3. Enable stream: true.
4. Shorten prompts and add stop sequences.
5. If latency is still too high, test a lower-latency model such as GPT-4o mini.
6. Then review workload separation and monitoring metrics.

• Performance and latency
• Performance and latency
• Performance and latency