Getting error max_prompt_tokens only when generating files @ Assistants

Mohamed Hussein 215

Hi,

At Azure OpenAi Assistants, all runs keep failing when asking to generate file.

Error details Max prompt tokens

while other types of prompts are normally operating

 
  "id": "run_4BCHhcmo09UaoFdPPt3XQnyO",  "object": "thread.run",  "created_at": 1731734642,  "assistant_id": "asst_3rY50WWj2cvOXU5jjealB7qy",  "thread_id": "thread_ePJeuEUaUaWM6Mos8CA0iNUo",  "status": "incomplete"

navba-MSFT 25,595 Reputation points Microsoft Employee

2024-11-18T06:22:46.81+00:00
@Mohamed Hussein Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

Quotas and limits reference are explained in this article. Please be aware that the prompt tokens per request varies based on the model.

May I know which GPT model and version are you using ?

Which region is your Azure OpenaI resource deployed to ?

Please share the exact error message and the screenshot of the error.

Are you getting that error on the portal or in your application.

Are you using this feature from the SDK or REST API ?

Please share the complete Resource URI in below format (in private message): /subscriptions/XXXXX/resourceGroups/XXXXX/providers/Microsoft.CognitiveServices/accounts/XXXXXX

Share the File assistant ID (GUID) and Trace ID for the failing scenario

Awaiting your reply.
Mohamed Hussein 215 Reputation points

2024-11-18T13:27:57.67+00:00
Good Day,

Please find answers below

May I know which GPT model and version are you using ?

gpt-4o (version:2024-05-13)

Which region is your Azure OpenaI resource deployed to ?

EastUS

Please share the exact error message and the screenshot of the error.

The request does not throw error. The run returns incomplete status, and incomplete_details is max_prompt_tokens. As you know status now is expired

Are you getting that error on the portal or in your application.

In my application. I think it's related when total thread tokens hit the 2M limit.

Are you using this feature from the SDK or REST API ?

Both SDK and REST API

Please share the complete Resource URI in below format (in private message): /subscriptions/XXXXX/resourceGroups/XXXXX/providers/Microsoft.CognitiveServices/accounts/XXXXXX

how to send a private message?

Share the File assistant ID (GUID) and Trace ID for the failing scenario

The file not generated, there's no trace ID returned from API

Awaiting your reply.
navba-MSFT 25,595 Reputation points Microsoft Employee

2024-11-18T13:33:57.0966667+00:00

@Mohamed Hussein Thanks for sharing the details. I just sent you the private message. Please reply to that with Resource URI and the time when this issue occurred (in UTC).
navba-MSFT 25,595 Reputation points Microsoft Employee

2024-11-19T04:01:00.5166667+00:00

@Mohamed Hussein Could you please share the screenshot from the Quota Page within Azure Ai studio for both Azure OpenAI Standard and Global Standard for the GPT-4 model as shown below ?

Awaiting your reply.
navba-MSFT 25,595 Reputation points Microsoft Employee

2024-11-19T04:48:12.8866667+00:00
@Mohamed Hussein Thanks for sharing the screenshot over private message.

The Screenshot says the token as used 100%. Under the Quota--> Global Standard tab, Please Click on Request Quota and to increase this limit.

.

To answer your question about the usage, you can try to leverage the metrics as shown below:

Open the Azure Open AI metrics tab

Add the metric for Processed Inference Token

Apply splitting for API Name and Model Name and Usage Channel as shown below:

On the top change the time range:
Mohamed Hussein 215 Reputation points

2024-11-19T13:49:01.5766667+00:00
@navba-MSFT

The error when hitting the rate limit (token per minute) is

"code": "rate_limit_exceeded", "message": "Rate limit is exceeded. Try again in XXXX seconds."

For example at Assistants, the avg token per request per user is 35K token/ minute

Having only 5 concurrent users would easily hit the 400K limit

Having that said, the error related to max_prompt_token is raised when setting a hard limit to max prompt tokens per request. However, when set the truncation strategy to Auto, this error shouldn't be raised.

Anyway, i've submitted the request to increase the quota

Thanks for help..

1 answer

navba-MSFT 25,595 Reputation points Microsoft Employee

2024-11-20T04:54:23.4033333+00:00

@Mohamed Hussein Thanks for getting back. As I mentioned earlier, the action plan to request the quota increase should help.

Once the quota is approved, you should get an email.

Then you can test again and ensure to be within the quota limits, Also I am sharing some of the useful metrics which you can rely on.

Please leverage some of the metrics to check the usage of your tokens.

Hope this helps.

**

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.
Please sign in to rate this answer.
navba-MSFT 25,595 Reputation points Microsoft Employee

2024-11-21T02:47:51.7733333+00:00

@Mohamed Hussein This is a quick follow-up to check if you got an approval on your quota request and got a chance to test this further. If your request for quota increase gets approved, you can expect an email from Cognitive Services Gating Support [email protected] or Microsoft Azure OpenAI [email protected] . Please be aware that, there is no ETA on when this would be approved considering East US has high demand and a most congested region.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Getting error max_prompt_tokens only when generating files @ Assistants

1 answer

Your answer