Cohere Rerank v4.0 Fast returns 500 error in Azure AI Foundry when using DefaultAzureCredential

Hello Arne Lieten,

Welcome to Microsoft Q&A and Thank you for reaching out.

In addition to inputs provided by Vinodh247 , please check if the following helps

In Azure AI Foundry, authentication support differs depending on whether a provider endpoint or a model deployment endpoint is invoked. Using Entra ID tokens against certain provider endpoints can result in internal service failures that surface as 500 errors.

Azure AI Foundry exposes two different endpoint patterns for Cohere models:

1. Provider Endpoint

      /providers/cohere/v2/rerank
   

   • Commonly expects API key–based authentication
   • Entra ID support may be limited or inconsistent
   • Valid Entra tokens may be accepted but fail during backend processing, resulting in a 500 error
2. Model deployment endpoint

/models/{deployment-name}/...

1. • Designed to support Entra ID (AAD) authentication
   • Recommended for identity‑based access using DefaultAzureCredential

Please check if the following troubleshooting steps help

1. Verify the Endpoint Being Used

Please check whether the request is sent to:

• /providers/cohere/v2/rerank - likely cause of the issue
• /models/{deployment-name} -preferred for Entra ID authentication

2. Try isolate authentication vs service behavior by invoking the same request using the project API key.If the API key succeeds, then

• There is no issue with model deployment
• Not a request payload issue
• Issue isolated to authentication compatibility

3. Validate request payload structure

At times even with correct authentication, malformed input may trigger server errors.

Please check to confirm if

• documents array is not empty
• query is a non‑null string
• Input size and document count are within supported limits

4. Confirm region and model availability

Cohere models are available only in specific regions.

So please check and validate the

• Project region supports Cohere Rerank v4.0 Fast
• Deployment is active and healthy in the selected region

References:

Authentication and authorization in Microsoft Foundry - Microsoft Foundry | Microsoft Learn

Deployment options for Microsoft Foundry Models (classic) - Microsoft Foundry (classic) | Microsoft Learn

Thank you!

Please 'Upvote'(Thumbs-up) and 'Accept' as answer if the reply was helpful. This will be benefitting other community members who face the same issue.

Hi ,

Thanks for reaching out to Microsoft Q&A.

A 500 Internal Server Error from Cohere Rerank v4.0 Fast in AI Foundry when using DefaultAzureCredential usually indicates a service-side request validation or routing issue rather than authentication. Since you already confirmed that incorrect credentials correctly return 401, authentication through Entra ID is working.

The most common reason is that Cohere provider endpoints in AI Foundry currently expect API-key based authentication, and AAD token-based auth is not fully supported for the providers/cohere/v2/rerank route in some regions. When a valid Entra token is passed, the request reaches the service but fails internally, resulting in a 500 instead of a proper authorization error.

In practice, this issue is typically resolved by using the project API key from AI Foundry instead of DefaultAzureCredential, or by routing the request through the Azure AI Foundry model endpoint that supports Entra authentication rather than the provider endpoint. If API keys are not acceptable in your environment, the recommended approach is to open an support ticket, as this behaviour usually indicates a provider integration limitation or backend bug in the Cohere connector rather than a client configuration problem.

Please 'Upvote'(Thumbs-up) and 'Accept' as answer if the reply was helpful. This will be benefitting other community members who face the same issue.