Microsoft Foundry Evaluation stuck at “In progress” for >1 hour (old UI ai.azure.com) – no execution starts

Question

Microsoft Foundry Evaluation stuck at “In progress” for >1 hour (old UI ai.azure.com) – no execution starts

Tomohiro Kamei 0

Hi all,

I’m troubleshooting an issue in Microsoft Foundry (resource type: Microsoft.CognitiveServices/accounts, kind=AIServices, project_management_enabled=true) where running an Evaluation from the old UI (ai.azure.com) gets stuck.

Symptoms

When I start an Evaluation from the old UI, the status remains “In progress” (Japanese UI: 「開始中」).
It stays in this state for more than 1 hour and appears that no job actually starts (no progress, no output).
The UI does not show a clear error message.

Environment (sanitized)

Region: Japan East
Resource type: AIServices under Microsoft.CognitiveServices
Foundry project exists under the account (default project).
Model deployment: gpt-4.1-mini (OpenAI format), SKU GlobalStandard (model version 2025-04-14)

Networking settings (Cognitive Account)

public_network_access_enabled = true
Network ACLs configured as:
- default_action = Deny
- ip_rules allow only our corporate egress (Zscaler) IP ranges
- bypass = AzureServices

Encryption / Identity (possible relevance)

Using Customer Managed Key (CMK) with our own Key Vault key (BYOK)
Managed identity configuration includes System Assigned + User Assigned
Storage account is configured/attached for the account (BYO storage / storage dependency exists)

What I’m looking for

Are there known cases where Evaluation jobs remain “In progress” indefinitely in the old Foundry UI?
Could the combination of Network ACL default deny (allowing only Zscaler IPs) + AzureServices bypass cause the backend evaluation job to be blocked even though public network access is enabled?
I found guidance suggesting CMK may not be supported in basic setups, and also that user-assigned managed identity may be unsupported with CMK during provisioning. Could this affect Evaluation execution (even if the resource already exists)?

If you have any recommended troubleshooting steps (e.g., temporarily switching to system-assigned identity only, relaxing ACLs to test, verifying connectivity/FQDN requirements, etc.), please share.

Thanks in advance.

1 answer

Your answer

Answer 1

Alex Burlachenko 19,530 Volunteer Moderator

Hi @Tomohiro Kamei,

this symptom where evaluation stays in progress indefinitely is usually caused by networking or identity configuration rather than a ui defect. when using aiservices with project management enabled the evaluation job runs on microsoft managed infrastructure and must be able to reach the cognitive account endpoint and any attached storage account. if ur network acls are set to default deny and only allow corporate zscaler ips then the backend evaluation compute may be blocked even though public network access is enabled. if u are using byo storage and that storage account has firewall rules enabled the evaluation pipeline may hang if azure services are not allowed.

when customer managed keys are enabled the managed identity of the cognitive account must have unwrap or decrypt permissions on the key vault key and misconfigured identity can prevent execution. Rbac https://learn.microsoft.com/azure/role-based-access-control/overview. i would temporarily relax the network acl to allow azure services and test whether the evaluation starts, verify that the system assigned identity has correct permissions on key vault and storage, and enable diagnostic logs to see whether authorization failures occur at job start.

In most cases this behaviour is caused by blocked backend connectivity or missing key permissions rather than a foundry execution engine fault.

rgds,

Alex

Tomohiro Kamei 0 Reputation points

2026-02-18T08:29:52.69+00:00
Hi Alex,

Thanks for the pointers. I ran a controlled A/B test in another Azure environment and compared Storage diagnostics in Log Analytics. The behavior looks strongly correlated with network ACLs on the AI Services (Foundry) account, not with Storage permissions.

What I tested

Case A: Foundry project with Storage attached, no Network ACLs on the AI Services account

Evaluation completes successfully.

The evaluation result JSON is written to the attached Storage account.

Storage logs during the run show expected operations, e.g.:

CreateContainer

PutBlob (x3)

ListBlobs

BlobPreflightRequest

GetBlob

This suggests the Foundry → Storage access path and permissions are generally OK.

Case B: Same setup, but Network ACLs enabled on the AI Services account (default deny; only my PC public IP allowed)

Evaluation stays at “In progress” indefinitely.

No new objects are created in the attached Storage account.

Storage diagnostics show no new requests at all after starting the evaluation (no 401/403 either).

What seems odd / question

If the evaluation runs on Microsoft-managed infrastructure, it may be calling the AI Services account endpoint from a Microsoft backend that does not originate from my client IP. In that case, restricting the AI Services account to only my PC IP would block the backend before it even reaches Storage (consistent with “no storage requests observed”).

However, we do have a requirement to restrict inbound access (e.g., allow-list corporate egress IPs / Zscaler). So my questions are:

Is it expected/required that Network ACLs on the AI Services (Foundry) account must allow “Azure services / Microsoft-managed backend” traffic for evaluations to run?

Is it expected that Evaluation cannot work with an IP allow-list–only posture on the AI Services (Foundry) account (e.g., default deny + allow only corporate egress IPs)?

If IP allow-listing is inherently incompatible because the evaluation runs on Microsoft-managed infrastructure, is the supported secure approach to use Private Endpoint / VNet integration for the AI Services account (and the attached Storage), so that the backend traffic stays within private networking?

Are there known logs for the evaluation backend failing to reach the AI Services account (since neither the UI nor Storage logs show an auth/network error)?

Any guidance on the correct networking model for Evaluations under AIServices + project management would be appreciated.

Regards,
Tomohiro
Tomohiro Kamei 0 Reputation points

2026-02-19T04:39:50.1266667+00:00
Hi Alex,

Quick update — I enabled diagnostic logs on the AIServices (Microsoft Foundry) account and monitored request history while running an Evaluation.

What I observed

When Network ACL = default deny and only my client/corporate egress IPs are allow-listed, I see requests hitting the AIServices endpoint from unexpected public IP addresses (not in my allow-list), and they return 40x. The operations include:

Evaluation_GetList

Get the specific version of the DatasetVersion

Create or start a pending upload of evaluation results for a specific version

Update the uploaded result to an evaluation run

If I add those source IPs to the allow-list, the Evaluation starts and completes.

Additional note (service tag mapping)

I checked the published Azure IP Ranges and Service Tags (Public Cloud) JSON and those source IPs appear to fall under the regional service tag AzureCloud.JapanEast (which is updated regularly). However, allow-listing the entire AzureCloud.JapanEast range is not realistic for us — it’s too broad and defeats the purpose of restricting inbound access by IP.

Also, “allow trusted Microsoft services” is already enabled on our side, so this does not look like a simple misconfiguration of that toggle.

Questions

Are these requests expected to originate from Microsoft-managed evaluation infrastructure (rather than the user’s browser/client), and therefore not match a client IP allow-list?

Do you know what service/component typically appears as the caller for these operations (Evaluation list, dataset version read, result upload/update)? From the diagnostics I only see public source IPs (which map broadly to AzureCloud.JapanEast), but that range is too large to allow-list safely. If there is a way to identify the actual originating service (e.g., service tag, resource/provider name, workload type, or any documented pattern), we could design an allow rule around that instead of chasing individual IPs.

Related: is there any recommended way to determine the origin (for example, specific log fields, correlation IDs to follow, or other telemetry) so we can apply the correct network access control?

Any guidance on the expected networking model for Microsoft Foundry Evaluations would be appreciated.

Regards, Tomohiro

Share via

Microsoft Foundry Evaluation stuck at “In progress” for >1 hour (old UI ai.azure.com) – no execution starts

1 answer

What I tested

What seems odd / question

What I observed

Additional note (service tag mapping)

Questions

Your answer