Custom vision ai is stuck in training days after starting it with just 8h budget

I think the service does not guarantee how long training takes, and if it stays the same for days, support is the place to check. You can keep going by starting a new iteration.

Hello @Holčiak Michal

Thank you for reaching out to Microsoft Q&A.

I understand that your Azure AI Custom Vision training job has been stuck in “Training” for several hours/days after starting an advanced training run (8-hour budget), and you’re also unable to create new iterations. That’s not expected behavior, so let’s go through the most likely causes and what you can do.

1. High training load / backend queue delays

In many cases, this behavior is due to high service demand or resource availability in the backend.

• During peak usage, many customers may be submitting training jobs simultaneously
• This can cause jobs to stay in queue longer than usual
• Earlier iterations may have finished quickly, but later ones can slow down due to system load

In most cases, the job is not stuck it is simply waiting in the training queue.

2. No manual cancel option

Once training starts:

There is currently no option to manually stop or cancel a training job

The system will either:

• Complete the training, or
• Stop automatically when the budget limit is reached

You will not be billed beyond the configured training budget, even if the job appears to run for a long time.

3. Region or capacity considerations try Sweden Central

Training performance can vary depending on regional capacity.

• If possible, try using or redeploying in Sweden Central
• Sweden Central is generally a stable region for AI workloads

If your current region is experiencing capacity constraints, switching to Sweden Central can often reduce delays.

4. Free tier or quota limitations

If you are using the free tier (F0), there are important constraints:

• Limited training hours per month
• Restricted training operations and dataset limits
• Can cause training delays or blocked iterations if quota is exhausted

What to do:

• Check your Training & Prediction resource tier in Azure Portal
• Upgrade to Standard (S0) if needed for consistent training performance

5. Training budget is not a time guarantee

The 8-hour budget means:

• A compute usage cap, not a guaranteed execution window
• It does not guarantee completion within 8 hours
• Large datasets or complex models may still take longer depending on backend availability

6. Dataset-related issues

Training can also appear stuck if there are dataset problems:

• Corrupt or unsupported image formats
• Very large or unbalanced datasets
• Missing or incorrect tags
• Images skipped during preprocessing

Suggested check:

• Validate dataset integrity
• Remove suspicious images
• Try a small dataset (10–20 images) for a new iteration test

Please refer this

Training job queueing / waiting behavior (fine-tuning guidance): https://learn.microsoft.com/azure/ai-services/openai/how-to/fine-tuning

Image vision fine-tuning troubleshooting (skipped images due to content): https://learn.microsoft.com/azure/foundry-classic/openai/how-to/fine-tuning-troubleshoot

Custom Vision “Train your model” (resource/pricing setup context): https://learn.microsoft.com/windows/ai/windows-ml/tutorials/image-classification-train-model

I Hope this helps. Do let me know if you have any further queries.

If this answers your query, please do click Accept Answer and Yes for was this answer helpful.

Thank you!