Azure AI Document Intelligence – Detailed evaluation metrics for custom classifier and extraction models

Lizos Christos 0 Reputation points
2026-06-26T07:58:23.8566667+00:00

Hello Microsoft Team,

We are using Azure AI Document Intelligence with the following flow:

Document upload

→ Custom classifier model

→ Routing to the corresponding custom extraction model based on the predicted document class

→ Field extraction and comparison with business data

We would like to understand whether Azure AI Document Intelligence provides any built-in report, API endpoint, or portal functionality that gives detailed model evaluation metrics beyond confidence scores and estimated accuracy values.

More specifically, we would like to know whether there is a native way to retrieve metrics such as:

  • Classification accuracy per document type
  • Confusion matrix for the custom classifier
  • Precision, recall, and F1-score
  • Field-level extraction accuracy
  • Missing field count / incorrect extraction count
  • Comparison between model versions
  • Accuracy trends over time based on validation data or production usage

From the documentation, we understand that analyze responses provide confidence scores, and some model types may provide estimated accuracy. However, confidence scores are not the same as actual evaluation statistics against ground-truth data.

Could you please confirm:

  1. Does Azure AI Document Intelligence provide built-in detailed evaluation reports for custom classifier and custom extraction models?
  2. If not, what is the Microsoft-recommended approach for calculating these metrics?
  3. Should we build an external validation process where we run a labeled test set through the classifier/extraction models and compare the Azure output with expected ground-truth values?
  4. What information should we log from production analyze responses in order to monitor model quality over time?
  5. Can Azure Monitor, Application Insights, or Log Analytics be used for model-quality reporting, or are they intended mainly for operational metrics such as processed pages, latency, errors, and usage?
  6. What is the recommended way to compare two model versions before switching production traffic to the newer model ID?

Our goal is to have a reliable reporting process that allows us to decide whether a retrained model is better than the current production model before changing the model ID used by our application.

Thank you.

Azure Document Intelligence in Foundry Tools

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Anshika Varshney 14,095 Reputation points Microsoft External Staff Moderator
    2026-07-02T19:42:13.2866667+00:00

    Hello @Lizos Christos

    Thank you for your question.

    Based on the currently available Azure AI Document Intelligence documentation, there is no documented built-in feature that provides comprehensive evaluation reports such as:

    • Confusion matrices
    • Precision, Recall, or F1 scores
    • Per-class classification accuracy
    • Field-level extraction accuracy against ground-truth data
    • Detailed comparison reports between model versions

    Azure AI Document Intelligence primarily provides prediction results and confidence scores as part of the analysis response.

    Evaluation Approach:-

    For scenarios requiring detailed model-quality assessment, the recommended approach is to maintain a labeled test dataset and perform external validation by comparing model predictions with ground-truth values.

    This allows you to calculate metrics such as:

    • Classification accuracy
    • Precision, Recall, and F1 score
    • Confusion matrices
    • Field-level extraction accuracy
    • Missing versus incorrectly extracted fields
    • Model-version comparisons

    Monitoring Models in Production:-

    Azure Monitor and diagnostic logging can be used to track operational metrics and identify errors, warnings, or anomalies after deployment. However, based on the available documentation, these services do not natively generate ground-truth-based evaluation metrics such as confusion matrices or field-level accuracy reports.

    For ongoing quality monitoring, many organizations log:

    • Predicted document type
    • Extracted field values
    • Confidence scores
    • Processing failures or exceptions

    and compare them periodically against manually validated samples.

    Comparing Model Versions:-

    The documented best practice is to evaluate candidate and production models against the same labeled validation dataset and compare the resulting metrics before promoting a newer model version to production.

    References:-

    If you are looking for a specific evaluation workflow, could you confirm whether you are using a custom classifier, a custom extraction model, or a combination of both?

    I hope this helps you get back on track! If you're still facing issues, could you share more details?

    Thankyou!

    Was this answer helpful?

    0 comments

  2. AI answer

    2026-06-26T07:59:02.0466667+00:00
    1. Azure AI Document Intelligence does not provide built‑in detailed evaluation reports such as confusion matrices, precision/recall/F1, or longitudinal accuracy dashboards for custom classifier or custom extraction models. For custom models, the service exposes:
    • Confidence scores per prediction (0–1) for words, key‑value pairs, selection marks, regions, signatures, and (with the 2024‑11‑30 GA API for custom models) tables, rows, and cells.
    • An estimated accuracy score for custom models as part of the build/train output, expressed as a percentage (0–100%), representing how well the model predicts labeled values on visually similar documents. Custom neural and generative models do not provide this training accuracy score.

    There is no mention of native confusion matrices, per‑class accuracy, precision/recall/F1, or version comparison reports in the provided documentation.

    1. The recommended approach is to use the service’s outputs (confidence scores and estimated accuracy) as raw signals and build custom evaluation logic around them:
    • Use the estimated accuracy score from the custom model build/train operation as a coarse quality indicator, targeting ≥80% and closer to 100% for sensitive domains.
    • Use confidence scores at field/word/table level to drive workflows (for example, thresholds for auto‑accept vs human review) and to prioritize which predictions to validate.
    • For classifiers and composed models, use confidence‑based routing and thresholds to control when to accept a classification vs ignore or send to fallback/human review.

    All other evaluation metrics (confusion matrix, precision/recall/F1, trend analysis) must be computed externally.

    1. Yes. To obtain true evaluation statistics, an external validation process is required:
    • Prepare a labeled test set (ground truth) for both classification (document type labels) and extraction (field‑level labels).
    • Run this test set through the custom classifier and custom extraction models.
    • Compare the JSON analysis output (predicted classes and extracted fields) with the ground‑truth labels.
    • From these comparisons, compute:
      • Classification accuracy per document type and confusion matrix.
      • Precision, recall, and F1 for each class and overall.
      • Field‑level extraction accuracy, missing/incorrect counts, and any business‑specific quality metrics.

    This aligns with how custom models are described: they are trained on labeled datasets and return confidence and estimated accuracy, but evaluation against ground truth beyond that must be implemented by the solution.

    1. For monitoring model quality over time from production traffic, the following should be logged from each analyze response:
    • Model metadata:
      • Model ID and version.
      • Model type (custom template vs custom neural, classifier vs extraction, composed model routing decision).
    • Request context:
      • Document identifier, type (if known), and any business label used as ground truth later.
      • Timestamp and environment (prod/test).
    • Prediction outputs:
      • For classification: predicted document class and its confidence.
      • For extraction: all extracted fields, their values, and confidence scores; for custom models with the 2024‑11‑30 GA API, also table/row/cell confidence where relevant.
      • Any read/layout confidence (readResults vs pageResults) to distinguish OCR quality issues from extraction issues.
    • Downstream validation:
      • Whether the prediction was accepted automatically or sent to human review based on confidence thresholds.
      • If human review or business system comparison is available, the corrected/validated values and a flag indicating whether the original prediction was correct.

    These logs allow offline computation of:

    • Accuracy per field and per document type.
    • Error types (missing vs incorrect vs low‑confidence but correct).
    • Quality trends per model version over time.
    1. Azure Monitor, Application Insights, and Log Analytics are primarily designed for operational telemetry (usage, latency, errors, throughput). However, they can also store and query custom metrics and logs, so they can be used as the storage and analytics layer for model‑quality reporting if the application emits the right data:
    • Use them to track operational metrics such as processed pages, latency, failures, and usage per model.
    • Additionally, emit custom events/metrics that include model ID, predicted vs actual labels, confidence, and correctness flags.
    • Build Kusto queries and dashboards to compute and visualize accuracy, error rates, and trends over time.

    The services do not automatically compute ML metrics; they provide the telemetry platform on which custom model‑quality reporting can be implemented.

    1. To compare two model versions before switching production traffic, a recommended pattern is:
    • Use the estimated accuracy score from training as an initial filter (prefer models with higher estimated accuracy, especially for sensitive scenarios).
    • Run an offline A/B evaluation on a held‑out labeled test set:
      • Send the same documents to both model versions.
      • Compare their outputs against ground truth and compute per‑field and per‑document metrics.
    • Optionally, perform a shadow or canary test in production:
      • Route a subset of production documents to both models in parallel (only one model’s output is used operationally; the other is logged for comparison).
      • Compare correctness and confidence distributions using the logged ground‑truth or human‑reviewed data.
    • Use confidence‑based routing and thresholds, especially when using composed models and classifiers, to ensure documents are routed to the most appropriate model and to ignore low‑confidence classifications when needed.

    Once the new model shows equal or better performance on the chosen metrics (and acceptable operational characteristics), update the model ID used by the application.

    References:

    AI-generated content may be incorrect. Read our transparency notes for more information.

    Was this answer helpful?

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.