Conversational Analysis Authoring - Get Model Evaluation Summary

Reference

Service:: Azure AI Services

API Version:: 2022-05-01

Gets the evaluation summary of a trained model. The summary includes high level performance measurements of the model e.g., F1, Precision, Recall, etc.

GET {Endpoint}/language/authoring/analyze-conversations/projects/{projectName}/models/{trainedModelLabel}/evaluation/summary-result?api-version=2022-05-01

URI Parameters

Name	In	Required	Type	Description
Endpoint	path	True	string	Supported Cognitive Services endpoint (e.g., https://.api.cognitiveservices.azure.com).
projectName	path	True	string maxLength: 100	The name of the project to use.
trainedModelLabel	path	True	string	The trained model label.
api-version	query	True	string	Client API version.

Request Header

Name	Required	Type	Description
Ocp-Apim-Subscription-Key	True	string	A subscription key for a Language service resource.

Responses

Name	Type	Description
200 OK	ConversationalAnalysisAuthoringEvaluationSummary	List of all evaluation results.
Other Status Codes	ErrorResponse	Error response.

Security

Ocp-Apim-Subscription-Key

A subscription key for a Language service resource.

Type: apiKey
In: header

Examples

Successful Get Model Evaluation Summary

Sample request

HTTP

GET {Endpoint}/language/authoring/analyze-conversations/projects/EmailApp/models/model1/evaluation/summary-result?api-version=2022-05-01

Sample response

Status code:: 200

{
  "entitiesEvaluation": {
    "confusionMatrix": {
      "Attachment": {
        "Attachment": {
          "normalizedValue": 100,
          "rawValue": 3
        }
      },
      "Category": {
        "Category": {
          "normalizedValue": 91.666664,
          "rawValue": 11
        },
        "$none": {
          "normalizedValue": 8.333333,
          "rawValue": 1
        }
      },
      "ContactName": {
        "ContactName": {
          "normalizedValue": 91.666664,
          "rawValue": 22
        },
        "SenderName": {
          "normalizedValue": 4.1666665,
          "rawValue": 1
        },
        "$none": {
          "normalizedValue": 4.1666665,
          "rawValue": 1
        }
      },
      "Date": {
        "Date": {
          "normalizedValue": 100,
          "rawValue": 2
        }
      },
      "EmailSubject": {
        "EmailSubject": {
          "normalizedValue": 93.33333,
          "rawValue": 9.333334
        },
        "$none": {
          "normalizedValue": 6.6666665,
          "rawValue": 0.6666667
        }
      },
      "FromRelationshipName": {
        "FromRelationshipName": {
          "normalizedValue": 100,
          "rawValue": 1
        }
      },
      "Line": {
        "Line": {
          "normalizedValue": 100,
          "rawValue": 2
        }
      },
      "Message": {
        "Message": {
          "normalizedValue": 81.2063,
          "rawValue": 6.496504
        },
        "EmailSubject": {
          "normalizedValue": 7.43007,
          "rawValue": 0.5944056
        },
        "$none": {
          "normalizedValue": 9.120047,
          "rawValue": 0.7296037
        },
        "Date": {
          "normalizedValue": 2.2435899,
          "rawValue": 0.17948718
        }
      },
      "OrderReference": {
        "OrderReference": {
          "normalizedValue": 100,
          "rawValue": 17
        }
      },
      "PositionReference": {
        "$none": {
          "normalizedValue": 100,
          "rawValue": 1
        }
      },
      "RelationshipName": {
        "RelationshipName": {
          "normalizedValue": 66.666664,
          "rawValue": 2
        },
        "$none": {
          "normalizedValue": 33.333332,
          "rawValue": 1
        }
      },
      "SearchTexts": {
        "SearchTexts": {
          "normalizedValue": 100,
          "rawValue": 4
        }
      },
      "SenderName": {
        "SenderName": {
          "normalizedValue": 88.888885,
          "rawValue": 8
        },
        "ContactName": {
          "normalizedValue": 11.111111,
          "rawValue": 1
        }
      },
      "Time": {
        "$none": {
          "normalizedValue": 100,
          "rawValue": 2
        }
      },
      "$none": {
        "$none": {
          "normalizedValue": 99.739265,
          "rawValue": 162.575
        },
        "Category": {
          "normalizedValue": 0.2607362,
          "rawValue": 0.425
        }
      }
    },
    "entities": {
      "ContactName": {
        "f1": 0.9361702799797058,
        "precision": 0.95652174949646,
        "recall": 0.9166666865348816,
        "truePositiveCount": 22,
        "trueNegativeCount": 0,
        "falsePositiveCount": 1,
        "falseNegativeCount": 2
      },
      "Category": {
        "f1": 0.8799999952316284,
        "precision": 0.8461538553237915,
        "recall": 0.9166666865348816,
        "truePositiveCount": 11,
        "trueNegativeCount": 0,
        "falsePositiveCount": 2,
        "falseNegativeCount": 1
      },
      "SenderName": {
        "f1": 0.8888888955116272,
        "precision": 0.8888888955116272,
        "recall": 0.8888888955116272,
        "truePositiveCount": 8,
        "trueNegativeCount": 0,
        "falsePositiveCount": 1,
        "falseNegativeCount": 1
      },
      "EmailSubject": {
        "f1": 0.8181817531585693,
        "precision": 0.75,
        "recall": 0.8999999761581421,
        "truePositiveCount": 9,
        "trueNegativeCount": 0,
        "falsePositiveCount": 3,
        "falseNegativeCount": 1
      },
      "Message": {
        "f1": 0.75,
        "precision": 0.75,
        "recall": 0.75,
        "truePositiveCount": 6,
        "trueNegativeCount": 0,
        "falsePositiveCount": 2,
        "falseNegativeCount": 2
      },
      "Date": {
        "f1": 0.800000011920929,
        "precision": 0.6666666865348816,
        "recall": 1,
        "truePositiveCount": 2,
        "trueNegativeCount": 0,
        "falsePositiveCount": 1,
        "falseNegativeCount": 0
      },
      "OrderReference": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 17,
        "trueNegativeCount": 0,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "SearchTexts": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 4,
        "trueNegativeCount": 0,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "Attachment": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 3,
        "trueNegativeCount": 0,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "RelationshipName": {
        "f1": 0.800000011920929,
        "precision": 1,
        "recall": 0.6666666865348816,
        "truePositiveCount": 2,
        "trueNegativeCount": 0,
        "falsePositiveCount": 0,
        "falseNegativeCount": 1
      },
      "Line": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 2,
        "trueNegativeCount": 0,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "Time": {
        "f1": 0,
        "precision": 0,
        "recall": 0,
        "truePositiveCount": 0,
        "trueNegativeCount": 0,
        "falsePositiveCount": 0,
        "falseNegativeCount": 2
      },
      "FromRelationshipName": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 1,
        "trueNegativeCount": 0,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "PositionReference": {
        "f1": 0,
        "precision": 0,
        "recall": 0,
        "truePositiveCount": 0,
        "trueNegativeCount": 0,
        "falsePositiveCount": 0,
        "falseNegativeCount": 1
      }
    },
    "microF1": 0.8923077,
    "microPrecision": 0.8969072,
    "microRecall": 0.8877551,
    "macroF1": 0.7766601,
    "macroPrecision": 0.7755879,
    "macroRecall": 0.78849214
  },
  "intentsEvaluation": {
    "confusionMatrix": {
      "AddFlag": {
        "AddFlag": {
          "normalizedValue": 100,
          "rawValue": 6
        }
      },
      "AddMore": {
        "AddMore": {
          "normalizedValue": 100,
          "rawValue": 17
        }
      },
      "Cancel": {
        "Cancel": {
          "normalizedValue": 100,
          "rawValue": 9
        }
      },
      "CheckMessages": {
        "CheckMessages": {
          "normalizedValue": 100,
          "rawValue": 9
        }
      },
      "Confirm": {
        "Confirm": {
          "normalizedValue": 100,
          "rawValue": 4
        }
      },
      "Delete": {
        "Delete": {
          "normalizedValue": 100,
          "rawValue": 5
        }
      },
      "Forward": {
        "Forward": {
          "normalizedValue": 100,
          "rawValue": 6
        }
      },
      "None": {
        "None": {
          "normalizedValue": 100,
          "rawValue": 1
        }
      },
      "QueryLastText": {
        "QueryLastText": {
          "normalizedValue": 100,
          "rawValue": 6
        }
      },
      "ReadAloud": {
        "ReadAloud": {
          "normalizedValue": 100,
          "rawValue": 16
        }
      },
      "Reply": {
        "Reply": {
          "normalizedValue": 100,
          "rawValue": 6
        }
      },
      "SearchMessages": {
        "SearchMessages": {
          "normalizedValue": 100,
          "rawValue": 9
        }
      },
      "SendEmail": {
        "SendEmail": {
          "normalizedValue": 100,
          "rawValue": 20
        }
      },
      "ShowNext": {
        "ShowNext": {
          "normalizedValue": 100,
          "rawValue": 4
        }
      },
      "ShowPrevious": {
        "ShowPrevious": {
          "normalizedValue": 100,
          "rawValue": 3
        }
      }
    },
    "intents": {
      "AddMore": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 17,
        "trueNegativeCount": 104,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "Cancel": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 9,
        "trueNegativeCount": 112,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "SendEmail": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 20,
        "trueNegativeCount": 101,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "CheckMessages": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 9,
        "trueNegativeCount": 112,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "AddFlag": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 6,
        "trueNegativeCount": 115,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "Reply": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 6,
        "trueNegativeCount": 115,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "ReadAloud": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 16,
        "trueNegativeCount": 105,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "QueryLastText": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 6,
        "trueNegativeCount": 115,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "SearchMessages": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 9,
        "trueNegativeCount": 112,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "Delete": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 5,
        "trueNegativeCount": 116,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "Forward": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 6,
        "trueNegativeCount": 115,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "Confirm": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 4,
        "trueNegativeCount": 117,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "ShowNext": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 4,
        "trueNegativeCount": 117,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "ShowPrevious": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 3,
        "trueNegativeCount": 118,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      },
      "None": {
        "f1": 1,
        "precision": 1,
        "recall": 1,
        "truePositiveCount": 1,
        "trueNegativeCount": 120,
        "falsePositiveCount": 0,
        "falseNegativeCount": 0
      }
    },
    "microF1": 1,
    "microPrecision": 1,
    "microRecall": 1,
    "macroF1": 1,
    "macroPrecision": 1,
    "macroRecall": 1
  },
  "evaluationOptions": {
    "kind": "percentage",
    "trainingSplitPercentage": 80,
    "testingSplitPercentage": 20
  }
}

Definitions

Name	Description
ConversationalAnalysisAuthoringEntitiesEvaluationSummary	Represents the evaluation result summary for the project's entities.
ConversationalAnalysisAuthoringEntityEvaluationSummary	Represents the evaluation result for an entity.
ConversationalAnalysisAuthoringEvaluationOptions	Represents the options used running the evaluation.
ConversationalAnalysisAuthoringEvaluationSummary	Represents the evaluation result summary.
ConversationalAnalysisAuthoringIntentEvaluationSummary	The evaluation summary for an intent.
ConversationalAnalysisAuthoringIntentsEvaluationSummary	Represents the evaluation summary for the project's intents.
Error	The error object.
ErrorCode	Human-readable error code.
ErrorResponse	Error response.
EvaluationKind	Represents the evaluation kind. By default, the evaluation kind is set to percentage.
InnerErrorCode	Human-readable error code.
InnerErrorModel	An object containing more specific information about the error. As per Microsoft One API guidelines - https://github.com/Microsoft/api-guidelines/blob/vNext/Guidelines.md#7102-error-condition-responses.

ConversationalAnalysisAuthoringEntitiesEvaluationSummary

Object

Represents the evaluation result summary for the project's entities.

Name	Type	Description
confusionMatrix	object	Represents the confusion matrix between two entities (the two entities can be the same). The matrix is between the entity that was labelled and the entity that was predicted.
entities	<string, ConversationalAnalysisAuthoringEntityEvaluationSummary>	Represents the entities evaluation summary.
macroF1	number (float)	Represents the macro F1
macroPrecision	number (float)	Represents the macro precision
macroRecall	number (float)	Represents the macro recall
microF1	number (float)	Represents the micro F1
microPrecision	number (float)	Represents the micro precision
microRecall	number (float)	Represents the micro recall

ConversationalAnalysisAuthoringEntityEvaluationSummary

Object

Represents the evaluation result for an entity.

Name	Type	Description
f1	number (double)	Represents the model precision
falseNegativeCount	integer (int32)	Represents the count of false negative
falsePositiveCount	integer (int32)	Represents the count of false positive
precision	number (double)	Represents the model recall
recall	number (double)	Represents the model F1 score
trueNegativeCount	integer (int32)	Represents the count of true negative
truePositiveCount	integer (int32)	Represents the count of true positive

ConversationalAnalysisAuthoringEvaluationOptions

Object

Represents the options used running the evaluation.

Name	Type	Description
kind	EvaluationKind	Represents the evaluation kind. By default, the evaluation kind is set to percentage.
testingSplitPercentage	integer (int32)	Represents the testing dataset split percentage. Only needed in case the evaluation kind is percentage.
trainingSplitPercentage	integer (int32)	Represents the training dataset split percentage. Only needed in case the evaluation kind is percentage.

ConversationalAnalysisAuthoringEvaluationSummary

Object

Represents the evaluation result summary.

Name	Type	Description
entitiesEvaluation	ConversationalAnalysisAuthoringEntitiesEvaluationSummary	Contains the data related to entities evaluation.
evaluationOptions	ConversationalAnalysisAuthoringEvaluationOptions	The options that were used while running the evaluation.
intentsEvaluation	ConversationalAnalysisAuthoringIntentsEvaluationSummary	Contains the data related to intents evaluation.

ConversationalAnalysisAuthoringIntentEvaluationSummary

Object

The evaluation summary for an intent.

Name	Type	Description
f1	number (double)	Represents the model precision
falseNegativeCount	integer (int32)	Represents the count of false negative
falsePositiveCount	integer (int32)	Represents the count of false positive
precision	number (double)	Represents the model recall
recall	number (double)	Represents the model F1 score
trueNegativeCount	integer (int32)	Represents the count of true negative
truePositiveCount	integer (int32)	Represents the count of true positive

ConversationalAnalysisAuthoringIntentsEvaluationSummary

Object

Represents the evaluation summary for the project's intents.

Name	Type	Description
confusionMatrix	object	Represents the confusion matrix between two intents (the two intents can be the same). The matrix is between the intent that was labelled and the intent that was predicted.
intents	<string, ConversationalAnalysisAuthoringIntentEvaluationSummary>	Represents the intents evaluation summary.
macroF1	number (float)	Represents the macro F1
macroPrecision	number (float)	Represents the macro precision
macroRecall	number (float)	Represents the macro recall
microF1	number (float)	Represents the micro F1
microPrecision	number (float)	Represents the micro precision
microRecall	number (float)	Represents the micro recall

Error

Object

The error object.

Name	Type	Description
code	ErrorCode	One of a server-defined set of error codes.
details	Error[]	An array of details about specific errors that led to this reported error.
innererror	InnerErrorModel	An object containing more specific information than the current object about the error.
message	string	A human-readable representation of the error.
target	string	The target of the error.

ErrorCode

Enumeration

Human-readable error code.

Value	Description
AzureCognitiveSearchIndexLimitReached
AzureCognitiveSearchIndexNotFound
AzureCognitiveSearchNotFound
AzureCognitiveSearchThrottling
Conflict
Forbidden
InternalServerError
InvalidArgument
InvalidRequest
NotFound
OperationNotFound
ProjectNotFound
QuotaExceeded
ServiceUnavailable
Timeout
TooManyRequests
Unauthorized
Warning

ErrorResponse

Object

Error response.

Name	Type	Description
error	Error	The error object.

EvaluationKind

Enumeration

Represents the evaluation kind. By default, the evaluation kind is set to percentage.

Value	Description
manual	Split the data according to the chosen dataset for every example in the data.
percentage	Split the data into training and test sets according to user-defined percentages.

InnerErrorCode

Enumeration

Human-readable error code.

Value	Description
AzureCognitiveSearchNotFound
AzureCognitiveSearchThrottling
EmptyRequest
ExtractionFailure
InvalidCountryHint
InvalidDocument
InvalidDocumentBatch
InvalidParameterValue
InvalidRequest
InvalidRequestBodyFormat
KnowledgeBaseNotFound
MissingInputDocuments
ModelVersionIncorrect
UnsupportedLanguageCode

InnerErrorModel

Object

An object containing more specific information about the error. As per Microsoft One API guidelines - https://github.com/Microsoft/api-guidelines/blob/vNext/Guidelines.md#7102-error-condition-responses.

Name	Type	Description
code	InnerErrorCode	One of a server-defined set of error codes.
details	object	Error details.
innererror	InnerErrorModel	An object containing more specific information than the current object about the error.
message	string	Error message.
target	string	Error target.

Share via

Conversational Analysis Authoring - Get Model Evaluation Summary

URI Parameters

Request Header

Responses

Security

Ocp-Apim-Subscription-Key

Examples

Successful Get Model Evaluation Summary

Sample request

Sample response

Definitions

ConversationalAnalysisAuthoringEntitiesEvaluationSummary

ConversationalAnalysisAuthoringEntityEvaluationSummary

ConversationalAnalysisAuthoringEvaluationOptions

ConversationalAnalysisAuthoringEvaluationSummary

ConversationalAnalysisAuthoringIntentEvaluationSummary

ConversationalAnalysisAuthoringIntentsEvaluationSummary

Error

ErrorCode

ErrorResponse

EvaluationKind

InnerErrorCode

InnerErrorModel