IntentResolutionEvaluator Class

Definition

Namespace:: Microsoft.Extensions.AI.Evaluation.Quality

Assembly:: Microsoft.Extensions.AI.Evaluation.Quality.dll

Package:: Microsoft.Extensions.AI.Evaluation.Quality v9.7.0

Source:: IntentResolutionEvaluator.cs

Important

Some information relates to prerelease product that may be substantially modified before it’s released. Microsoft makes no warranties, express or implied, with respect to the information provided here.

An IEvaluator that evaluates an AI system's effectiveness at identifying and resolving user intent.

public ref class IntentResolutionEvaluator sealed : Microsoft::Extensions::AI::Evaluation::IEvaluator

[System.Diagnostics.CodeAnalysis.Experimental("AIEVAL001")]
public sealed class IntentResolutionEvaluator : Microsoft.Extensions.AI.Evaluation.IEvaluator

public sealed class IntentResolutionEvaluator : Microsoft.Extensions.AI.Evaluation.IEvaluator

[<System.Diagnostics.CodeAnalysis.Experimental("AIEVAL001")>]
type IntentResolutionEvaluator = class
    interface IEvaluator

type IntentResolutionEvaluator = class
    interface IEvaluator

Public NotInheritable Class IntentResolutionEvaluator
Implements IEvaluator

Inheritance: Object
IntentResolutionEvaluator

Attributes: ExperimentalAttribute

Implements: IEvaluator

Remarks

IntentResolutionEvaluator evaluates an AI system's effectiveness at identifying and resolving user intent based on the supplied conversation history and the tool definitions supplied via ToolDefinitions.

Note that at the moment, IntentResolutionEvaluator only supports evaluating calls to tools that are defined as AIFunctions. Any other AITool definitions that are supplied via ToolDefinitions will be ignored.

IntentResolutionEvaluator returns a NumericMetric that contains a score for 'Intent Resolution'. The score is a number between 1 and 5, with 1 indicating a poor score, and 5 indicating an excellent score.

Note: IntentResolutionEvaluator is an AI-based evaluator that uses an AI model to perform its evaluation. While the prompt that this evaluator uses to perform its evaluation is designed to be model-agnostic, the performance of this prompt (and the resulting evaluation) can vary depending on the model used, and can be especially poor when a smaller / local model is used.

The prompt that IntentResolutionEvaluator uses has been tested against (and tuned to work well with) the following models. So, using this evaluator with a model from the following list is likely to produce the best results. (The model to be used can be configured via ChatClient.)

GPT-4o

Constructors

IntentResolutionEvaluator()

Properties

EvaluationMetricNames	Gets the Names of the EvaluationMetrics produced by this IEvaluator.
IntentResolutionMetricName	Gets the Name of the NumericMetric returned by IntentResolutionEvaluator.

Methods

EvaluateAsync(IEnumerable<ChatMessage>, ChatResponse, ChatConfiguration, IEnumerable<EvaluationContext>, CancellationToken)

Evaluates the supplied modelResponse and returns an EvaluationResult containing one or more EvaluationMetrics.

Extension Methods

EvaluateAsync(IEvaluator, ChatMessage, ChatMessage, ChatConfiguration, IEnumerable<EvaluationContext>, CancellationToken)	Evaluates the supplied `modelResponse` and returns an EvaluationResult containing one or more EvaluationMetrics.
EvaluateAsync(IEvaluator, ChatMessage, ChatResponse, ChatConfiguration, IEnumerable<EvaluationContext>, CancellationToken)	Evaluates the supplied `modelResponse` and returns an EvaluationResult containing one or more EvaluationMetrics.
EvaluateAsync(IEvaluator, ChatMessage, ChatConfiguration, IEnumerable<EvaluationContext>, CancellationToken)	Evaluates the supplied `modelResponse` and returns an EvaluationResult containing one or more EvaluationMetrics.
EvaluateAsync(IEvaluator, ChatResponse, ChatConfiguration, IEnumerable<EvaluationContext>, CancellationToken)	Evaluates the supplied `modelResponse` and returns an EvaluationResult containing one or more EvaluationMetrics.
EvaluateAsync(IEvaluator, String, ChatConfiguration, IEnumerable<EvaluationContext>, CancellationToken)	Evaluates the supplied `modelResponse` and returns an EvaluationResult containing one or more EvaluationMetrics.
EvaluateAsync(IEvaluator, String, String, ChatConfiguration, IEnumerable<EvaluationContext>, CancellationToken)	Evaluates the supplied `modelResponse` and returns an EvaluationResult containing one or more EvaluationMetrics.

Applies to