TaskAdherenceEvaluator Class

Definition

public ref class TaskAdherenceEvaluator sealed : Microsoft::Extensions::AI::Evaluation::IEvaluator

[System.Diagnostics.CodeAnalysis.Experimental("AIEVAL001")]
public sealed class TaskAdherenceEvaluator : Microsoft.Extensions.AI.Evaluation.IEvaluator

public sealed class TaskAdherenceEvaluator : Microsoft.Extensions.AI.Evaluation.IEvaluator

[<System.Diagnostics.CodeAnalysis.Experimental("AIEVAL001")>]
type TaskAdherenceEvaluator = class
    interface IEvaluator

type TaskAdherenceEvaluator = class
    interface IEvaluator

Public NotInheritable Class TaskAdherenceEvaluator
Implements IEvaluator

Inheritance

Object

TaskAdherenceEvaluator

Attributes

ExperimentalAttribute

Implements

IEvaluator

Remarks

TaskAdherenceEvaluator measures how accurately an AI system adheres to the task assigned to it by examining the alignment of the supplied response with instructions and definitions present in the conversation history, the accuracy and clarity of the response, and the proper use of tool definitions supplied via ToolDefinitions.

Note that at the moment, TaskAdherenceEvaluator only supports evaluating calls to tools that are defined as AIFunctions. Any other AITool definitions that are supplied via ToolDefinitions will be ignored.

TaskAdherenceEvaluator returns a NumericMetric that contains a score for 'Task Adherence'. The score is a number between 1 and 5, with 1 indicating a poor score, and 5 indicating an excellent score.

Note: TaskAdherenceEvaluator is an AI-based evaluator that uses an AI model to perform its evaluation. While the prompt that this evaluator uses to perform its evaluation is designed to be model-agnostic, the performance of this prompt (and the resulting evaluation) can vary depending on the model used, and can be especially poor when a smaller / local model is used.

The prompt that TaskAdherenceEvaluator uses has been tested against (and tuned to work well with) the following models. So, using this evaluator with a model from the following list is likely to produce the best results. (The model to be used can be configured via ChatClient.)

GPT-4o