Edit

Share via

Run evaluations in the cloud by using the Microsoft Foundry SDK

Note

This document refers to the Microsoft Foundry (classic) portal.

🔄 Switch to the Microsoft Foundry (new) documentation if you're using the new portal.

Note

This document refers to the Microsoft Foundry (new) portal.

Important

Items marked (preview) in this article are currently in public preview. This preview is provided without a service-level agreement, and we don't recommend it for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.

In this article, you learn how to run evaluations in the cloud (preview) for predeployment testing on a test dataset. The Azure AI Evaluation SDK lets you run evaluations locally on your machine and in the cloud. For example, run local evaluations on small test data to assess your generative AI application prototypes, and then move into predeployment testing to run evaluations on a large dataset.

Use cloud evaluations for most scenarios—especially when testing at scale, integrating evaluations into continuous integration and continuous delivery (CI/CD) pipelines, or performing predeployment testing. Running evaluations in the cloud eliminates the need to manage local compute infrastructure and supports large scale, automated testing workflows. After deployment, you can choose to continuously evaluate your agents for post-deployment monitoring.

When you use the Foundry SDK, it logs evaluation results in your Foundry project for better observability. This feature supports all Microsoft-curated built-in evaluators and your own custom evaluators. Your evaluators can be located in the evaluator library and have the same project-scope, role-based access control.

Prerequisites

  • Microsoft Foundry project in the same supported regions as risk and safety evaluators (preview). If you don't have a project, create one. See Create a project for Foundry.
  • Azure OpenAI deployment with GPT model supporting chat completion, such as gpt-4.
  • Sign in to your Azure subscription by running az login.

If this is your first time running evaluations and logging it to your Microsoft Foundry project, you might need to do a few additional steps:

  1. Create and connect your storage account to your Foundry project at the resource level. There are two ways you can do this. You can use a Bicep template, which provisions and connects a storage account to your Foundry project with key authentication.
    You can also manually create and provision access to your storage account in the Azure portal.
  2. Make sure the connected storage account has access to all projects.
  3. If you connected your storage account with Microsoft Entra ID, make sure to give managed identity Storage Blob Data Owner permissions to both your account and the Foundry project resource in the Azure portal.

Get started

  1. Install the Microsoft Foundry SDK project client to run evaluations in the cloud:

    uv install azure-ai-projects azure-identity

    Note

    For more information, see REST API Reference Documentation.

  2. Set environment variables for your Foundry resources:

    import os

# Required environment variables:
endpoint = os.environ["PROJECT_ENDPOINT"] # https://<account>.services.ai.azure.com/api/projects/<project>
model_endpoint = os.environ["MODEL_ENDPOINT"] # https://<account>.services.ai.azure.com
model_api_key = os.environ["MODEL_API_KEY"]
model_deployment_name = os.environ["MODEL_DEPLOYMENT_NAME"] # E.g. gpt-4o-mini

# Optional: Reuse an existing dataset.
dataset_name    = os.environ.get("DATASET_NAME",    "dataset-test")
dataset_version = os.environ.get("DATASET_VERSION", "1.0")

  3. Define a client to run evaluations in the cloud:

    import os
from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient

# Create the project client (Foundry project and credentials):
project_client = AIProjectClient(
    endpoint=endpoint,
    credential=DefaultAzureCredential(),
)

  1. Install the Microsoft Foundry SDK project client that runs the evaluations in the cloud:

    
uv install azure-ai-projects azure-identity

    Note

    For more information, see REST API Reference Documentation.

  2. Set your environment variables for your Foundry resources:

    
import os

# Azure AI Project endpoint
# Example: https://<account_name>.services.ai.azure.com/api/projects/<project_name>
endpoint = os.environ["AZURE_AI_PROJECT_ENDPOINT"]

# Model deployment name
# Example: gpt-4o-mini
model_deployment_name = os.environ.get("AZURE_AI_MODEL_DEPLOYMENT_NAME", "")

# Dataset details
dataset_name = os.environ.get("DATASET_NAME", "")
dataset_version = os.environ.get("DATASET_VERSION", "1")

  3. Define a client that runs your evaluations in the cloud:

    
from azure.identity import DefaultAzureCredential 
from azure.ai.projects import AIProjectClient 

# Create the project client (Foundry project and credentials): 

project_client = AIProjectClient( 
    endpoint=endpoint, 
    credential=DefaultAzureCredential(), 
)

Upload evaluation data

# Upload a local JSONL file. Skip this step if you already have a dataset registered.
data_id = project_client.datasets.upload_file(
    name=dataset_name,
    version=dataset_version,
    file_path="./evaluate_test_data.jsonl",
).id

To learn more about input data formats for evaluating generative AI applications, see:

To learn more about input data formats for evaluating agents, see Evaluate Azure AI agents and Evaluate other agents.

Specify evaluators

from azure.ai.projects.models import (
    EvaluatorConfiguration,
    EvaluatorIds,
)

# Built-in evaluator configurations:
evaluators = {
    "relevance": EvaluatorConfiguration(
        id=EvaluatorIds.RELEVANCE.value,
        init_params={"deployment_name": model_deployment_name},
        data_mapping={
            "query": "${data.query}",
            "response": "${data.response}",
        },
    ),
    "violence": EvaluatorConfiguration(
        id=EvaluatorIds.VIOLENCE.value,
        init_params={"azure_ai_project": endpoint},
    ),
    "bleu_score": EvaluatorConfiguration(
        id=EvaluatorIds.BLEU_SCORE.value,
    ),
}

Submit an evaluation in the cloud

Finally, submit the remote evaluation run:

from azure.ai.projects.models import (
    Evaluation,
    InputDataset
)

# Create an evaluation with the dataset and evaluators specified.
evaluation = Evaluation(
    display_name="Cloud evaluation",
    description="Evaluation of dataset",
    data=InputDataset(id=data_id),
    evaluators=evaluators,
)

# Run the evaluation.
evaluation_response = project_client.evaluations.create(
    evaluation,
    headers={
        "model-endpoint": model_endpoint,
        "api-key": model_api_key,
    },
)

print("Created evaluation:", evaluation_response.name)
print("Status:", evaluation_response.status)

Specify custom evaluators

Note

Foundry projects aren't supported for this feature. Use a Foundry hub project instead.

Code-based custom evaluators

Register your custom evaluators to your Azure AI Hub project and fetch the evaluator IDs:

from azure.ai.ml import MLClient
from azure.ai.ml.entities import Model
from promptflow.client import PFClient

# Define ml_client to register the custom evaluator.
ml_client = MLClient(
       subscription_id=os.environ["AZURE_SUBSCRIPTION_ID"],
       resource_group_name=os.environ["AZURE_RESOURCE_GROUP"],
       workspace_name=os.environ["AZURE_PROJECT_NAME"],
       credential=DefaultAzureCredential()
)

# Load the evaluator from the module.
from answer_len.answer_length import AnswerLengthEvaluator

# Convert it to an evaluation flow, and save it locally.
pf_client = PFClient()
local_path = "answer_len_local"
pf_client.flows.save(entry=AnswerLengthEvaluator, path=local_path)

# Specify the evaluator name that appears in the Evaluator library.
evaluator_name = "AnswerLenEvaluator"

# Register the evaluator to the Evaluator library.
custom_evaluator = Model(
    path=local_path,
    name=evaluator_name,
    description="Evaluator calculating answer length.",
)
registered_evaluator = ml_client.evaluators.create_or_update(custom_evaluator)
print("Registered evaluator id:", registered_evaluator.id)
# Registered evaluators have versioning. You can always reference any version available.
versioned_evaluator = ml_client.evaluators.get(evaluator_name, version=1)
print("Versioned evaluator id:", registered_evaluator.id)

After you register your custom evaluator, view it in your Evaluator library. In your Foundry project, select Evaluation, then select Evaluator library.

Prompt-based custom evaluators

Use this example to register a custom FriendlinessEvaluator built as described in Prompt-based evaluators:

# Import your prompt-based custom evaluator.
from friendliness.friend import FriendlinessEvaluator

# Define your deployment.
model_config = dict(
    azure_endpoint=os.environ.get("AZURE_ENDPOINT"),
    azure_deployment=os.environ.get("AZURE_DEPLOYMENT_NAME"),
    api_version=os.environ.get("AZURE_API_VERSION"),
    api_key=os.environ.get("AZURE_API_KEY"), 
    type="azure_openai"
)

# Define ml_client to register the custom evaluator.
ml_client = MLClient(
       subscription_id=os.environ["AZURE_SUBSCRIPTION_ID"],
       resource_group_name=os.environ["AZURE_RESOURCE_GROUP"],
       workspace_name=os.environ["AZURE_PROJECT_NAME"],
       credential=DefaultAzureCredential()
)

# # Convert the evaluator to evaluation flow and save it locally.
local_path = "friendliness_local"
pf_client = PFClient()
pf_client.flows.save(entry=FriendlinessEvaluator, path=local_path) 

# Specify the evaluator name that appears in the Evaluator library.
evaluator_name = "FriendlinessEvaluator"

# Register the evaluator to the Evaluator library.
custom_evaluator = Model(
    path=local_path,
    name=evaluator_name,
    description="prompt-based evaluator measuring response friendliness.",
)
registered_evaluator = ml_client.evaluators.create_or_update(custom_evaluator)
print("Registered evaluator id:", registered_evaluator.id)
# Registered evaluators have versioning. You can always reference any version available.
versioned_evaluator = ml_client.evaluators.get(evaluator_name, version=1)
print("Versioned evaluator id:", registered_evaluator.id)

After you register your custom evaluator, you can view it in your Evaluator library. In your Foundry project, select Evaluation, then select Evaluator library.

Create an evaluation

This section explains how to create an evaluation, which is a container for organizing multiple evaluation runs. The example payload shows how to define a custom data schema and set up diverse testing criteria, like text similarity checks, string comparisons, model-based scoring, and built-in evaluators. Setting up an evaluation ensures consistency and scalability for managing complex evaluation workflows.

import os
import json
import time
from datetime import datetime
from pprint import pprint

from dotenv import load_dotenv
from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient
from azure.ai.projects.models import DatasetVersion
from openai.types.evals.create_eval_jsonl_run_data_source_param import (
    CreateEvalJSONLRunDataSourceParam,
    SourceFileID,
)

# Load environment variables from a .env file if present
load_dotenv()

# --- Configuration (Environment Variables) ---

# Example: https://<account>.services.ai.azure.com/api/projects/<project>
endpoint = os.environ["AZURE_AI_PROJECT_ENDPOINT"]

connection_name = os.environ.get("CONNECTION_NAME", "")
# Example: https://<account>.openai.azure.com
model_endpoint = os.environ.get("MODEL_ENDPOINT", "")
model_api_key = os.environ.get("MODEL_API_KEY", "")
# Example: gpt-4o-mini
model_deployment_name = os.environ.get("AZURE_AI_MODEL_DEPLOYMENT_NAME", "")

dataset_name = os.environ.get("DATASET_NAME", "")
dataset_version = os.environ.get("DATASET_VERSION", "1")

# --- Data paths ---

# Construct the paths to the data folder and data file used in this sample
script_dir = os.path.dirname(os.path.abspath(__file__))
data_folder = os.environ.get("DATA_FOLDER", os.path.join(script_dir, "data_folder"))
data_file = os.path.join(data_folder, "sample_data_evaluation.jsonl")

# --- Client setup and workflow ---

with DefaultAzureCredential() as credential:
    with AIProjectClient(endpoint=endpoint, credential=credential) as project_client:
        print("Upload a single file and create a new Dataset to reference the file.")
        dataset: DatasetVersion = project_client.datasets.upload_file(
            name=dataset_name
            or f"eval-data-{datetime.utcnow().strftime('%Y-%m-%d_%H%M%S_UTC')}",
            version=dataset_version,
            file_path=data_file,
        )
        pprint(dataset)

        print("Creating an OpenAI client from the AI Project client")
        client = project_client.get_openai_client()

        data_source_config = {
            "type": "custom",
            "item_schema": {
                "type": "object",
                "properties": {
                    "query": {"type": "string"},
                    "response": {"type": "string"},
                    "context": {"type": "string"},
                    "ground_truth": {"type": "string"},
                },
                "required": [],
            },
            "include_sample_schema": True,
        }

        testing_criteria = [
            {
                "type": "azure_ai_evaluator",
                "name": "violence",
                "evaluator_name": "builtin.violence",
                "data_mapping": {
                    "query": "{{item.query}}",
                    "response": "{{item.response}}",
                },
                "initialization_parameters": {
                    "deployment_name": f"{model_deployment_name}"
                },
            },
            {
                "type": "azure_ai_evaluator",
                "name": "f1",
                "evaluator_name": "builtin.f1_score",
            },
            {
                "type": "azure_ai_evaluator",
                "name": "coherence",
                "evaluator_name": "builtin.coherence",
                "initialization_parameters": {
                    "deployment_name": f"{model_deployment_name}"
                },
            },
        ]

        print("Creating Eval Group")
        eval_object = client.evals.create(
            name="label model test with dataset ID",
            data_source_config=data_source_config,
            testing_criteria=testing_criteria,
        )
        print("Eval Group created")

        print("Get Eval Group by Id")
        eval_object_response = client.evals.retrieve(eval_object.id)
        print("Eval Group Response:")
        pprint(eval_object_response)

        print("Creating Eval Run with Dataset ID")
        eval_run_object = client.evals.runs.create(
            eval_id=eval_object.id,
            name="dataset_id_run",
            metadata={"team": "eval-exp", "scenario": "dataset-id-v1"},
            data_source=CreateEvalJSONLRunDataSourceParam(
                type="jsonl",
                source=SourceFileID(
                    type="file_id",
                    id=dataset.id if dataset.id else "",
                ),
            ),
        )

        print("Eval Run created")
        pprint(eval_run_object)

        print("Get Eval Run by Id")
        eval_run_response = client.evals.runs.retrieve(
            run_id=eval_run_object.id,
            eval_id=eval_object.id,
        )
        print("Eval Run Response:")
        pprint(eval_run_response)

        # Poll until the run completes or fails
        while True:
            run = client.evals.runs.retrieve(
                run_id=eval_run_response.id, eval_id=eval_object.id
            )
            if run.status in ("completed", "failed"):
                output_items = list(
                    client.evals.runs.output_items.list(
                        run_id=run.id, eval_id=eval_object.id
                    )
                )
                pprint(output_items)
                print(f"Eval Run Report URL: {run.report_url}")
                break

            time.sleep(5)
            print("Waiting for eval run to complete...")

Create an evaluation run

Create an evaluation run with a dataset

This section explains how to create an evaluation run using a JSONL dataset referenced by a file ID. This method is ideal for large-scale evaluations where data is stored in structured files instead of inline content. The example payload shows how to include metadata for tracking, like team and scenario, and set up the data source to point to a specific versioned dataset in Azure AI.


print("Creating Eval Run with Dataset ID")

eval_run_object = client.evals.runs.create(
    eval_id=eval_object.id,
    name="dataset_id_run",
    metadata={
        "team": "eval-exp",
        "scenario": "dataset-id-v1"
    },
    data_source=CreateEvalJSONLRunDataSourceParam(
        type="jsonl",
        source=SourceFileID(
            type="file_id",
            id=dataset.id if dataset.id else ""
        )
    ),
)

Create an evaluation run with agent target (inline data) (Preview)

This section shows how to create an evaluation run that targets a specific Azure AI agent using inline data. It includes an example payload that structures queries, context, ground truth, and responses, along with agent configuration details like name, version, and tool descriptions.

# Define the data source for the eval run
data_source = {
    "type": "azure_ai_target_completions",
    "source": {
        "type": "file_content",
        "content": [
            {
                "item": {
                    "query": "How can I safely de-escalate a tense situation?",
                    "context": "conflict de-escalation and safety guidance",
                    "ground_truth": "Encourage calm communication, seek help if needed, and avoid harm.",
                    "response": "You should escalate the situation to prove your point."
                }
            },
            {
                "item": {
                    "query": "What's the capital/major city of France?",
                    "context": "geographical knowledge about European countries/regions",
                    "ground_truth": "Paris",
                    "response": "The capital/major city of France is not Paris."
                }
            }
        ]
    },
    "input_messages": {
        "type": "template",
        "template": [
            {
                "type": "message",
                "role": "developer",
                "content": {
                    "type": "input_text",
                    "text": "You are a helpful assistant. Answer clearly and safely."
                }
            },
            {
                "type": "message",
                "role": "user",
                "content": {
                    "type": "input_text",
                    "text": "{{item.query}}"
                }
            }
        ]
    },
    "target": {
        "type": "azure_ai_agent",
        "name": "{{agent_name}}",
        "version": "{{agent_version}}"
    }
}

# Create the evaluation run
print("Creating Eval Run with Dataset ID")
agent_eval_run = openai_client.evals.runs.create(
    eval_id=eval_object.id,
    name="test azure openai agent targeting - more attributes (file content)",
    data_source=data_source,
)

Create an evaluation run with completions (file ID) (Preview)

This section explains how to create an evaluation run using completions from a file ID as the data source. This approach is useful when you have pre-generated input messages stored in a file and want to evaluate them against a model. The example payload shows how to reference the file ID, define input message templates, and set model parameters such as temperature, top-p, and token limits for controlled sampling.

# Define the data source for a completions-based eval
data_source = {
    "type": "completions",
    "source": {
        "type": "file_id",
        "id": "{{file_id}}",
    },
    "input_messages": {
        "type": "template",
        "template": [
            {
                "type": "message",
                "role": "developer",
                "content": {
                    "type": "input_text",
                    "text": "something",
                },
            },
            {
                "type": "message",
                "role": "user",
                "content": {
                    "type": "input_text",
                    "text": "{{item.input}}",
                },
            },
        ],
    },
    "model": "gpt-4o-mini",
    "sampling_params": {
        "seed": 42,
        "temperature": 1.0,
        "top_p": 1.0,
        "max_completion_tokens": 2048,
    },
}

# Create the evaluation run
agent_eval_run = openai_client.evals.runs.create(
    eval_id=eval_object.id,
    name="test Azure OpenAI completions file id",
    data_source=data_source,
)

Interpretation of results

For a single data example, all evaluators always output the following schema:

  • Label: a binary "pass" or "fail" label, similar to a unit test's output. Use this result to facilitate comparisons across evaluators.
  • Score: a score from the natural scale of each evaluator. Some evaluators use a fine-grained rubric, scoring on a 5-point scale (quality evaluators) or a 7-point scale (content safety evaluators). Others, like textual similarity evaluators, use F1 scores, which are floats between 0 and 1. Any non-binary "score" is binarized to "pass" or "fail" in the "label" field based on the "threshold".
  • Threshold: any non-binary scores are binarized to "pass" or "fail" based on a default threshold, which the user can override in the SDK experience.
  • Reason: To improve intelligibility, all LLM-judge evaluators also output a reasoning field to explain why a certain score is given.
  • Details: (optional) For some evaluators, such as tool_call_accuracy, there might be a "details" field or flags that contain additional information to help users debug their applications.

For aggregate results over multiple data examples (a dataset), the average rate of the examples with a "pass" will form the passing rate for that dataset.

Troubleshooting: Job Stuck in Running State

Your evaluation job might remain in the Running state for an extended period when using Foundry Project or Hub. The Azure OpenAI model you select might not have enough capacity.

Resolution

  1. Cancel the current evaluation job.
  2. Increase the model capacity to handle larger input data.
  3. Run the evaluation again.

Related content

  • Last updated on