Best practices for creating dataset from traces for agent evaluation.

Hello Leon Kannanovich,

Welcome to Microsoft Q&A and Thank you for reaching out.

Agent tracing in Azure AI Foundry is designed to help observe, understand and troubleshoot agent behaviour. It provides helpful visibility into how an agent run executes, such as the sequence of steps, latency and high‑level tool usage. However, traces are not intended to function as structured datasets for evaluation or long‑term data analysis.

Trace data cannot be used as an evaluation dataset as the purpose of agent traces using the traces view in Azure AI Foundry is to focus on debugging agent runs,monitoring performance and reliability and inspecting execution flow at a session level

As a result, traces should not be considered a canonical data export layer, since they are not designed to support full replay of agent executions and therefore cannot reliably capture complete interaction flows. Because of these limitations, traces should not be treated as ground-truth datasets for evaluation purposes.

When trace data flows into Azure Application Insights, it is distributed across multiple tables—such as traces, customEvents, and dependencies , which results in only partial information being visible in any single query rather than a complete, unified view of the trace.

The best-practice architecture is to use a dual logging approach, where observability logs are clearly separated from evaluation data. In this setup, agent execution sends telemetry and debugging information to Azure Application Insights for monitoring purposes, while simultaneously writing structured dataset logs to a storage system for use in evaluation workflows.

Please ensure that the structured dataset logging captures

• User input
• Model output
• Tool names
• Tool arguments
• Tool responses
• Timestamps and correlation identifiers

With the recommended storage options that include:

• Azure Blob Storage
• Azure Data Lake
• Cosmos DB

Please check if the following troubleshooting steps help:

1. Inspect raw telemetry

      traces
      | take 10
      | project customDimensions
   

2. Check additional tables - customEvents and dependencies
3. Review sampling behavior

      union traces, customEvents
      | summarize count() by itemType
   

4. Confirm instrumentation by ensuring OpenTelemetry instrumentation is enabled and verbose logging is configured where applicable
5. Review payload size for large tool inputs or outputs may be truncated due to telemetry limits

References:

Set Up Tracing for AI Agents in Microsoft Foundry - Microsoft Foundry | Microsoft Learn

Agent tracing in Microsoft Foundry (preview) - Microsoft Foundry | Microsoft Learn

Application Insights telemetry data model - Azure Monitor | Microsoft Learn

Thank you!

Please 'Upvote'(Thumbs-up) and 'Accept' as answer if the reply was helpful. This will be benefitting other community members who face the same issue.

Agent traces are best used for evaluation by turning them into a curated evaluation dataset, rather than exporting raw telemetry from Application Insights.

Recommended approach:

1. Use traces as the source for evaluation datasets
   Traces capture the journey of a request or workflow through the agent, including tool calls, as spans with attributes and events. These are intended to be the raw material for evaluation datasets, where selected traces become test cases for quality, regression, and comparison of agent versions.
   Evaluation datasets are then used to:
   • Improve quality using known problematic examples.
   • Prevent regressions with a “golden set”.
   • Compare prompts/models/logic on the same data.
   • Target specific features (safety, domain knowledge, edge cases).
2. Instrument traces with rich span attributes for tool calls
   To make traces useful for dataset creation, ensure tool calls are clearly represented in spans:
   • Use spans to represent each tool call (for example, one span per tool invocation).
   • Add attributes for tool name, arguments, results, and any quality or status indicators.
   • Follow OpenTelemetry semantic conventions where possible so attributes are consistent and easy to query.
     This makes it much easier to later filter and extract the tool-call spans that should become evaluation records.
3. Apply tracing best practices to support evaluation
   When instrumenting agents and tools:
   • Use consistent span attribute names across agents and tools so queries and downstream processing are simple.
   • Correlate evaluation run IDs in attributes so quality/performance analysis can be tied back to specific evaluation runs.
   • Redact or avoid storing sensitive content (secrets, credentials, personal data) in prompts, tool arguments, or span attributes before they reach telemetry.
4. Exporting traces: concepts and limitations
   In Azure AI Foundry, traces are exported via OpenTelemetry trace exporters to backends such as Azure Monitor / Application Insights or other OTLP-compatible platforms. These exporters send trace data (traces, spans, attributes) for storage and analysis.
   The Application Insights view and its KQL queries typically surface trace and span metadata and attributes. If only metadata is visible, ensure that:
   • The agent and tools are instrumented to attach the needed attributes (for example, tool call details) to spans.
   • Sensitive data redaction policies are not stripping the fields required for evaluation.
5. Programmatic access and dataset building pattern
   While the context does not describe a dedicated “export traces to dataset” API, the recommended pattern is:
   • Use OpenTelemetry-compatible exporters or SDKs to send traces (including tool-call spans) to a backend.
   • From that backend, query spans with the attributes that identify tool calls and important cases (for example, by tool name, error flag, or quality score attribute).
   • Transform those query results into an evaluation dataset schema used by the evaluation framework (for example, a table or structured dataset with input, expected output, and metadata).
   • Store the resulting evaluation dataset in a governed store (such as a cataloged table) and keep it versioned for regression testing.
6. Security and governance when exporting
   Treat trace data as production telemetry:
   • Do not export secrets or credentials when building datasets.
   • Minimize or redact personal data before it appears in telemetry or in the derived evaluation dataset.
   • Apply the same access controls and retention policies to trace-derived datasets that are used for logs and metrics.

This pattern—rich, consistent tracing with OpenTelemetry, exporting via trace exporters, querying spans with tool-call attributes, and transforming them into a structured evaluation dataset—is the recommended way to leverage agent traces for evaluation.

References:

• Trace and observe AI agents in Microsoft Foundry (preview) (classic)
• Agent tracing overview (preview)
• Building MLflow evaluation datasets
• Tutorial: Evaluate and improve a GenAI application