Guidance Request: Dataset Creation for Agent Evaluation in Foundry Portal

Question

Guidance Request: Dataset Creation for Agent Evaluation in Foundry Portal

Ipsita Dutta 0

Our goal is to evaluate agents in the Foundry portal, but I need guidance on how to design the dataset for the following two scenarios, especially given our restriction that synthetic data cannot be used.

Document Q&A / extraction agent (user uploads a document and asks questions; agent extracts or summarizes information).
Agent that takes a document, a JSON schema, and a prompt, and returns a JSON response.

Could you please advise:

How to construct the dataset for these cases?
What are the mandatory dataset fields and their exact names?
How can we handle large inputs (2–3 MB documents / JSON) in the dataset, since extracting text is not feasible in our environment?Our goal is to evaluate agents in the Foundry portal, but I need guidance on how to design the dataset for the following two scenarios, especially given our restriction that synthetic data cannot be used.
1. Document Q&A / extraction agent (user uploads a document and asks questions; agent extracts or summarizes information).
2. Agent that takes a document, a JSON schema, and a prompt, and returns a JSON response.
Could you please advise:
- How to construct the dataset for these cases?
- What are the mandatory dataset fields and their exact names?
- How to handle moderate size input files?
- How can we handle large inputs (2–3 MB documents / JSON) in the dataset, since extracting text is not feasible in our environment?

Anshika Varshney 14,870 Reputation points Microsoft External Staff Moderator

2026-07-08T17:21:34.1033333+00:00
Hello @Ipsita Dutta

Thank you for reaching out.

For Agent Evaluation in Azure AI Foundry, the dataset structure depends on the evaluation scenario. Since your environment does not allow synthetic data, you can create the evaluation dataset from real user interactions, historical documents, validated business records, or production test cases.

For a document Q&A or extraction agent, a dataset typically includes:

Query (user question)

Input document reference or content

Expected answer (ground truth)

Optional evaluation metadata such as category or scenario

For an agent that processes a document, JSON schema, and prompt to generate structured JSON output, the dataset should contain:

Input document reference

Prompt/instructions

Schema definition

Expected JSON output (ground truth)

Regarding large files (2–3 MB documents/JSON), it is generally recommended to store the source files externally and reference them in the evaluation workflow rather than embedding the entire content directly into dataset rows. This helps keep datasets manageable and easier to maintain.

You may also find the following guidance helpful:

Agent evaluation concepts in Azure AI Foundry: https://learn.microsoft.com/azure/ai-foundry/how-to/evaluate-generative-ai-app

Evaluation and dataset guidance: https://learn.microsoft.com/azure/ai-foundry/concepts/evaluation-approach-gen-ai

To better assist, could you please clarify:

Are you using Agent Evaluation from the Foundry portal UI or the Evaluation SDK?

What dataset format are you planning to use (JSONL, CSV, or another format)?

Are your documents stored in Azure Storage, SharePoint, or another repository?

This will help provide more specific guidance for your scenario.

Thankyou!
Anshika Varshney 14,870 Reputation points Microsoft External Staff Moderator

2026-07-09T20:22:34.65+00:00

Hello @Ipsita Dutta

Did you get any chance to review the response.

Thankyou!
Anshika Varshney 14,870 Reputation points Microsoft External Staff Moderator

2026-07-12T21:35:10.1033333+00:00

Hello @Ipsita Dutta

Please let me know if the issue persists after these checks. If you have any remaining questions or need additional details, I’ll be glad to provide further clarification or guidance.

Thankyou!
Ipsita Dutta 0 Reputation points

2026-07-13T03:37:48.85+00:00

sure, I will check and reply, thanks so much for reply.
Ipsita Dutta 0 Reputation points

2026-07-13T04:45:07.7666667+00:00
Thanks for the clarification questions. Here is the current understanding:

We plan to use the Agent Evaluation from the Foundry portal UI.

The dataset format will be JSONL. Other than csv. jsonl any other format supported?

Some source documents may be temporarily stored in Azure Blob Storage, but not all documents will be maintained there permanently.

Regarding dataset structure for document Q&A / extraction scenarios, I understand the typical fields include:

Query (user question)

Input document reference or content

Expected answer (ground truth)

Optional evaluation metadata (category / scenario)

I would like to confirm:

Which of these fields are mandatory in the Foundry evaluation dataset?

Is the ground truth field required/mandatory for evaluation to run?

Does Foundry perform any semantic comparison of ground truth ?

For the second scenario (document + JSON schema + prompt → structured JSON output):

If the expected JSON output (ground truth) is not available (since it is not provided by business and the output is large and complex), how should we proceed with evaluation?

· In our case, both the JSON schema and the prompt are also large in size. But the same JSON schema and the prompt is used for all the testcases.

Also, regarding large files (2–3 MB):

Could you please advise the recommended way to store source files externally and reference them in the evaluation workflow . Thanks for the clarification questions. Here is the current understanding:

We plan to use the Agent Evaluation from the Foundry portal UI.

The dataset format will be JSONL. Other than csv. jsonl any other format supported?

Some source documents may be temporarily stored in Azure Blob Storage, but not all documents will be maintained there permanently.

Regarding dataset structure for document Q&A / extraction scenarios, I understand the typical fields include:

Query (user question)

Input document reference or content

Expected answer (ground truth)

Optional evaluation metadata (category / scenario)

I would like to confirm:

Which of these fields are mandatory in the Foundry evaluation dataset?

Is the ground truth field required/mandatory for evaluation to run?

Does Foundry perform any semantic comparison of ground truth ?

For the second scenario (document + JSON schema + prompt → structured JSON output):

If the expected JSON output (ground truth) is not available (since it is not provided by business and the output is large and complex), how should we proceed with evaluation?

· In our case, both the JSON schema and the prompt are also large in size. But the same JSON schema and the prompt is used for all the testcases. Also, regarding large files (2–3 MB):

Could you please advise the recommended way to store source files externally and reference them in the evaluation workflow .
Ipsita Dutta 0 Reputation points

2026-07-13T04:47:51.32+00:00

Additionally, could you confirm up to what size it is recommended to include the full content of files in the dataset (rather than referencing them via external storage links)?
Anshika Varshney 14,870 Reputation points Microsoft External Staff Moderator

2026-07-15T21:03:35.93+00:00
Hello @Ipsita Dutta

Thank you for the additional details.

Based on the information provided, the exact required fields depend on the evaluation configuration and evaluator you select in the Azure AI Foundry portal. The dataset schema is typically mapped during evaluation setup, so there is no single mandatory set of field names that applies to all evaluation scenarios.

Regarding ground truth, its necessity depends on the evaluation method being used. Evaluations that measure correctness against expected outputs generally require a reference answer or expected result, while some AI-assisted evaluation approaches may assess quality dimensions without requiring explicit ground truth.

For your Document Q&A / Extraction scenario, a JSONL dataset would commonly contain:

{ "query": "user question", "document": "document content or reference", "ground_truth": "expected answer", "category": "optional metadata" }

For your Translator Agent, a typical evaluation dataset could contain:

{ "source_text": "original text", "target_language": "French", "ground_truth_translation": "expected translated text" }

or, if documents are being translated:

{ "document_reference": "blob/file reference", "target_language": "German", "ground_truth_translation": "expected translated output" }

For the scenario where large documents, prompts, and JSON schemas are reused across all test cases, it is generally preferable to store those artifacts externally and reference them rather than duplicating them in every dataset row. This helps keep the dataset manageable and easier to maintain.

Please allow me to engage the Azure AI Foundry product team or subject matter experts for confirmation on these platform-specific behaviors. Their guidance would provide the most accurate and authoritative answer for your implementation.

Thank you!
Ipsita Dutta 0 Reputation points

2026-07-16T13:03:57.7866667+00:00

Hi @Anshika Varshney , Thanks so much for the support. Please take your time to check with the team.

One quick question how I will map the fields for translator agent , source_text and target_language? I guess source text I can map with query but what about target_language?
Anshika Varshney 14,870 Reputation points Microsoft External Staff Moderator

2026-07-16T17:18:54.74+00:00
Hello @Ipsita Dutta

For a translator agent, you can structure the evaluation dataset similar to other text generation scenarios.

Example fields:

{ "query": "Hello, how are you?", "target_language": "French", "ground_truth": "Bonjour, comment allez-vous ?" }

or for document translation:

{ "document_reference": "blob://sample-document.pdf", "target_language": "German", "ground_truth": "Expected translated content" } ``

Regarding field mapping:

query → Source text (or the document content/reference if translating a document)

target_language → The language the translation should be generated in (for example: French, German, Spanish, Japanese)

ground_truth → Expected translated output (if available)

If your translator agent requires both the source text and target language as inputs, then both should be included as separate dataset columns and mapped accordingly during evaluation. For translation quality metrics, having a ground-truth translation is recommended because many evaluators compare the generated translation against the expected output. If a ground truth is not available, you may be limited to quality or LLM-based evaluation methods rather than reference-based comparison.

Thankyou!
Ipsita Dutta 0 Reputation points

2026-07-17T13:25:47.7633333+00:00

Hi @Anshika Varshney Thanks, but still not clear about target_language field mapping, I understand I should have this field in my dataset jsonl , but in the portal with which option I can map this to, I am not getting any appropriate mapping. PFA for the same, with which one (field) in the portal it should be mapped
Anshika Varshney 14,870 Reputation points Microsoft External Staff Moderator

2026-07-19T23:10:19.7933333+00:00

Hello @Ipsita Dutta

We need more details to assist you on this case. I’ve reached out to you via private message to request a few additional details that will help us investigate this further.

Thankyou!

2 answers

Your answer

Anshika Varshney 14,870 Reputation points Microsoft External Staff Moderator

2026-07-08T17:21:34.1033333+00:00

Hello @Ipsita Dutta

Thank you for reaching out.

For Agent Evaluation in Azure AI Foundry, the dataset structure depends on the evaluation scenario. Since your environment does not allow synthetic data, you can create the evaluation dataset from real user interactions, historical documents, validated business records, or production test cases.

For a document Q&A or extraction agent, a dataset typically includes:

Query (user question)

Input document reference or content

Expected answer (ground truth)

Optional evaluation metadata such as category or scenario

For an agent that processes a document, JSON schema, and prompt to generate structured JSON output, the dataset should contain:

Input document reference

Prompt/instructions

Schema definition

Expected JSON output (ground truth)

Regarding large files (2–3 MB documents/JSON), it is generally recommended to store the source files externally and reference them in the evaluation workflow rather than embedding the entire content directly into dataset rows. This helps keep datasets manageable and easier to maintain.

You may also find the following guidance helpful:

Agent evaluation concepts in Azure AI Foundry: https://learn.microsoft.com/azure/ai-foundry/how-to/evaluate-generative-ai-app

Evaluation and dataset guidance: https://learn.microsoft.com/azure/ai-foundry/concepts/evaluation-approach-gen-ai

To better assist, could you please clarify:

Are you using Agent Evaluation from the Foundry portal UI or the Evaluation SDK?

What dataset format are you planning to use (JSONL, CSV, or another format)?

Are your documents stored in Azure Storage, SharePoint, or another repository?

This will help provide more specific guidance for your scenario.

Thankyou!
Anshika Varshney 14,870 Reputation points Microsoft External Staff Moderator

2026-07-09T20:22:34.65+00:00

Hello @Ipsita Dutta

Did you get any chance to review the response.

Thankyou!
Anshika Varshney 14,870 Reputation points Microsoft External Staff Moderator

2026-07-12T21:35:10.1033333+00:00

Hello @Ipsita Dutta

Please let me know if the issue persists after these checks. If you have any remaining questions or need additional details, I’ll be glad to provide further clarification or guidance.

Thankyou!
Ipsita Dutta 0 Reputation points

2026-07-13T03:37:48.85+00:00

sure, I will check and reply, thanks so much for reply.
Ipsita Dutta 0 Reputation points

2026-07-13T04:47:51.32+00:00

Additionally, could you confirm up to what size it is recommended to include the full content of files in the dataset (rather than referencing them via external storage links)?
Anshika Varshney 14,870 Reputation points Microsoft External Staff Moderator

2026-07-15T21:03:35.93+00:00

Hello @Ipsita Dutta

Thank you for the additional details.

Based on the information provided, the exact required fields depend on the evaluation configuration and evaluator you select in the Azure AI Foundry portal. The dataset schema is typically mapped during evaluation setup, so there is no single mandatory set of field names that applies to all evaluation scenarios.

Regarding ground truth, its necessity depends on the evaluation method being used. Evaluations that measure correctness against expected outputs generally require a reference answer or expected result, while some AI-assisted evaluation approaches may assess quality dimensions without requiring explicit ground truth.

For your Document Q&A / Extraction scenario, a JSONL dataset would commonly contain:

{ "query": "user question", "document": "document content or reference", "ground_truth": "expected answer", "category": "optional metadata" }

For your Translator Agent, a typical evaluation dataset could contain:

{ "source_text": "original text", "target_language": "French", "ground_truth_translation": "expected translated text" }

or, if documents are being translated:

{ "document_reference": "blob/file reference", "target_language": "German", "ground_truth_translation": "expected translated output" }

For the scenario where large documents, prompts, and JSON schemas are reused across all test cases, it is generally preferable to store those artifacts externally and reference them rather than duplicating them in every dataset row. This helps keep the dataset manageable and easier to maintain.

Please allow me to engage the Azure AI Foundry product team or subject matter experts for confirmation on these platform-specific behaviors. Their guidance would provide the most accurate and authoritative answer for your implementation.

Thank you!
Ipsita Dutta 0 Reputation points

2026-07-16T13:03:57.7866667+00:00

Hi @Anshika Varshney , Thanks so much for the support. Please take your time to check with the team.

One quick question how I will map the fields for translator agent , source_text and target_language? I guess source text I can map with query but what about target_language?
Anshika Varshney 14,870 Reputation points Microsoft External Staff Moderator

2026-07-16T17:18:54.74+00:00

Hello @Ipsita Dutta

For a translator agent, you can structure the evaluation dataset similar to other text generation scenarios.

Example fields:

{ "query": "Hello, how are you?", "target_language": "French", "ground_truth": "Bonjour, comment allez-vous ?" }

or for document translation:

{ "document_reference": "blob://sample-document.pdf", "target_language": "German", "ground_truth": "Expected translated content" } ``

Regarding field mapping:

query → Source text (or the document content/reference if translating a document)

target_language → The language the translation should be generated in (for example: French, German, Spanish, Japanese)

ground_truth → Expected translated output (if available)

If your translator agent requires both the source text and target language as inputs, then both should be included as separate dataset columns and mapped accordingly during evaluation. For translation quality metrics, having a ground-truth translation is recommended because many evaluators compare the generated translation against the expected output. If a ground truth is not available, you may be limited to quality or LLM-based evaluation methods rather than reference-based comparison.

Thankyou!
Ipsita Dutta 0 Reputation points

2026-07-17T13:25:47.7633333+00:00

Hi @Anshika Varshney Thanks, but still not clear about target_language field mapping, I understand I should have this field in my dataset jsonl , but in the portal with which option I can map this to, I am not getting any appropriate mapping. PFA for the same, with which one (field) in the portal it should be mapped
Anshika Varshney 14,870 Reputation points Microsoft External Staff Moderator

2026-07-19T23:10:19.7933333+00:00

Hello @Ipsita Dutta

We need more details to assist you on this case. I’ve reached out to you via private message to request a few additional details that will help us investigate this further.

Thankyou!

Answer 1

Ipsita Dutta 0

Hi @Anshika Varshney Can you please answer my questions when you have a moment?

0 comments

Answer 2

Ipsita Dutta 0

We also have a translator agent that can take either plain text input or an uploaded document and returns the translated output. Could you please guide us on how to structure the evaluation dataset for this type of agent?

0 comments