Create an evaluation dataset and evaluators

The agent optimizer evaluates your agent against a dataset - a collection of tasks - scored by evaluators. You can generate both automatically from the CLI or create a dataset manually for full control.

Both parts are essential to good optimization: the dataset defines what to test, and the evaluators define how to judge each response. Weak evaluators produce noisy scores that lead to poor optimization, so invest in strong evaluators as much as representative tasks.

Creating these assets is the second step in the optimization workflow, after you make your agent optimizer-ready. The optimizer uses them to score your baseline and rank candidates.

Prerequisites

• A Foundry project with a deployed hosted agent
• The azure.ai.agents CLI extension installed (see Quickstart: Optimize a hosted agent)

Generate a dataset and evaluators (recommended)

The fastest way to create evaluation assets is with azd ai agent eval generate. The command auto-detects your agent and generates everything the optimizer needs:

azd ai agent eval generate

By default, it generates:

• A seed dataset of tasks tuned to your agent's domain.
• Evaluators that score responses - a built-in evaluator (such as builtin.task_adherence) plus a custom rubric evaluator tailored to your agent.
• A runnable eval.yaml that wires them together.

Use --gen-instruction or --gen-instruction-file to describe your test scenarios. Select the generation model with --eval-model, the output configuration with --out-file, and the sample count with --max-samples (15-1,000). Add --no-wait to submit without waiting.

Use --dataset to supply an existing dataset and --evaluator to select evaluator names. This quick-evaluation workflow writes eval.yaml; the separate project evaluation extension writes evals/azure.eval.yaml.

After generation, azd ai agent optimize auto-detects eval.yaml:

azd ai agent optimize

To customize the generated assets, see Customize evaluators and Create a custom dataset. To change run options, edit eval.yaml; see Configure the optimization run.

Customize evaluators (advanced)

Evaluators score each agent response. The optimizer supports two kinds:

• Built-in evaluators, such as builtin.task_adherence, which scores each task-level criterion as pass or fail.
• Custom rubric evaluators, which score responses across several quality dimensions tuned to your agent. azd ai agent eval generate creates one automatically as an editable rubric_dimensions.json file.

Edit the generated rubric_dimensions.json to refine dimensions, then run azd ai agent eval update to register the changes. Use --dataset-only or --evaluator-only to limit which assets are updated, and --config to select a configuration.

To wire evaluators into your run configuration, see Configure the optimization run.

Create a custom dataset (advanced)

Create a custom dataset when you need precise control over test scenarios or have production data to use directly. The recommended approach is to iterate on top of the seed dataset that azd ai agent eval generate produces—refine it into a local dataset, or point to another dataset already registered in your Foundry project.

Choose a dataset source

A dataset can come from either of two sources:

• Foundry dataset — a dataset already registered in your Foundry project. Reference it in eval.yaml by name and version.
• Local dataset — a JSONL file you author and keep in your project. Reference it in eval.yaml by local_uri.

Both sources use the same task schema described in the next section. For the eval.yaml wiring, see Configure the optimization run.

Dataset schema

A dataset uses JSONL (JSON Lines) format. Each line is one JSON object that represents a single evaluation task—an individual scenario. A task has a prompt (query) and, optionally, task-level criteria.

{"name": "task_1", "query": "Your prompt here"}
{"name": "task_2", "query": "Another prompt", "ground_truth": "Expected answer"}

When you use a local dataset, validate the JSONL syntax before you run optimization:

python -c "import json; [json.loads(l) for l in open('eval.jsonl')]"

Add task-level criteria

Criteria are optional. The evaluators you configure in eval.yaml apply to every task in the dataset. Add per-task criteria only when a specific task needs checks beyond those shared evaluators. When present, a task's criteria are scored and aggregated together with the shared evaluators to produce the task's overall score.

The following customer-support dataset shows tasks with task-level criteria:

{"name": "refund_policy", "query": "What is your refund policy?", "criteria": [{"name": "mentions_30_days", "instruction": "Response must mention the 30-day refund window"}, {"name": "polite_tone", "instruction": "Response must be professional and empathetic"}]}
{"name": "order_status", "query": "Where is my order #12345?", "criteria": [{"name": "asks_for_details", "instruction": "Agent should ask for email or order details to look up the order"}, {"name": "no_hallucination", "instruction": "Agent must NOT make up a fake order status"}]}
{"name": "out_of_scope", "query": "Can you help me fix my car?", "criteria": [{"name": "polite_decline", "instruction": "Agent should politely explain this is outside its scope"}, {"name": "redirect", "instruction": "Agent should suggest contacting an appropriate service"}]}

Register and version a dataset with the CLI

For reusable datasets beyond one optimizer run, install the separately released dataset extension:

azd ext install azure.ai.dataset
azd ai project set <project-endpoint>
azd ai dataset create support-golden --from-file ./support-golden.jsonl

--from-file accepts a JSONL file or a directory containing one. Use --version to choose a version explicitly; otherwise the command publishes the next version. Keep the version used for an evaluation fixed when comparing agent changes.

Publish revised data and inspect the versions:

azd ai dataset update support-golden --from-file ./support-golden-v2.jsonl
azd ai dataset list --all
azd ai dataset show support-golden
azd ai dataset versions list support-golden

list returns 20 entries by default. Use --limit for a smaller page, --all for all pages, and repeat --tag key=value to require multiple tag matches.

Download a registered version to reproduce a run:

azd ai dataset download support-golden --version 1 \
  --output-dir ./downloaded-dataset

Use --output-file only for a single-file dataset. Existing files aren't overwritten unless you pass --force. Omitting --version downloads the latest version and reports which version was selected.

To remove a version, run azd ai dataset delete support-golden --version <version>. The --version flag is required; omitting it returns an error. Noninteractive deletion requires explicit --force.

The azd ai eval dataset group exposes the same lifecycle through the evaluation extension. Registering data doesn't make every JSONL schema interchangeable: keep the row shape required by your evaluator and target.

Tips for writing good datasets

Include edge cases

Test beyond the happy path. Include:

• Out-of-scope requests — Inputs your agent should decline or redirect
• Ambiguous queries — Tasks where the agent should ask for clarification
• Adversarial inputs — Attempts to trick the agent into bad behavior
• Multi-step tasks — Complex requests that require structured reasoning

Size guidelines

Larger datasets give broader coverage but take longer to evaluate.

Provide ground truth when useful

The ground_truth field gives evaluators a reference answer to compare against. It's not required - evaluators can also judge responses from their instructions and any task-level criteria alone.

{"name": "geography_fact", "query": "What is the largest city in France by population?", "ground_truth": "Paris", "criteria": [{"name": "correct_answer", "instruction": "Response must state that Paris is the largest city in France by population"}]}

Write prompts like real users

Use actual messages from your users if possible. Real prompts capture the vocabulary and context that your agent faces in production, which also helps you write realistic task-level criteria.

Be specific in criteria

Vague criteria lead to inconsistent scoring. Make each criterion specific and testable.

Bad:

{"name": "good_answer", "instruction": "The response should be good"}

Good:

{"name": "mentions_30_days", "instruction": "Response must explicitly mention the 30-day refund window"}

Troubleshooting

Related content

• Agent optimizer overview
• Make your agent optimizer-ready
• Optimize agent instructions, skills, tools, and models
• Quickstart: Optimize a hosted agent
• Run agent evaluations with the azd CLI