Create an evaluation dataset and evaluators

The agent optimizer evaluates your agent against a dataset - a collection of tasks - scored by evaluators. You can generate both automatically from the CLI or create a dataset manually for full control.

Both parts are essential to good optimization: the dataset defines what to test, and the evaluators define how to judge each response. Weak evaluators produce noisy scores that lead to poor optimization, so invest in strong evaluators as much as representative tasks.

Creating these assets is the second step in the optimization workflow, after you make your agent optimizer-ready. The optimizer uses them to score your baseline and rank candidates.

Prerequisites

• A Foundry project with a deployed hosted agent.
• The Azure Developer CLI (azd), version 1.37.0 or later, which the azure.ai.agents extension requires.
• The azure.ai.agents CLI extension, version 1.0.0-beta.22 or later, installed (see Quickstart: Optimize a hosted agent).
• The azure.ai.evaluations extension, which provides the azd ai eval commands and requires azd 1.33.0 or later. The microsoft.foundry extension doesn't include it, so install it separately.
• An authenticated azd session. To check your authentication status, run azd auth status. If you're not signed in, run azd auth login.

Generate a dataset and evaluators (recommended)

The optimizer reuses evaluation assets that you create with the azd ai eval commands from the azure.ai.evaluations extension. The azd ai agent eval command group is deprecated. It only prints migration guidance to azd ai eval, and it doesn't make service requests or change project files. The optimizer doesn't support the legacy agent eval.yaml file.

Generate a dataset and a rubric evaluator from your agent's instructions:

azd extension install azure.ai.evaluations
azd ai eval generate \
  --from prompt \
  --target <agent-name> \
  --generation-model <generation-deployment> \
  --agent-instruction "<agent-instructions>" \
  --no-prompt

By default, the command generates:

• A dataset of rows for your agent.
• A custom rubric evaluator, which is an editable JSON file of weighted scoring dimensions.

The command downloads both artifacts and adds a catalog entry for each to the evaluation configuration, which is evals/azure.eval.yaml by default. Unless you set --dataset-name or --evaluator-name, the dataset is named <deployed-agent>-turn-tests and the evaluator is named <deployed-agent>-evaluator. For the interactive and non-interactive options and details about the generated artifacts, see Generate a dataset and an evaluator.

Then scaffold an evaluation that uses the generated assets. The optimizer reuses turn-level evaluations that target your agent, so set --evaluation-level turn and --target. This example adds the built-in builtin.task_adherence evaluator to the generated rubric evaluator:

azd ai eval init \
  --source dataset \
  --evaluation-level turn \
  --target <agent-name> \
  --dataset <generated-dataset-name> \
  --evaluator <generated-evaluator-name>,builtin.task_adherence \
  --judge-model <judge-deployment> \
  --name optimizer-eval \
  --no-prompt

Run the optimizer and select the evaluation by name:

azd ai agent optimize --eval optimizer-eval --optimize-model <optimization-deployment>

To customize the generated assets, see Customize evaluators and Create a custom dataset. To choose which evaluation a run uses, see Select the evaluation for a run. For the other run options, run azd ai agent optimize --help.

Select the evaluation for a run

azd ai agent optimize reads the dataset and evaluators from an evaluation that azd ai eval init declared. Select the evaluation with one of these options:

The optimizer applies these rules:

• It considers only evaluations that have an agent target. The target name must match the deployed agent name or the agent's service name in azure.yaml.
• It reuses only turn-level evaluations that read a dataset. Evaluations that declare a source or a simulation, or that use the conversation level, aren't supported, and neither are evaluators that set dataMapping.
• If you pass neither option and the project has one matching evaluation, the optimizer uses it. If several match, the optimizer prompts you to select one. With --no-prompt or -o json, the command fails and asks you to pass --eval.
• If you pass --eval or --eval-config and no evaluation matches, the command fails.

The --config option takes a standalone optimization request file. You can't combine it with --eval or --eval-config.

Override or extend what the evaluation provides with these flags:

• --dataset replaces the evaluation's dataset. Pass a local JSONL file or a registered dataset as name:version.
• --evaluator adds a built-in or custom evaluator by name. Repeat the flag to add more than one. An evaluator that the evaluation already lists isn't added twice.
• --eval-model sets the model that judges responses. Without it, the optimizer uses the model that the evaluation's evaluators declare in their initialization parameters. If they declare different models, pass --eval-model.

If the project has no evaluation that targets your agent, pass the dataset and evaluators directly:

azd ai agent optimize <agent-name> \
  --dataset eval.jsonl \
  --evaluator builtin.task_adherence \
  --eval-model <judge-deployment> \
  --optimize-model <optimization-deployment>

Customize evaluators (advanced)

Evaluators score each agent response. The optimizer supports two kinds:

• Built-in evaluators, such as builtin.task_adherence, which scores each task-level criterion as pass or fail.
• Custom rubric evaluators, which score responses across several quality dimensions tuned to your agent. azd ai eval generate creates one and downloads it as an editable rubric JSON file.

For most agents, the generated rubric evaluator gives the most meaningful scores because it's tailored to your domain. Edit the generated rubric file to refine dimensions, and then run azd ai eval evaluator update <name> --from-file <path> to publish the changes as a new version. For details on generating, editing, and versioning evaluators, see Generate a dataset and an evaluator.

To use an evaluator in a run, list it in the evaluation with azd ai eval init --evaluator, or pass it with azd ai agent optimize --evaluator <name>. See Select the evaluation for a run.

Create a custom dataset (advanced)

Create a custom dataset when you need precise control over test scenarios or have production data to use directly. The recommended approach is to iterate on top of the dataset that azd ai eval generate produces. Refine it into a local dataset, or point to another dataset already registered in your Foundry project.

Choose a dataset source

A dataset can come from either of two sources:

• Foundry dataset: a dataset already registered in your Foundry project. Pass it with --dataset <name>:<version>, or declare it in the datasets catalog of your evaluation configuration with a name and a version. The optimizer needs both the name and the version.
• Local dataset: a JSONL file you author and keep in your project. Pass its path with --dataset, or declare it in the datasets catalog with file. A relative --dataset path resolves from the current directory. A relative file path resolves from the evaluation configuration file.

Both sources use the same task schema described in the next section. To select a dataset for a run, see Select the evaluation for a run.

Dataset schema

A dataset uses JSONL (JSON Lines) format. Each line is one JSON object that represents a single evaluation task—an individual scenario. A task has a prompt (query) and, optionally, task-level criteria.

{"name": "task_1", "query": "Your prompt here"}
{"name": "task_2", "query": "Another prompt", "ground_truth": "Expected answer"}

When you use a local dataset, validate the JSONL syntax before you run optimization:

python -c "import json; [json.loads(l) for l in open('eval.jsonl')]"

Add task-level criteria

Criteria are optional. The evaluators you select for the run apply to every task in the dataset. Add per-task criteria only when a specific task needs checks beyond those shared evaluators. When present, a task's criteria are scored and aggregated together with the shared evaluators to produce the task's overall score.

The following customer-support dataset shows tasks with task-level criteria:

{"name": "refund_policy", "query": "What is your refund policy?", "criteria": [{"name": "mentions_30_days", "instruction": "Response must mention the 30-day refund window"}, {"name": "polite_tone", "instruction": "Response must be professional and empathetic"}]}
{"name": "order_status", "query": "Where is my order #12345?", "criteria": [{"name": "asks_for_details", "instruction": "Agent should ask for email or order details to look up the order"}, {"name": "no_hallucination", "instruction": "Agent must NOT make up a fake order status"}]}
{"name": "out_of_scope", "query": "Can you help me fix my car?", "criteria": [{"name": "polite_decline", "instruction": "Agent should politely explain this is outside its scope"}, {"name": "redirect", "instruction": "Agent should suggest contacting an appropriate service"}]}

Register and version a dataset with the CLI

For reusable datasets beyond one optimizer run, install the separately released dataset extension:

azd ext install azure.ai.dataset
azd ai project set <project-endpoint>
azd ai dataset create support-golden --from-file ./support-golden.jsonl

--from-file accepts a JSONL file or a directory containing one. Use --version to choose a version explicitly; otherwise the command publishes the next version. Keep the version used for an evaluation fixed when comparing agent changes.

Publish revised data and inspect the versions:

azd ai dataset update support-golden --from-file ./support-golden-v2.jsonl
azd ai dataset list --all
azd ai dataset show support-golden
azd ai dataset versions list support-golden

list returns 20 entries by default. Use --limit for a smaller page, --all for all pages, and repeat --tag key=value to require multiple tag matches.

Download a registered version to reproduce a run:

azd ai dataset download support-golden --version 1.0 \
  --output-dir ./downloaded-dataset

Use --output-file only for a single-file dataset. Existing files aren't overwritten unless you pass --force. Omitting --version downloads the latest version and reports which version was selected.

To remove a version, run azd ai dataset delete support-golden --version <version>. The --version flag is required; omitting it returns an error. Noninteractive deletion requires explicit --force.

The azd ai eval dataset group exposes the same lifecycle through the evaluation extension. Registering data doesn't make every JSONL schema interchangeable: keep the row shape required by your evaluator and target.

Tips for writing good datasets

Include edge cases

Test beyond the happy path. Include:

• Out-of-scope requests — Inputs your agent should decline or redirect
• Ambiguous queries — Tasks where the agent should ask for clarification
• Adversarial inputs — Attempts to trick the agent into bad behavior
• Multi-step tasks — Complex requests that require structured reasoning

Size guidelines

Larger datasets give broader coverage but take longer to evaluate.

Provide ground truth when useful

The ground_truth field gives evaluators a reference answer to compare against. It's not required - evaluators can also judge responses from their instructions and any task-level criteria alone.

{"name": "geography_fact", "query": "What is the largest city in France by population?", "ground_truth": "Paris", "criteria": [{"name": "correct_answer", "instruction": "Response must state that Paris is the largest city in France by population"}]}

Write prompts like real users

Use actual messages from your users if possible. Real prompts capture the vocabulary and context that your agent faces in production, which also helps you write realistic task-level criteria.

Be specific in criteria

Vague criteria lead to inconsistent scoring. Make each criterion specific and testable.

Bad:

{"name": "good_answer", "instruction": "The response should be good"}

Good:

{"name": "mentions_30_days", "instruction": "Response must explicitly mention the 30-day refund window"}

Troubleshooting

Related content

• Agent optimizer overview
• Make your agent optimizer-ready
• Optimize agent instructions, skills, tools, and models
• Quickstart: Optimize a hosted agent
• Run agent evaluations with the azd CLI