Run agent evaluations with the azd CLI (preview)

Use the azd ai eval extension to add a measured quality loop to an agent built with Microsoft Foundry. You scaffold an evaluation next to your project, optionally generate a dataset and a rubric evaluator, run the evaluation against your agent, and read the results without leaving the terminal.

The same evaluation can run from a pipeline, and --fail-on turns its results into a build gate.

This article covers the first evaluation with azd ai eval init and azd ai eval run start.

Prerequisites

• An Azure subscription with access to Microsoft Foundry.
• The Azure Developer CLI (azd), version 1.33.0 or later. For installation instructions, see Install the Azure Developer CLI.
• The azure.ai.evaluations extension, version 1.0.0-beta.2 or later, which provides the azd ai eval commands. See Install the extension.
• An authenticated azd session. To check your authentication status, run azd auth status. If you're not signed in, run azd auth login.
• The Foundry User role on the Foundry resource (previously named Azure AI User). For more information, see Role-based access control for Microsoft Foundry.
• For all evaluation role requirements, see Set up permissions for evaluation workflows.
• An initialized azd project and environment that identifies your Foundry project. For setup, see Set the Foundry project context.
• An agent for turn datasets, simulated conversations, or traces. To let init detect it locally, declare an agent service in azure.yaml, as azd ai agent init does. Otherwise, pass its deployed name with --target. Static conversations don't require an agent target. For hosted agents, see Hosted agents.
• Supported model deployments for the roles you use: grading, optional artifact generation, and optional user simulation. Set each role explicitly; using one deployment for a role doesn't configure the others.
• A real JSONL dataset or registered dataset for dataset-sourced evaluations. Alternatively, generate a dataset before configuring the evaluation. init doesn't create sample rows.

Install the extension

Install the azure.ai.evaluations extension, which provides the azd ai eval commands. The microsoft.foundry extension doesn't include it, so install it separately:

azd extension install azure.ai.evaluations

If the extension is already installed, upgrade it to the latest version:

azd extension upgrade azure.ai.evaluations

If you need the standalone azd ai dataset commands, also install azure.ai.dataset:

azd extension install azure.ai.dataset

Both extensions require azd 1.33.0 or later.

Verify the installation

Check the installed extension versions and the available flags:

azd version
azd extension list --installed
azd ai eval version
azd ai eval init --help
azd ai eval generate --help
azd ai eval run start --help

If you also use azure.ai.agents or the standalone azure.ai.dataset extension, check them independently:

azd ai agent version
azd ai dataset version

Confirm that the evaluation extension is version 1.0.0-beta.2 or later.

These extensions are independently versioned. The agents extension doesn't replace the evaluation extension, and the standalone dataset extension isn't required for azd ai eval dataset. If init --help doesn't list --conversation-mode and --simulation-model, upgrade the extension before you follow the conversation examples.

How azd evaluations work

You can describe an evaluation by using the evals/azure.eval.yaml file. You can read, edit, and commit this file. The commands either write that file or act on what it declares.

azd ai eval init          # scaffold the configuration without invoking an agent
azd ai eval generate      # optional: synthesize a dataset and a rubric evaluator
azd ai eval create        # register the eval in the Foundry project
azd ai eval run start     # run it and summarize the results

Use -o json for scripting and --debug for diagnostics. Service-backed commands accept --project-endpoint; init uses local project and environment metadata instead.

Choose where the rows come from

An evaluation grades rows. The examples in this article use one of two sources:

• --source traces evaluates what your agent already did by reading the traces it emitted. You don't need to author a dataset.
• --source dataset evaluates a fixed set of examples, either yours or generated. A fixed dataset lets you compare results across agent versions.

Trace-sourced evaluations need an agent that emits traces. Dataset-sourced evaluations need a .jsonl file or a registered dataset.

For a dataset, choose what a row represents:

The only --conversation-mode values are static and simulation. Either value implies --source dataset and --evaluation-level conversation when you omit those options. Traces are a separate source, not a conversation mode.

For conversation datasets, interactive init prompts for the mode. Without prompts, an omitted mode defaults to static. Select simulation explicitly when your dataset contains scenario seeds rather than completed transcripts.

Scaffold the evaluation

Use your existing initialized azd project. If you need a new local workspace for an already deployed agent, run these commands from a new directory:

azd init --minimal --no-prompt -e evaluation
azd env set FOUNDRY_PROJECT_ENDPOINT "<your-project-endpoint>"

Replace <your-project-endpoint> with your Foundry project endpoint. This setup doesn't deploy the agent.

Run init from your project root to choose the inputs interactively:

azd ai eval init

init reads local project and environment metadata to detect agent services and model deployments. It doesn't discover deployed agents or models from the service. A local agent service key passed to --target resolves to that service's declared remote agent name; you can also pass a deployed agent name directly.

init writes evals/azure.eval.yaml and adds an evaluation service to azure.yaml. It doesn't deploy or invoke an agent, generate a dataset, or register evaluation resources. You can author the configuration before deployment, but agent-based runs need a deployed target.

init works offline except for two lookups. The builtin.* evaluators that you select, including the default under --no-prompt, can trigger a best-effort catalog lookup with a five-second timeout. A qualified connection-name/model-deployment simulation reference, or connection discovery for --simulation-model, looks up the connections in the Foundry project. If the evaluator catalog lookup can't be completed, init keeps the references as written. If the lookup succeeds, init rejects a builtin.* name that the catalog doesn't list. Canceling the command still stops it.

If a required value can't be resolved locally, interactive init prompts for it. Under --no-prompt or -o json, provide all unresolved inputs explicitly. For agent-based evaluations in a project without an agent service, supply --target.

Choose one of the following initialization examples for the source and mode you need. The examples use placeholder agent, dataset, and deployment names. Replace them with your own resources; the extension doesn't supply them.

For a trace-sourced evaluation, pass the decisions directly:

azd ai eval init \
  --source traces \
  --target support-agent \
  --judge-model judge-deployment \
  --name support-trace-eval \
  --no-prompt

To evaluate a turn dataset you already have:

azd ai eval init \
  --source dataset \
  --evaluation-level turn \
  --target support-agent \
  --dataset golden-turns \
  --evaluator builtin.intent_resolution,builtin.task_adherence \
  --judge-model judge-deployment \
  --name turn-quality \
  --no-prompt

--dataset takes a local .jsonl path or the name of a registered dataset. For a local file, provide an existing dataset with the columns your evaluation requires.

--evaluator is repeatable and comma-separated. Use builtin.<name> for a built-in evaluator, or a declared custom evaluator compatible with the evaluation level. Passing --evaluator replaces the defaults; it doesn't submit generation jobs.

create validates evaluator availability, supported evaluation levels, initialization parameters, and required columns against the service schema. A successfully scaffolded file doesn't prove that every evaluator is available or compatible with your data.

To discover the built-in names:

azd ai eval evaluator list --builtin

Score completed conversations

Use a dataset of completed conversation messages, without an agent target:

azd ai eval init \
  --name static-quality \
  --conversation-mode static \
  --dataset completed-transcripts \
  --judge-model judge-deployment \
  --no-prompt

Static mode doesn't write a target or simulation configuration. It rejects an explicit --target.

Simulate conversations with an agent

Use a dataset of scenario seeds and specify the simulated user's model separately from the judge:

azd ai eval init \
  --name simulation-quality \
  --conversation-mode simulation \
  --target support-agent \
  --dataset scenario-seeds \
  --simulation-model simulator-deployment \
  --judge-model judge-deployment \
  --num-conversations 1 \
  --max-turns 5 \
  --no-prompt

The agent is invoked when you start the run, not during init. --num-conversations sets conversations per seed from 1 to 5 and defaults to 1. --max-turns accepts 1 to 20; omit it for the service default. An explicit zero isn't valid for either setting.

Before writing configuration, init validates locally available simulation seeds and prompts for a corrected dataset if they're invalid. If you specify --no-prompt or --output json, init rejects invalid local seeds without changing authored files. It doesn't fetch registered-only dataset contents.

--simulation-model takes a deployment name, or a qualified connection-name/model-deployment reference that selects an Azure OpenAI connection in the project. A deployment name doesn't need a connection lookup.

Put per-row turn settings in a seed under simulation_configuration, not at the row's top level:

{"test_case_description":"Ask about an order, then request a refund.","simulation_configuration":{"desired_num_turns":3,"max_num_turns":5}}

The desired turn count must be positive and no greater than the row's max_num_turns, or otherwise --max-turns, or the default of 20. A row's maximum overrides the authored maximum. Neither setting guarantees an exact completed conversation length.

Keep the three model roles separate:

You can explicitly select the same deployment for more than one role. The simulation model isn't inferred from the generation or judge model.

Validate local inputs and choose names

For every dataset mode, init checks local files for nonempty JSONL object rows before it creates configuration files, lock files, ignore files, or artifact directories. Malformed JSON, arrays, scalars, empty objects, and empty datasets fail without those writes. Keep the configuration destination separate from the input dataset; equivalent paths to the same file are rejected.

An explicit --name takes precedence. Otherwise, names identify the dataset or trace source, the evaluation level, and static or simulation mode. Existing declarations aren't renamed. A numeric suffix is added only when the descriptive name already exists. A static evaluation can use a sole local agent as a naming hint without invoking it.

On Windows, avoid filenames that contain command-shell environment-variable syntax, such as %TEMP%. The shell can expand that text before the extension receives the path.

Generate a dataset and an evaluator

If you don't have a dataset, or you want a rubric written for this agent, use its instructions for prompt-seeded generation:

azd ai eval generate \
  --from prompt \
  --target support-agent \
  --generation-model generation-deployment \
  --no-prompt

By default this generates both a dataset and a rubric evaluator, downloads them under evals/, and adds a catalog entry for each to evals/azure.eval.yaml. Add --dataset to generate only a dataset. To generate only an evaluator, add --evaluator and remove --from prompt, because --from applies only to dataset generation.

This unattended example uses --from prompt. If the extension can't read the target agent's instructions, supply --agent-instruction-file <path>. Agent-seeded generation with --from agent is a separate choice; --no-prompt doesn't authorize a second, billable prompt-only retry after a service failure.

Use --max-samples to request 15 to 1000 rows (default: 15). The value is a request to the generation service, so check the row count of the generated dataset.

generate submits jobs that cost model calls. To provide your own generation instructions, use --agent-instruction or --agent-instruction-file. Describe the agent's actual purpose and what you need to test.

To generate scenario seeds for simulated conversations, rather than turn-level examples:

azd ai eval generate \
  --dataset \
  --from prompt \
  --target support-agent \
  --evaluation-level conversation \
  --generation-model generation-deployment \
  --dataset-name scenario-seeds \
  --max-samples 15 \
  --no-prompt

Generated conversation datasets contain seeds, not completed transcripts. The suggested init command selects simulation mode and preserves a custom --path. Complete it with the simulation and judge model deployments; unattended use also needs --no-prompt.

A catalog entry doesn't attach the artifact to an existing evaluation. Run init with the generated dataset's name, or update the intended evaluation in evals/azure.eval.yaml. A trace-sourced evaluation continues to read traces until you configure a dataset-sourced evaluation.

To submit both jobs and collect their artifacts later:

azd ai eval generate \
  --from prompt \
  --target support-agent \
  --generation-model generation-deployment \
  --no-wait
azd ai eval job list --dataset
azd ai eval job show <dataset-job-id> --dataset
azd ai eval job list --evaluator
azd ai eval job show <evaluator-job-id> --evaluator

Use each job's distinct ID with its corresponding --dataset or --evaluator flag. While a job is running, job show reports its status without collecting an artifact. Run job show again after that job succeeds to download and declare its artifact, then attach the artifacts to your evaluation.

Supply generation instructions and names

Generated rows and rubrics are only as specific as the instructions describing the agent. generate uses the first source that exists:

1. --agent-instruction or --agent-instruction-file.
2. The baseline instructions that azd ai agent optimize leaves under .agent_configs/baseline/ in the agent's service folder.
3. The instructions published with the agent.
4. An interactive prompt. Choose Type instructions or Load from file. At the file prompt, enter the path to a nonempty text file without shell quotes. You can correct an unreadable or empty file without restarting other selections, or cancel before you submit a job.

With --no-prompt or JSON output, missing instructions produce an error that names --agent-instruction and --agent-instruction-file instead of a prompt. The confirmation identifies the instruction source, not its contents.

A one-line instruction produces generic rows. For a hosted agent whose instructions aren't published, such as a bring-your-own code agent, pass --agent-instruction or --agent-instruction-file with the agent's purpose, the kinds of requests it handles, its policies, and the edge cases you want to test.

Generated artifact names use the agent's declared deployed name, rather than its local service key or instruction source. The original target selector still determines the generation request. Generated dataset names have a 50-character limit. Long defaults are shortened deterministically, while an overlong explicit --dataset-name fails instead of being truncated. This limit doesn't rename existing artifacts or apply to evaluator names.

Edit a rubric document

Generation, completed-job collection, and rubric downloads produce an editable JSON document that contains type: "rubric", dimensions, and an optional pass_threshold. Each dimension includes id, description, weight, and always_applicable. The document preserves other authored definition and dimension fields, including numeric precision. Weights are integers from 1 to 10. A supplied threshold is between 0 and 1.

Service envelopes, generated wiring, and catalog metadata aren't rubric fields. Publishing an edited rubric preserves existing catalog metadata when you don't explicitly replace it. Prompt-based evaluator documents remain separate. Use azd ai eval evaluator show <name> --version <version> -o json to inspect the full service response.

Read the generated rubric before you rely on it. A rubric written from a brief instruction can pass nearly every row, which makes it a weak gate. Tighten the dimensions and the pass_threshold, then run create again.

Review azure.eval.yaml

Review the file that init writes. A trace-sourced evaluation looks like this:

evals:
    - name: support-trace-eval
      description: Basic quality evaluation for support-agent
      source:
        type: traces
        lookbackHours: 168
        maxTraces: 20
        agentName: support-agent
      evaluationLevel: turn
      evaluators:
        - evaluator: builtin.task_completion
          initializationParameters:
            model: judge-deployment

A turn dataset evaluation names the dataset instead of a trace source and records the agent it targets. The earlier turn-quality command passes a registered dataset name, so it writes this file and declares golden-turns in the datasets catalog without a local file:

datasets:
    - name: golden-turns
evals:
    - name: turn-quality
      description: Basic quality evaluation for support-agent
      dataset: golden-turns
      evaluationLevel: turn
      evaluators:
        - evaluator: builtin.intent_resolution
          initializationParameters:
            model: judge-deployment
        - evaluator: builtin.task_adherence
          initializationParameters:
            model: judge-deployment
      target:
        type: agent
        name: support-agent

A local dataset names its rows with file:, and a custom evaluator names its rubric with source:. Both paths are relative to the configuration file. If you pass a local path such as --dataset tests/support-golden.jsonl, init declares a dataset named support-golden with file: ../tests/support-golden.jsonl when the configuration is in evals/. A registered dataset can declare its name and optional version without a local file.

The generated .jsonl and evaluator JSON are ordinary files: edit them, then run create again to register a new version.

Commit this file. It's the reproducible part of the evaluation.

Review evaluator input mappings

For standard datasets and traces, the default mappings contain exactly these evaluator inputs:

Generated and retrieved outputs keep their separate provenance. Model output uses sample.output_text. Agent output and stored responses use sample.output_items or sample.output_text according to the evaluator's response type. Agent-generated tools use the corresponding sample fields. An agent name used to filter traces doesn't invoke that agent or turn trace fields into generated output.

Explicit mappings override the corresponding defaults. Use either messages or the turn query/response format, not both. The extension doesn't infer context or ground_truth, even when the dataset contains those columns. Supply real supporting data and map additional evaluator inputs explicitly.

Before it publishes dependencies, the extension checks that primary mapped inputs exist in every applicable dataset row. Optional tool columns aren't made mandatory, and generated outputs aren't mistaken for dataset input columns. Correct mappings alone don't guarantee successful groundedness or other hosted evaluator results.

Migrate an existing evaluation deliberately

Stored mappings are immutable. Upgrading the extension doesn't replace mappings in an existing evaluation. For a deliberate migration:

1. Record the old evaluation ID and inspect it with azd ai eval show <eval-id> -o json.
2. Remove an explicit id: from the declaration you intend to migrate, or create a separate unpinned declaration. An explicit ID keeps selecting the old definition.
3. Set the intended dataMapping entries and change an evaluator reference's criterion name. Changing only the evaluation group's name isn't enough.
4. Run azd ai eval create <eval-name>, then inspect the resulting ID and mappings before you start a run.

The previous evaluation and its runs remain available through the recorded ID. Changing source mode creates a new definition when the stored input contract differs. Changing only compatible run filters, time windows, or response IDs doesn't discard history.

Create the eval and run it

Use create to register the selected evaluation and its declared datasets and evaluators.

The commands in the remaining sections use support-trace-eval. Replace it with the name from your selected initialization example: turn-quality, static-quality, or simulation-quality for dataset evaluations.

azd ai eval create

If the configuration declares exactly one evaluation, you can omit its name. Otherwise, select one with azd ai eval create <eval-name>.

Then run it:

azd ai eval run start

run start waits for the run by default and prints a per-evaluator table with a pass rate and a mean score, plus a link to the run in the portal. Use --no-wait to submit and return.

If the configuration declares more than one eval, name the one you mean:

azd ai eval run start --eval support-trace-eval

Select a dataset version and control run size

A registered dataset run preserves the service-issued identity of the selected dataset version. Version selection uses the declared catalog version, then a recorded publication, then the latest service version. Keeping a local file entry after publication doesn't make the run use local rows instead.

A positive --max-samples value or evaluation-level maxSamples setting isn't supported for registered datasets. The command rejects it before submitting the run, even if the cap exceeds the dataset's row count. The command doesn't silently ignore the cap, replace the registered dataset with inline rows, or publish a temporary subset.

Remove the configured cap to evaluate the whole selected version. For a turn or static dataset evaluation declared in azure.eval.yaml, you can override a configured cap explicitly:

azd ai eval run start --eval turn-quality --max-samples 0

Omitting --max-samples still uses any configured maxSamples. An explicit zero disables that cap for the declared evaluation; it doesn't mean zero rows. Negative values aren't valid.

For simulation, remove maxSamples from the evaluation configuration entirely. --max-samples 0 doesn't repair an invalid simulation configuration that declares a positive cap.

For a smaller registered run, intentionally publish a smaller dataset and declare it in the datasets catalog. Select its catalog name, not a file path or URI:

azd ai eval run start --eval turn-quality --dataset small-golden

This example assumes small-golden is registered, declared, and compatible with the evaluation's column schema. Remove any positive cap before running it.

Inline rows and positive caps are available only for local, unregistered datasets after the command verifies their absence from the service. A failed or inconclusive lookup stops the run instead of falling back to local rows. Publish local edits to an already registered dataset before evaluating them.

For trace-sourced evaluations, use source.maxTraces in the configuration or init --max-traces, not run start --max-samples. Rerunning by a bare evaluation ID reuses its prior source and rejects an explicit --max-samples, including zero; use the declared evaluation name to select its configured dataset.

Inspect the results

The summary tells you whether quality moved. The per-sample rows tell you why:

azd ai eval run output list --eval support-trace-eval
azd ai eval run output list --eval support-trace-eval --failed-only

To see runs over time, and what the service holds for one eval:

azd ai eval list
azd ai eval run list --eval support-trace-eval
azd ai eval show support-trace-eval

show returns the eval's identity in the project (ID, name, when it was created, and who created it) and, when available, its evaluators and data source type. What the eval does is in your evals/azure.eval.yaml.

run list carries one pass rate per run. The per-evaluator breakdown is in -o json, under per_testing_criteria_results, because a column per evaluator stops being readable once runs score different evaluators.

To take the results elsewhere:

azd ai eval run output list --eval support-trace-eval --output-file rows.json
azd ai eval run output export --eval support-trace-eval --output-file run.json

The two differ, and the difference matters: run output list --output-file writes the rows as JSON, with any --failed-only or --status filter applied, and writes every row unless you set --limit. run output export writes the complete run as one JSON document: the run as the service described it, and every evaluated row beneath it with each evaluator's score, verdict, and reason. Derive any other shape from that file, for example jq -c '.items[]' run.json > rows.jsonl.

Gate a build

Pass --fail-on to turn the run into a check. It exits non-zero when the run misses the threshold, which is how a pipeline fails a change that regressed quality:

azd ai eval run start --eval support-trace-eval --fail-on pass-rate=0.8
azd ai eval run start --eval support-trace-eval --fail-on any-failure

Without --fail-on, a completed run with failing samples still exits 0. Failing samples are the expected output of a working evaluation, not a tool error, so gating is opt-in.

pass-rate takes a number between 0 and 1. A threshold that isn't one is refused before the run is submitted, so a mistyped gate costs nothing.

--fail-on needs a run that has finished. On run show, pair it with --wait.

Deploy evaluations with the rest of the project

The init command adds an evaluation service to azure.yaml, so the evaluation is part of the project:

azd up

The azd up command provisions the project, then deploys the evaluation service to reconcile the declared datasets, evaluators, and evaluations. The azd ai eval create command performs evaluation-resource reconciliation independently; it doesn't provision the project's infrastructure. A project that has no infrastructure to provision, such as a workspace that you create with azd init --minimal, has nothing for azd up to provision. In that project, run azd ai eval create to register the evaluation.

Change the agent and re-evaluate

After you change the agent, deploy it:

azd deploy

For trace-sourced evaluations, invoke the updated agent so it emits new traces; run start only reads existing traces. Then run the selected evaluation again:

azd ai eval run start --eval support-trace-eval

When azd ai eval run start invokes an agent, it identifies the agent by name and doesn't pin a version. To compare versions, run the evaluation before you deploy the change and name each run so you can tell them apart:

azd ai eval run start --eval turn-quality --name baseline
azd deploy
azd ai eval run start --eval turn-quality --name updated

To improve a hosted agent with the Agent Optimizer and then verify the result with an independent evaluation, see Quickstart: Optimize a hosted agent.

For controlled comparisons, use a dataset-sourced evaluation, pin the dataset and custom evaluator versions, and keep the grading configuration and thresholds unchanged. Trace-sourced runs can read different interactions, so they don't guarantee an identical set of cases.

After changing evals/azure.eval.yaml or generated artifacts, run create again to reconcile them with the project. Earlier run results remain available, but reusing an evaluation doesn't pin future dataset or evaluator versions.

Best practices

• Start with --source traces if the agent already runs and emits traces. It measures what happened, and there's nothing to author.
• Move to --source dataset once you want a fixed set of cases you can compare across versions.
• Read the generated dataset and rubric before trusting the scores. Check that the generation context represents the behavior you need to test.
• Use more than one evaluator. A single criterion moves the number without telling you why.
• Commit evals/azure.eval.yaml and the generated artifacts, so the evaluation is reviewable.
• Gate with --fail-on in CI, and keep the threshold where a real regression trips it.
• Run a named baseline before you deploy a changed agent, because run start doesn't pin an agent version.

Limitations

• The extension is in preview, and the command surface can change.
• generate submits billed jobs. azd provision doesn't create datasets or evaluators.
• A trace-sourced evaluation can only read traces the agent already emitted.
• azd collapses an extension's exit code, so a gate breach and an operational failure both surface as a non-zero exit. Read the gate message to tell them apart.
• Use synthetic test data, and don't share logs that contain credentials, tokens, or credential-bearing URLs.

Related content

• Evaluate your agents
• Quickstart: Optimize a hosted agent
• Built-in evaluators
• Custom evaluators
• Evaluate deployed interactions
• Set up tracing for your agent
• Monitor agents with the Agent Monitoring Dashboard