Evaluate an agent (preview)

[This article is prerelease documentation and is subject to change.]

The Evaluate tab in the new agent experience provides structured, repeatable testing for your agent. Use evaluations to measure agent quality across test cases, track improvements over time, and validate your agent's behavior before you publish it.

Important

What is agent evaluation?

Agent evaluation lets you systematically test your agent's responses against quality standards. Instead of testing each scenario manually in the Preview tab, create evaluations with test conversations, define how to measure responses, and run the evaluations to get quantitative results.

Evaluation helps you answer questions like these:

  • Does the agent answer correctly across a range of expected scenarios?
  • Did a configuration change improve or degrade response quality?
  • Does the agent call tools when it should?

Key concepts

The following concepts are key when running evaluations in Copilot Studio:

Conversations

A conversation is a test case that represents a scenario you want your agent to handle. Each conversation includes user messages and optionally expected agent responses. Organize conversations into evaluations. Create conversations manually, generate them with AI, or upload them from a CSV file.

Evaluations

An evaluation is a named test set that combines conversations with one or more test methods. You create evaluations from the Evaluate tab, add conversations to them, and then run them to produce scored results.

Test methods

Test methods define how the agent's responses are scored. Each new evaluation starts with General quality, and you can add up to five test methods to an evaluation. The test methods available to you can vary by environment.

Test method What it checks
General quality Whether responses meet quality standards, such as relevance and completeness. It doesn't compare responses to expected responses.
Content safety Whether responses contain harmful content of the selected content type: Violent, Hateful, Self-harm, or Sexual.
Compare meaning Whether answers have the same meaning as the expected response, even if they're worded differently.
Capability use Whether the agent used specific tools or skills to generate an answer.
Keyword match Whether answers contain matching words and phrases.
Text similarity How close the text of an agent's answer is to the expected response. This method might miss differences in meaning.
Exact match Whether agent answers match the expected response completely.
Classification Checks and labels agent answers according to instructions you define.

For Compare meaning, Text similarity, and Exact match, include an expected agent response in each test conversation so the method can compare the agent's answer to it.

Content safety checks one content type at a time, so add a separate Content safety test method for each content type you want to check. For each one, set the Maximum severity from level 0 to level 7. Level 0 is the strictest, and level 7 allows graphic or extreme content to pass. The default is level 3. A response fails when its severity for the content type is above the maximum severity you set.

Content safety evaluations help identify potentially harmful content, but they don't guarantee that an agent is safe or appropriate for every scenario.

User profile

Evaluations and connections used by tools run with the currently signed-in user profile. You can't connect with a different user while you're signed in. To run an evaluation as a different user, sign out, sign in with that user's account, and then start the evaluation.

Credit usage

Evaluate surfaces estimated Copilot Credit usage throughout the testing process, so you can understand the cost of testing quality before you run larger test suites. You can see which experiences use credits, an estimated total for each evaluation run, a breakdown between generating and scoring responses, and credit usage at the individual test-case level. Learn more in Understand estimated credit usage.

Evaluation workflow

Follow these steps to evaluate an agent:

  1. Create an evaluation: On the Evaluate tab, start a new evaluation. Learn more in Create a test set for an agent.
  2. Add conversations: Add test conversations by writing them manually, generating them with AI, or uploading a CSV file.
  3. Configure: Name the evaluation, select a test method, and choose the agent version.
  4. Run the evaluation: Select Evaluate to run the test and wait for the results. Learn more in Run an evaluation for an agent.
  5. Review results: Analyze scores and identify areas for improvement. Learn more in View evaluation results for an agent.
  6. Iterate: Adjust your agent's instructions, knowledge, or tools, and then run the evaluation again to measure the impact.

Limitations

Evaluations only support agents set to use No authentication or Authenticate with Microsoft. Agents set to Authenticate manually aren't supported.