Assess model changes with Copilot Agent Kit (preview)

[This article is prerelease documentation and is subject to change.]

Model assessment in Copilot Agent Kit helps you compare how an agent performs before and after a model change. Run the same tests against both model versions, review improvements and regressions, and use the results to decide which model to keep.

Model assessment supports two test engines:

Test engine Tests that run
Copilot Studio evaluations Active evaluation test sets with MODELCHECK in the name
Copilot Agent Kit tests A Kit test set run through a pipeline-enabled Agent Configuration against the published agent

You can't compare results from different test engines because the engines score different things.

Prerequisites

Before you use the model assessment feature in the Copilot Agent Kit, ensure that you:

  • Installed Copilot Agent Kit and shared it with makers.
  • Assigned the CSK - Maker or CSK - Administrator security role to each user who runs assessments.
  • Connected the Copilot Studio Kit - Dataverse connection reference.
  • Added a Microsoft Copilot Studio connection to the Copilot Agent Kit - Automated Tests connection reference. The connection's user must have access to the agents you want to assess.
  • Turned on the following model assessment cloud flows:
    • Model Assessment | Run Copilot Studio Evaluations
    • Model Assessment | Run Copilot Agent Kit Tests
  • Added at least one supported test set to each agent. Learn more in Make an agent eligible.

Assessment runs use capacity for agent responses and AI summaries. Learn more in Licensing and Copilot Credits.

Note

Copilot Agent Kit tests run against the published agent. Copilot Studio evaluations don't require the agent to be published.

Set up model assessment

The following sections explain how to turn on the cloud flows, set up the connection references, and make an agent eligible for assessment.

Turn on the cloud flows

To turn on the cloud flows:

  1. Sign in to Power Apps and select the environment where you installed Copilot Agent Kit.
  2. On the side bar, select Solutions, and then select Copilot Agent Kit from the list of solutions. The details page opens.
  3. At the end of the Recent Items list, select See all to view all objects.
  4. In the Objects pane, select Cloud flows.
  5. Search for the following cloud flows:
    • Model Assessment | Run Copilot Agent Kit Tests
    • Model Assessment | Run Copilot Studio Evaluations
  6. Select the three dots (⋮) next to each cloud flow and select Turn on.

Set up the connection references

To set up the connection references:

  1. Sign in to Power Apps and select the environment where you installed Copilot Agent Kit.
  2. On the side bar, select Solutions, and then select Copilot Agent Kit from the list of solutions. The details page opens.
  3. At the end of the Recent Items list, select See all to view all objects.
  4. In the Objects pane, select Connections references.
  5. Search for Copilot Studio Kit - Dataverse and ensure it has a Dataverse connection.
  6. Search for Copilot Agent Kit - Automated Tests and ensure it has a Microsoft Copilot Studio connection. The connection's user must have access to the agents you want to assess.

Make an agent eligible

An agent appears in model assessment when it has at least one of the following test configurations.

Option A: Copilot Studio evaluations

  1. In Copilot Studio, open the agent and select Evaluate.
  2. Create an evaluation test set with MODELCHECK in its name, such as MODELCHECK - HR policies.
  3. Add test cases and graders, and keep the test set active. Only active MODELCHECK test sets run.

Learn more in Create a test set for an agent.

Option B: Copilot Agent Kit tests

  1. In Copilot Agent Kit, create a new Agent Configuration record for the agent and select Configuration Type(s) as Test Automation.
  2. Select the configuration record's General tab and turn on Pipeline Enabled in the Scheduled Testing section.
  3. Create a test set with tests for the agent.

Note

Kit test automation supports agents powered by the standard harness. You can assess agents powered by the Microsoft Copilot harness with Copilot Studio evaluations.

Assess an agent

The following sections explain how to open Model Assessment, run a baseline assessment, and compare results after changing the agent's model.

Select an agent for model assessment

To select an agent for model assessment:

  1. Open the Copilot Agent Kit app.
  2. On the side bar, select Advanced Testing in the Productivity section. The Advanced Testing page opens.
  3. Select Model assessment. A list of agents opens.
  4. Select an eligible agent from the list.

The list shows each agent's previous and current model, regression status, last run, and latest pass, fail, and error counts. Use the filters to find agents that need attention or search by agent name. Select Refresh agents to reload the list.

Screenshot of the Model Assessment agents list showing models, regression status, latest results, and filters.

Run a baseline assessment

The first run records a baseline for the agent's current model. To run a baseline assessment:

  1. Select an agent from the Model Assessment agents list to open its details panel.
  2. If no baseline exists, select Run.
  3. Wait for the assessment to complete. You can close the details while the run continues in the background.

Only one assessment can run for an agent at a time. Select Cancel assessment to stop a run; a canceled run isn't scored.

Change the model and run the assessment again

After you capture a baseline, change the agent's model and run another assessment.

  1. In the agent details panel, select a model from the Model picker, or change the model in Copilot Studio.
  2. If you changed the model in Copilot Studio, return to Model Assessment in Copilot Agent Kit and confirm that the app detected the change.
  3. Select Run.

When you select a model in Model Assessment, the app saves it to the agent. For Kit tests, the app also publishes the agent because tests run against the published version. If the model hasn't changed since the last run, change it before you select Run again.

Model history shows the baseline and later model changes. The app records a model change only after an assessment completes without errors. If a run has errors, address them and run the assessment again.

Screenshot of the agent details showing the model picker and model history for a Copilot Agent Kit test configuration.

Choose the test engine

If an agent has both Copilot Studio evaluations and Copilot Agent Kit tests, choose which test engine to run. For Kit tests, select an Agent Configuration and test set. If there's only one configuration, the app selects it for you.

Screenshot of the agent details showing the Copilot Agent Kit or Copilot Studio evaluation selector.

Compare runs and review results

To compare runs and review results:

  1. In the agent's model history, select a completed run to open Compare model runs panel.

  2. Select up to three completed runs. The app displays them from oldest to newest.

  3. Review the Summary tab for each model, run status, regression status, test results, Copilot Credits where available, and AI analysis.

  4. Review the Test runs tab for each test case and its result in each run. Select Needs attention to filter the list to failed or error results.

  5. Select Export to PDF to download the comparison.

    Compare model runs dialog showing a model recommendation.

Use the model runs picker to change which completed runs appear in the comparison:

Screenshot of the model runs picker for selecting completed model assessment runs to compare.

The summary includes a card for each run, followed by AI summaries and the model recommendation:

Screenshot of the comparison summary showing model run cards and AI summaries.

The app compares runs only when they use the same test engine. If a previous run used a different engine, then select a run of the same kind to compare.

Screenshot of comparison notice explaining that runs from different test engines can't be compared.

The Model recommendation card appears when you select exactly two runs that use the same test engine. It includes a verdict, confidence level, reasons, trade-offs, and caveats. Use the recommendation as guidance, not a guarantee. Review the individual test results before you decide which model to keep.

In the following screenshot, the recommendation indicates that neither model is recommended:

Screenshot of model recommendation indicating that neither model is recommended.

In the following screenshot, the recommendation indicates that there isn't a clear winner:

Screenshot of model recommendation indicating there is no clear winner.

For Kit tests, the recommendation shows low confidence when the comparison has few shared checks:

Screenshot of comparison of Copilot Agent Kit test runs with a low-confidence model recommendation.

The Test runs tab shows each evaluation test case and its result in the selected runs:

Screenshot of test runs comparison showing evaluation test cases and results across model runs.

Select Needs attention to filter the list to failed or error results:

Screenshot of test runs comparison filtered to results that need attention.

For Kit test runs, the tab shows each Kit test and its result in each selected run:

Screenshot of Copilot Agent Kit test runs comparison showing test results across model runs.

Understand regression status

Regression status summarizes the current run's scored results.

Regression status Meaning
Healthy No scored checks failed
Warning Up to 20 percent of scored checks failed after rounding down
Critical More than 20 percent of scored checks failed after rounding down
Not Assessed No checks were scored

A check is one result. In Copilot Studio evaluations, each metric for a test case is a check. In Kit tests, each test is a check. Only checks with a Pass or Fail result are scored; errors, excluded checks, and pending checks aren't included in the percentages. Percentages are rounded down to whole numbers before the status is assigned.

A regression is a check that passed in the baseline and fails in the current run. A check that failed in both runs is a persistent failure, not a regression. The status considers both the failed-check rate in the current run and the regression rate compared with the baseline. Because regressions are a subset of failures, the regression rate can't exceed the failed-check rate. Review the regression count to see what changed from the baseline.

Understand Copilot Credit reporting

Note

Each Model Assessment run costs approximately 150 Copilot Credits.

  • For Copilot Studio evaluations, Model Assessment records Copilot Credits when Copilot Studio reports them for an agent on the Microsoft Copilot harness. The usage can include credits for agent responses and evaluation.
  • Copilot Agent Kit tests don't report Copilot Credits in Model Assessment.
  • AI summaries and recommendations can use capacity even when a run doesn't report credit totals.

Don't use credit totals to compare runs from different test engines.

Troubleshooting

Issue What to check
An agent isn't listed Confirm that the agent has an active MODELCHECK evaluation or a pipeline-enabled Agent Configuration with a test set. Then select Refresh agents.
An assessment can't start because a flow is off Turn on the corresponding Model Assessment cloud flow in the Copilot Agent Kit solution.
A run reports a connector error Check that both connection references are connected and that the Copilot Studio connection's user can access the agent.
No evaluation test set runs Confirm that the test set name contains MODELCHECK and that the test set is active.
Kit tests fail or time out Confirm that the Agent Configuration has Pipeline Enabled turned on, that the selected test set has tests, and that the agent is published.
The model change isn't recorded Resolve errors in the run and run the assessment again. Model history is updated only after a run completes without errors.
The app reports that the model hasn't changed Change the model in Model Assessment or Copilot Studio, then run the assessment.