Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
The following sections apply to every model change, whether it's a proactive upgrade or a response to a retirement. Define the gates first, capture a baseline from the current model, and then work through the phases.
Define migration acceptance gates
Define acceptance criteria before evaluating the candidate model, so that the agent evaluation produces a decision instead of only a set of scores. Include:
- Minimum overall pass rate
- Required pass rate for business-critical scenarios
- Critical failures that block migration regardless of aggregate score
- Allowed latency and reliability variance
- Safety, compliance, and regional-processing approval
- Acceptable consumption or cost impact
- Required owner and release approver signoff
Compare the current and candidate model by using the same agent configuration, test data, test set, user profiles, and environment assumptions. Investigate individual regressions instead of relying only on an average score.
Model outputs are probabilistic. Run important scenarios more than once when variability can affect the decision.
Establish a reusable evaluation baseline
Before you change the model, create a test set that represents the agent's business-critical and high-frequency scenarios. Run the test set against the current production model to establish a baseline.
Use both test chat and agent evaluation:
- Use test chat to explore complete conversations and inspect orchestration with the activity map.
- Use agent evaluation to run repeatable test sets, measure results, and compare runs over time.
The test methods available to you depend on the agent's harness, so confirm them before designing the test set. Learn more in Confirm which test methods your agent supports.
Choose how to compare models
After you define what to test and establish acceptance criteria, choose how to compare the current model with candidate models. Use the same test set and equivalent agent configurations so that the comparison reveals model-related differences rather than unrelated changes. You can perform the comparison manually by using agent copies, or use the Copilot Agent Kit's model assessment functionality to coordinate testing and review the results.
Compare manually by using agent copies
Create nonproduction copies of the agent to evaluate models without affecting production users.
- Prepare equivalent agent copies. Keep instructions, topics, knowledge sources, tools, connections, security settings, and other configuration consistent. Record any differences that could affect the comparison.
- Select the models. Keep the current model on one copy as the baseline and select a candidate model on each additional copy.
- Run the same evaluation test set against each copy. Use the same test cases, evaluation methods, and representative test identities. Repeat business-critical scenarios when variability could affect the decision.
- Compare the results against your acceptance criteria. Review individual improvements, regressions, and persistent failures rather than relying only on aggregate pass rates. Use test chat and the activity map to investigate changes in orchestration, tool selection, and instruction following.
- Measure latency and consumption separately. Record these measures for representative scenarios on each model, and export the evaluation results to preserve the comparison.
Keep the test set unchanged for the initial comparison. If you then refine instructions or other agent artifacts to address a regression, rerun the tests and record those changes alongside the results.
Compare by using Model Assessment (preview)
Model assessment in the Copilot Agent Kit captures a baseline, reruns tests after a model change, and highlights improvements, regressions, and persistent failures. Instead of managing separate agent copies for each model, assess successive models on a nonproduction copy and compare the recorded runs.
Model Assessment supports two testing options:
| Testing option | When to use it | Requirements |
|---|---|---|
| Copilot Studio evaluations | Compare results from evaluation test sets created in Copilot Studio. Available for Microsoft Copilot harness agents and standard harness agents, using the evaluation methods each harness supports. | Create an active evaluation test set with MODELCHECK in its name. The agent doesn't need to be published. |
| Copilot Agent Kit tests | Compare results from the Kit's test automation against a published agent. Available for standard harness agents only. | Configure a pipeline-enabled Agent Configuration and a Kit test set. |
To compare models:
- Capture the baseline. Run an assessment against the current model before changing it.
- Select the candidate model. Change the model through Model Assessment or in Copilot Studio.
- Rerun the same tests. Keep the test engine, test set, and other agent settings unchanged for the initial comparison. Results from the two test engines aren't directly comparable because they score different things.
- Review the comparison. Inspect per-test results, regression status, and AI-generated summaries. You can compare up to three completed runs side by side and export the comparison to PDF.
- Apply your acceptance criteria. When you select two runs from the same test engine, Model Assessment provides a model recommendation and confidence level. Treat the recommendation as supporting evidence, not a replacement for your migration acceptance gates. Review Copilot Credit consumption where reported, and measure latency separately.
Important
Assess candidate models on a nonproduction copy of the agent. Selecting a model in Model Assessment changes the agent's configuration and, for Kit tests, also publishes the agent.
Learn more about setup requirements, assessment steps, and guidance on interpreting results in Assess model changes with Copilot Agent Kit.
Evaluate the full agent behavior
Don't approve a replacement model just because its aggregate pass rate is similar to the current model.
Cover the following areas. Each one is a behavior that commonly changes when the model changes, so a test set that omits an area can't detect a regression in it.
| Evaluation area | What a model change can break | How to check it |
|---|---|---|
| Answer quality | Relevance, completeness, accuracy, clarity, and consistency on the questions the agent receives most often. | General quality |
| Groundedness and knowledge | The model answers from training data instead of the configured knowledge source, drops citations, or handles incomplete or conflicting sources differently. | General quality |
| Abstention | The model answers an out-of-scope question instead of declining or escalating. | General quality, or Custom with Answered and Declined labels |
| Instruction following | Instructions that the model reliably followed are skipped, such as escalation rules, boundaries, prohibited behaviors, or a mandatory disclaimer. | Keyword match on required phrases, or Custom with instruction-specific labels |
| Fixed values and output format | A precise value such as a phone number, code, or ID is paraphrased or invented, or the output shape changes and breaks a downstream parser or channel integration. | Exact match, or Keyword match on the required format |
| Tool selection and restraint | The model selects a different tool, calls none, or calls a tool when no tool is needed. | Tool use with the expected tools or topics defined, plus a review of the activity map |
| Tool inputs and sequencing | Parameters are extracted, formatted, or defaulted differently, or steps in a multitool task are reordered, merged, or dropped. | Tool use with all expected tools, plus General quality |
| Confirmation and failure handling | The model stops asking for confirmation before a consequential action, or a tool error is surfaced differently instead of reported. | Custom with confirmation labels, plus Keyword match on the expected error language |
| Multiturn behavior | Context from earlier turns is lost or reinterpreted, or clarification, topic changes, and recovery are handled differently. | General quality on a conversation test set |
| Messy and adversarial input | Weaker handling of typos, fragments, and unclear intent, or a different response to prompt injection and role override attempts. | General quality, plus Custom with Refused and Complied labels |
| Safety and compliance | Harmful-content handling, data access, permissions, regional processing, and responsible AI requirements. | Custom labels, alongside a responsible AI review |
| Latency and reliability | Response time, timeouts, variability, failed calls, and retries under representative conditions. | Not reported by evaluation. Measure in test chat and production monitoring. |
| Consumption and cost | Copilot Credit consumption or other model-related costs for representative scenarios. | Not reported by evaluation. Measure in capacity and consumption reporting. |
| Languages and channels | Quality and behavior across supported languages, user profiles, and deployment channels. | Run the test set for each language, user profile, and channel that matters |
Pay particular attention to instruction following and latency. A candidate model can improve answer quality but introduce slower responses, different tool-selection behavior, or failures in instructions that were reliable with the previous model.
Confirm which test methods your agent supports
The test methods available to your agent depend on its harness.
Learn more in:
- Evaluate agents powered by the standard harness
- Evaluate agents powered by the Microsoft Copilot harness
Build and maintain the test set
- Cover business-critical happy paths and high-volume requests first, then difficult cases: edge cases, adversarial inputs, requests that should be refused or escalated, tool failures, long conversations, and multilingual requests.
- Start from real traffic. Analytics themes and recorded conversations produce more representative test cases than invented ones.
- Favor coverage over polish. A larger set of imperfect cases finds more regressions than a small set of perfectly worded ones.
- Score the current model first, then the candidate. The comparison, not the absolute number, tells you whether the migration is safe.
- Keep the set with the agent in source control, and rerun it unchanged for every model change.
- Split the set by risk area. A test set powered by the standard harness holds up to 100 test cases, so use separate sets for knowledge accuracy, tool behavior, and safety-sensitive scenarios.
- Export results. Evaluation results are retained for 89 days, so export them to CSV to keep a record of what each model scored at migration time.
- Convert production incidents and user feedback into new regression tests.
Learn more in Design and operationalize agent evaluation.
Note
Agent evaluation measures correctness and performance, not AI ethics or safety problems. An agent can pass every test case and still produce an inappropriate answer. Use responsible AI reviews and content safety filters alongside evaluation.
Automate recurring evaluations
Copilot Studio supports running evaluations through the Power Platform API, so you can integrate model validation into release workflows and continuous integration pipelines. For a large agent estate, automation is what makes repeated candidate testing sustainable instead of manual. Learn more in Automate evaluations with the Power Platform API.
Update the agent artifacts a model change affects
A model change rarely affects only the model setting. Translate the applicable provider guidance into the agent artifacts you control, and then validate every change against your evaluation baseline. A remediation that isn't retested is a guess.
| Artifact | Typical change |
|---|---|
| Agent instructions | Make implicit expectations explicit, remove workarounds written for the previous model, restate boundaries and escalation rules and prohibited behaviors unambiguously, resolve conflicting instructions, and calibrate how much independent action the agent should take. |
| Topic and node instructions | Apply the same treatment at the topic level, and verify that generative answer nodes still behave as expected. |
| Tool and action descriptions | Rewrite for clarity. This description is what the model reads to decide whether and when to call a tool, and vague descriptions cause both over-calling and under-calling. |
| Input and output parameter descriptions | Tighten the format, units, examples, and required or optional semantics so that parameter generation stays correct. |
| Knowledge source configuration | Recheck source selection, scoping, and grounding instructions, and verify citation behavior. |
| Response formatting instructions | Restate the required structure, length, disclaimers, and exact values explicitly, because newer models change default verbosity. |
| Confirmation and safety gates | Reassert explicit confirmation requirements before consequential actions. |
| Downstream consumers | Update Power Automate flows, adaptive cards, channel integrations, and any parser that reads agent output. |
| The test set itself | Add the new failure patterns discovered during the migration. |
Migration phases
The following phases put the preceding sections in execution order. Use them as the working plan for a single agent's model change, whether the change is proactive or driven by a retirement.
Phase 0: Prepare
- Confirm the candidate model is valid against the prerequisites: generally available or default, available in-region, cross-geo posture acceptable, administrator-enabled, and the right use category for the agent's purpose.
- Read the model provider's upgrade guidance and note the instruction and tool changes it implies.
- Identify the affected agents from the inventory, recording environment, owners, and criticality.
- Confirm which evaluation test methods the agent's harness supports.
- Define and approve the migration acceptance gates.
Phase 1: Baseline the current model
- Build or refresh the regression test set so it covers business-critical and high-frequency scenarios.
- Run the test set against the current production model to establish the baseline. Do this while the current model is still in place, because after the model changes the baseline can't be reconstructed.
- Record latency and Copilot Credit consumption separately. Evaluations don't report them.
- Export the results to CSV to preserve the record beyond the retention window.
Phase 2: Evaluate the candidate in a nonproduction environment
Prepare a nonproduction copy of the agent, following the Copilot Studio guidance for application lifecycle management and agent testing. Configure the environment so it represents production:
- Use the same agent instructions, topics, knowledge configuration, tools, flows, connectors, languages, and security assumptions.
- Use representative test identities and connections.
- Apply the same data policies and relevant administrator controls.
- Confirm regional availability and whether cross-geo data movement is required.
- Record any differences between test and production that could affect the result.
Change the model. Go to the agent's Overview page and select the candidate primary model in the Model section. Separate settings exist for deep reasoning, generative responses, and the prompt builder, so check whether the agent uses those capabilities and whether they need to change too.
Rerun the same test set, unchanged, against the candidate model.
Compare the two runs against the acceptance gates to identify improvements and regressions.
Inspect orchestration qualitatively in test chat, using the activity map to confirm which tools were selected, in what order, and with which parameters.
Measure latency and consumption for the candidate and compare them against the baseline.
Phase 3: Remediate
- Update the agent artifacts the model change affects, then rerun the test set against the remediated agent. Learn more in Improve agents using evaluation-driven triage and remediation.
- Iterate until the acceptance gates are met, or conclude that the candidate model isn't suitable and document why. Deciding not to upgrade is a legitimate, evidence-based outcome.
If your migration also affects a Microsoft 365 Copilot plugin, follow the guidance in Monitor, update, and retire your plugin or agent to test, package, and roll out the update.
Phase 4: Approve and deploy
- Get signoff on the acceptance gates from the agent owner and release approver, including safety and compliance approval when cross-geo or external models are involved.
- Deploy through the established ALM process, promoting the solution from test to production. Don't manually edit the production agent.
- Stage the rollout when the channel allows it. Publish to a pilot audience or a single channel first, observe the result, then broaden.
Phase 5: Monitor and close
- Monitor production behavior against the acceptance gates.
- Add newly discovered failure patterns to the regression test set.
- Close the migration only after the acceptance criteria remain satisfied in production.
- Preserve the evaluation evidence and the migration decision record for audit and for the next lifecycle event.
Important
The rollback path for a model change is redeploying the previously validated solution version, which is slower than a configuration toggle. This difference makes the evaluation gate in phase 2 important. Catching a regression before deployment is less expensive than reversing one afterward.
Monitor after migration
Model lifecycle work continues after deployment. Monitor:
- Agent evaluation results and critical scenario pass rates.
- Production analytics, transcripts, activity, errors, and user feedback.
- Instruction-following and tool-selection failures.
- Latency, timeouts, and reliability.
- Consumption changes.
- Safety, compliance, and regional-processing concerns.
When production monitoring identifies a new failure pattern, add a representative case to the regression test set. This practice improves the next model evaluation and turns production learning into a durable quality baseline.
Environment-level telemetry emits OpenTelemetry GenAI spans to Application Insights, including the model used for each agent invocation. Use it to confirm which model production traffic actually runs on, and to compare tool selection and reliability before and after a migration.