Nota
L'accesso a questa pagina richiede l'autorizzazione. È possibile provare ad accedere o modificare le directory.
L'accesso a questa pagina richiede l'autorizzazione. È possibile provare a modificare le directory.
The Microsoft 365 Copilot Agent Evaluations CLI (@microsoft/m365-copilot-eval) helps you test, measure, and improve the quality of your agents with structured evaluations and rich result reports with AI-based scoring.
What you can do
The evaluation tool provides the following capabilities:
- Run batch and interactive evaluations.
- Automatically score responses by using Microsoft Foundry cloud evaluation and built-in metrics.
- Test using JSON datasets, inline prompts, or interactive input.
- Generate reports in HTML, JSON, or CSV formats.
Evaluation metrics
Each response is scored using standard evaluation metrics.
| Evaluator | Type | Scale | Default Threshold | Default |
|---|---|---|---|---|
| Relevance | LLM-based | 1-5 | 3 | Yes |
| Coherence | LLM-based | 1-5 | 3 | Yes |
| Groundedness | LLM-based | 1-5 | 3 | No |
| Similarity | LLM-based | 1-5 | 3 | No |
| RetrievalQuery | Non-LLM | Pass/fail | N/A | No |
| RetrievalResult | Non-LLM | Proportional | 1.0 | No |
| Citations | Count-based | >= 0 | 1 | No |
| ExactMatch | String match | boolean | N/A | No |
| PartialMatch | String match | 0.0-1.0 | 0.5 | No |
For detailed information about each evaluator, including configuration options and sample datasets, see Evaluators reference.
How the evaluation workflow works
Evaluations follow a consistent workflow:
- Install and configure the CLI.
- Provide environment configuration and credentials.
- Create a dataset of test prompts.
- Run evaluations against your agent.
- Review results and iterate.
Required environment variables
The evaluation tool uses environment variables to authenticate and connect to your tenant and Microsoft Foundry project.
| Variable | Description | Default |
|---|---|---|
TENANT_ID |
Microsoft Entra tenant ID where your agent is deployed. | None |
AZURE_AI_PROJECT_ENDPOINT |
Microsoft Foundry project endpoint used for cloud evaluation. | None |
AZURE_AI_MODEL_NAME |
Model deployment name in your Microsoft Foundry project. | gpt-5-mini |
M365_TITLE_ID (optional) |
Title ID used to auto-detect the Microsoft 365 agent ID for evaluation. | None |
M365_AGENT_ID (optional) |
Explicit agent ID for evaluation. | Auto-detected from M365_TITLE_ID |
These values enable authentication and allow the tool to run LLM-based evaluation scoring by using Microsoft Foundry. For details about how to get these values, see Get values for environment variables.