หมายเหตุ
การเข้าถึงหน้านี้ต้องได้รับการอนุญาต คุณสามารถลอง ลงชื่อเข้าใช้หรือเปลี่ยนไดเรกทอรีได้
การเข้าถึงหน้านี้ต้องได้รับการอนุญาต คุณสามารถลองเปลี่ยนไดเรกทอรีได้
Tool-call evaluators inspect runtime diagnostics (Developer Info) to verify tool selection and HTTP execution outcomes. They are deterministic, don't use an LLM judge, and don't judge response text or require expected_response.
Note
Tool-call evaluators require Microsoft 365 Copilot Evaluations CLI v2 (preview) and schema v2. They aren't available in CLI v1 (latest).
| Evaluator | Checks | Scale | Threshold | Enabled by default |
|---|---|---|---|---|
| ToolCallSelection | Whether the expected tool identities were selected | Pass/fail/error | 1.0, fixed | No |
| ToolCallExecution | Whether each expected call has a distinct successful HTTP execution | Pass/fail/error | 1.0, fixed | No |
For general dataset setup, see Configure evaluators.
Set up the preview CLI
Follow the tenant access and runtime prerequisites in the quickstart. Microsoft Foundry scoring configuration isn't required if you run only deterministic evaluators.
Install the preview release and accept the license terms:
npm install -g @microsoft/m365-copilot-eval@preview
runevals accept-eula
Set TENANT_ID for your Microsoft 365 tenant in your evaluation environment. For a deployed declarative agent, pass --m365-agent-id "<your-agent-id>" when running a dataset.
For standalone plugin evaluations, provision and enable the plugin in Microsoft 365 Copilot first, and complete any required connector sign-in in Copilot. With no configured agent ID, v2 preview evaluates Copilot chat. Keep M365_AGENT_ID and legacy AGENT_ID out of your shell and loaded environment files for these runs. This evaluates Copilot responses that use an available plugin, not the plugin endpoint in isolation.
Installing either npm release tag replaces the global runevals command. To return to v1, reinstall @microsoft/m365-copilot-eval@latest.
Allow tool calls
When a tool requires confirmation, configure both approval and evaluator expectations. default_allowed_tools at document level and allowed_tools per prompt or turn let the CLI approve tool-confirmation prompts. Every requested consent identifier must be allowed; otherwise, the prompt stops with a toolCallNotAllowed error.
Important
Allowing a call grants consent. Review the operations you're approving and start with read-only scenarios. An expectation doesn't grant consent; approval doesn't force selection or prove execution. Tools that don't request confirmation aren't controlled by this list, so it isn't a security boundary.
Consent identifiers are different from calls[].tool identities. Don't split consent identifiers to populate evaluator identity fields.
Find tool-call names
For declarative agents, use runevals --list-tools to list consent identifiers and copy-ready evaluator identities. Microsoft 365 Agents Toolkit can also inspect the acquired agent's launch information. For standalone plugins, --list-tools isn't supported; construct consent identifiers from the app manifest instead.
Declarative agents
# List consent identifiers and evaluator identities for declarative agent actions.
npx -y --package @microsoft/m365-copilot-eval@preview runevals --list-tools "<your-agent-id>"
# Alternatively, inspect launch information with Agents Toolkit.
# Use the package Title ID without the .declarativeAgent suffix.
atk launchinfo --title-id "<your-title-id>"
Dynamically discovered tools might not be listed. After a tool-call evaluation, confirm their identities in the JSON report's diagnostics.tool_calls.
Agents Toolkit requires access to the app and a separate Microsoft 365 sign-in (atk auth login m365). Its account can differ from the account used for evaluation.
Standalone plugins
Construct each consent identifier from your plugin's app manifest (agentConnectors[]) as follows:
<connector display name with only A-Z, a-z, and 0-9 retained>_<function name>
Preserve casing. Remove spaces, hyphens, underscores, punctuation, and non-ASCII characters from the connector display name only. Keep the function name unchanged.
| Connector display name | Function name | allowed_tools entry |
|---|---|---|
Contoso Catalog |
get_product |
ContosoCatalog_get_product |
Contoso Catalog-Tools_1! |
get_product |
ContosoCatalogTools1_get_product |
Alternatively, run an evaluation once: if a tool requires consent and isn't allowed, the result message includes its tool name.
Tool-call identity
All three identity fields are required inside calls[].tool:
| Field | Type | Description |
|---|---|---|
plugin_id |
string | Copy M365_TITLE_ID from your deployed agent or plugin project's environment file, for example env\.env.local. This is the package Title ID (plugin.titleId in Developer Info), not plugin.id. Required even when the CLI already targets that declarative agent. |
toolset_id |
string | For a declarative agent action, use its actions[].id in the declarative agent manifest. For a standalone plugin, use its agentConnectors[].id in the app manifest. |
function_name |
string | Function name defined by your plugin, for example get_product. Preserve casing; this is reported as functionDisplayName in Developer Info. |
The tuple (plugin_id, toolset_id, function_name) uniquely identifies a tool. Matching is case-sensitive. Use action or connector IDs, not display names. For example:
{
"plugin_id": "T_1234",
"toolset_id": "catalog",
"function_name": "get_product"
}
Evaluator configuration
Configure these options under ToolCallSelection or ToolCallExecution in default_evaluators, or in an item's or turn's evaluators object:
| Option | Type | Required | Description |
|---|---|---|---|
calls |
array | Yes | Nonempty list of expected selections or executions. |
calls[].id |
string | No | Nonempty, evaluator-local diagnostic label; not a runtime call ID. Use a unique label for each expectation. |
calls[].tool |
object | Yes | All three fields from Tool-call identity. |
allow_additional_calls |
boolean | No | ToolCallSelection only. Defaults to true. Set to false to reject any extra selected identity, including unidentified selections. |
ToolCallSelection
ToolCallSelection checks the set of expected tool identities, ignoring order and duplicates. It doesn't verify execution success or call counts.
ToolCallSelection sample dataset
Save this dataset as evals\tool-selection.json. Replace the sample identities and consent identifier with your own:
{
"schemaVersion": "2.0.0",
"default_allowed_tools": ["ContosoCatalog_get_product"],
"default_evaluators": {
"ToolCallSelection": {
"calls": [
{
"tool": {
"plugin_id": "T_1234",
"toolset_id": "catalog",
"function_name": "get_product"
}
}
],
"allow_additional_calls": false
}
},
"items": [
{
"prompt": "Find product SKU-123 in the Contoso Catalog."
}
]
}
Run against a deployed declarative agent:
runevals --m365-agent-id "<your-agent-id>" --prompts-file evals\tool-selection.json --output results\tool-selection.json
For a standalone plugin scenario, omit --m365-agent-id and ensure no agent ID is configured, as described in Set up the preview CLI.
The following outcomes assume plugin_id and toolset_id match the example.
| Observed selections | Result in this sample |
|---|---|
get_product, including duplicate entries |
Pass |
get_product and another function |
Fail: additional selection |
No selected functions ([]) |
Fail: required selection missing |
Unavailable telemetry (null) |
Error |
ToolCallExecution
ToolCallExecution requires a distinct matching execution with an integer HTTP status 200-299 for each entry in calls. Repeat an identity to require multiple successful executions.
Additional attempts, including failed retries, are allowed. HTTP success alone doesn't establish business correctness or answer quality.
ToolCallExecution sample dataset
Save this dataset as evals\tool-execution.json and replace the sample identities and consent identifier with your own:
{
"schemaVersion": "2.0.0",
"default_evaluators": {
"ToolCallExecution": {
"calls": [
{
"id": "first-lookup",
"tool": {
"plugin_id": "T_1234",
"toolset_id": "catalog",
"function_name": "get_product"
}
},
{
"id": "second-lookup",
"tool": {
"plugin_id": "T_1234",
"toolset_id": "catalog",
"function_name": "get_product"
}
}
]
}
},
"items": [
{
"prompt": "Look up products SKU-123 and SKU-456 individually in the Contoso Catalog.",
"allowed_tools": ["ContosoCatalog_get_product"]
}
]
}
Run the dataset with the same target configuration as the selection example:
runevals --m365-agent-id "<your-agent-id>" --prompts-file evals\tool-execution.json --output results\tool-execution.json
| Observed matching HTTP outcomes | Result in this sample |
|---|---|
200, 200 |
Pass |
500, 200, 200 |
Pass: two successes despite a failed attempt |
200 |
Fail: only one success |
200, unknown |
Error: the unknown outcome could satisfy the second call |
This sample checks two successful calls, not which product IDs were used. Each expectation requires a distinct execution; one execution can't satisfy both.
Interpret results
Use these rules to distinguish failures from evaluator errors:
| Observation | Result |
|---|---|
| Every expected selection is present, and rules for additional selections are met | Selection passes |
| A required selection is absent from an authoritative list | Selection fails |
| Enough distinct HTTP 2xx executions match every expected identity | Execution passes |
| An expected execution identity is missing, or confirmed outcomes make the required success count impossible | Execution fails |
| An observed call has an incomplete identity | Not credited to any expected call |
| Required telemetry is unavailable, multiple Developer Info artifacts prevent an authoritative snapshot, or unparseable HTTP outcomes could change the verdict | Evaluator error: toolCallEvidenceIndeterminate |
A definite failure takes precedence if unknown evidence can't complete all expectations. Unreadable extra attempts don't invalidate enough confirmed successes.
Pass = 1.0; fail = 0.0. Errors omit score and threshold; they aren't passes or necessarily agent defects. JSON score keys are toolCallSelection and toolCallExecution:
{
"scores": {
"toolCallSelection": {
"score": 1.0,
"result": "pass",
"threshold": 1.0,
"reason": "All expected tool functions were selected."
},
"toolCallExecution": {
"result": "error",
"error": "Developer Info execution telemetry is unavailable.",
"code": "toolCallEvidenceIndeterminate"
}
}
}
Diagnostics and saved-response evaluation
| Output | Tool-call evidence |
|---|---|
| JSON | diagnostics.tool_calls |
| CSV | Complete serialized diagnostics object |
| HTML / console | Tool Call Diagnostics with selections and HTTP outcomes |
Selection display deduplicates identities; execution diagnostics preserve every attempt. null means unavailable evidence; [] means an authoritative observation of no calls.
Re-score saved selections and execution outcomes without executing prompts. Each saved item or turn must include a response:
npx -y --package @microsoft/m365-copilot-eval@preview runevals --evaluate-only results\captured-tool-calls.json --output results\rescored.html
Warning
Diagnostics can contain sensitive tool and execution details. EVALS_SUPPRESS_DIAGNOSTICS=true omits retained diagnostics.
It doesn't redact prompts, responses, evaluator options, or native validation messages. Review artifacts before sharing. Suppressed evidence isn't available for replay.