Rediger

Simulate conversations with the Microsoft Foundry SDK (preview)

Important

Items marked (preview) in this article are currently in public preview. This preview is provided without a service-level agreement, and we don't recommend it for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.

Generate simulated conversations from scenario descriptions and evaluate them at the conversation level. Use this scenario to test your agent's behavior in controlled situations before deployment. The service generates realistic conversations based on your scenario descriptions and then evaluates them.

Prerequisites

The examples use the SDK client configured in Set up the SDK client.

Understand conversation simulation

This approach is useful for:

  • Pre-deployment testing: Validate agent behavior across diverse scenarios without real user traffic.
  • Edge case coverage: Test scenarios that rarely occur naturally but are important to handle well.
  • Regression testing: Ensure agent updates don't degrade performance on known scenarios.
  • Scale testing: Generate many conversations quickly to stress-test agent capabilities.

Conversation simulation follows these steps:

  1. You provide a dataset of scenario descriptions—each row describes a situation the simulated user tries to accomplish.
  2. The service uses a simulator model to play the role of the user, interacting with your agent based on the scenario.
  3. Each scenario generates one or more complete conversations.
  4. Conversation-level evaluators assess the generated conversations.
  5. Your project stores both the conversations and evaluation results.

Prepare scenario data

Tip

Instead of authoring scenarios by hand, generate them by using the Simulation seed (multi-turn) task type. The generated dataset contains the required test_case_description field and can also contain id, category, and desired_num_turns. Use the generated dataset's ID as scenarios_id in the simulation run and skip the upload step. See Generate a simulation seed dataset.

Create a JSONL file where each line describes a scenario for the simulated user. Each row must contain test_case_description. The id, category, and desired_num_turns fields are optional. Include details about the user's goal, context, and constraints. For a complete example, see the conversation evaluation samples in the SDK.

{"id": "contoso_refund_timeline", "test_case_description": "Customer returned an item to Contoso Electronics 5 days ago and hasn't received their refund yet. They want to know how long Contoso refunds take.", "desired_num_turns": 10}
{"id": "contoso_store_hours_lookup", "test_case_description": "Customer wants to know what time the Contoso Electronics store closes today. Simple single-fact question with possibly one clarifying turn about which location.", "desired_num_turns": 3}

Use these parameters to configure the simulation:

Parameter Required Description
num_conversations No Number of conversations to generate per scenario. Defaults to 5, server-side cap of 5.
max_turns No Maximum number of turns (exchanges) per conversation. Defaults to 10, server-side cap of 50.
model Yes Model deployment to use for simulating the user. For example, gpt-4.1. The model router isn't supported as the simulator model; it can only be used as the evaluation target.
sampling_params No Sampling parameters for the simulator model, including temperature, top_p, and max_completion_tokens.
data_mapping No Maps fields from your scenario JSONL to simulation parameters. Common mappings: test_case_description, id, desired_num_turns.

Define evaluators

Select evaluators designed for conversation-level assessment. The simulated conversations automatically map to the evaluators.

import os
from openai.types.eval_create_params import DataSourceConfigCustom
from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient
from azure.ai.projects.models import TestingCriterionAzureAIEvaluator, PromptAgentDefinition

endpoint = os.environ["AZURE_AI_PROJECT_ENDPOINT"]
model_deployment_name = os.environ["AZURE_AI_MODEL_DEPLOYMENT_NAME"]
agent_name = os.environ.get("FOUNDRY_AGENT_NAME", "")

with (
    DefaultAzureCredential() as credential,
    AIProjectClient(endpoint=endpoint, credential=credential) as project_client,
    project_client.get_openai_client() as openai_client,
):
    # Simulation uses the same "custom" eval group type as dataset evaluation (S1),
    # since the generated conversations follow the same messages schema.
    data_source_config = DataSourceConfigCustom(
        type="custom",
        item_schema={
            "type": "object",
            "properties": {
                "messages": {"type": "array"},
            },
            "required": ["messages"],
        },
        include_sample_schema=False,
    )

    testing_criteria = [
        TestingCriterionAzureAIEvaluator(
            type="azure_ai_evaluator",
            name="customer_satisfaction",
            evaluator_name="builtin.customer_satisfaction",
            initialization_parameters={"model": model_deployment_name},
            data_mapping={"messages": "{{item.messages}}"},
        ),
        TestingCriterionAzureAIEvaluator(
            type="azure_ai_evaluator",
            name="task_completion",
            evaluator_name="builtin.task_completion",
            initialization_parameters={"model": model_deployment_name},
            data_mapping={"messages": "{{item.messages}}"},
        ),
    ]

Create the evaluation and run

Download sample_data_simulation_scenarios.jsonl.

# Create (or update) an agent to simulate against
agent = project_client.agents.create_version(
    agent_name=agent_name,
    definition=PromptAgentDefinition(
        model=model_deployment_name,
        instructions="You are a helpful customer service agent. Be empathetic and solution-oriented.",
    ),
)

# Upload scenario data
scenarios_id = project_client.datasets.upload_file(
    name="simulation-scenarios",
    version="1",
    file_path="./sample_data_simulation_scenarios.jsonl",
).id

# Create the evaluation
eval_object = openai_client.evals.create(
    name="Multi-turn Conversation Simulation",
    data_source_config=data_source_config,
    testing_criteria=testing_criteria,
)

# Create a simulation run
eval_run = openai_client.evals.runs.create(
    eval_id=eval_object.id,
    name="conversation-simulation-run",
    data_source={
        "type": "azure_ai_target_completions",
        "source": {
            "type": "file_id",
            "id": scenarios_id,
        },
        "target": {
            "type": "azure_ai_agent",
            "name": agent.name,
            "version": agent.version,
        },
        "item_generation_params": {
            "type": "conversation_gen_preview",
            "model": model_deployment_name,
            "num_conversations": 2,
            "max_turns": 5,
            "sampling_params": {
                "temperature": 0.7,
                "top_p": 1.0,
                "max_completion_tokens": 800,
            },
            "data_mapping": {
                "test_case_description": "test_case_description",
                "id": "id",
                "desired_num_turns": "desired_num_turns",
            },
        },
    },
    extra_body={"evaluation_level": "conversation"},
)

Next steps

  • Last updated on