Generate synthetic evaluation data with Microsoft Foundry SDK

Generate individual test queries, send them to a model or agent target, and evaluate the target responses.

Prerequisites

Complete the cloud evaluation prerequisites and client setup.
A model deployment that supports the Responses API for query generation.
A model or agent target to evaluate.

The examples use the SDK client configured in Set up the SDK client.

Generate synthetic queries

Use the azure_ai_synthetic_data_gen_preview data source type to generate synthetic test queries, send them to a deployed model or Foundry agent, and evaluate the responses. Use this scenario when you don't have a test dataset. The service generates queries based on a prompt you provide (and/or from the agent's instructions), runs them against your target, and evaluates the responses.

Important

Before you begin, complete client setup.

How synthetic data evaluation works

The service generates synthetic queries based on your prompt and optional seed data files.
Each query is sent to the specified target (model or agent) to generate a response.
Evaluators score each response using the generated query and response.
The generated queries are stored as a dataset in your project for reuse.

Parameters

Parameter	Required	Description
`samples_count`	Yes	Number of synthetic test queries to generate. Specify a value from 15 through 1,000, inclusive.
`model_deployment_name`	Yes	Model deployment to use for generating synthetic queries. Only models with Responses API capability are supported. For availability, see Responses API region availability. The model router isn't supported here; it can only be used as the evaluation target.
`prompt`	No	Instructions describing the type of queries to generate. Optional when the agent target has instructions configured.
`output_dataset_name`	No	Name for the output dataset where generated queries are stored. If you don't provide a name, the service generates one automatically.
`sources`	No	Seed data files (by file ID) to improve relevance of generated queries. Currently only one file is supported.

Set up evaluators and data mappings

For target evaluations, map every required evaluator input explicitly. In this synthetic-data example, generated queries are available as {{item.query}}, and target responses are available as {{sample.output_text}}.

from azure.ai.projects.models import TestingCriterionAzureAIEvaluator

data_source_config = {"type": "azure_ai_source", "scenario": "synthetic_data_gen_preview"}

testing_criteria = [
    TestingCriterionAzureAIEvaluator(
        type="azure_ai_evaluator",
        name="coherence",
        evaluator_name="builtin.coherence",
        initialization_parameters={"model": model_deployment_name},
        data_mapping={
            "query": "{{item.query}}",
            "response": "{{sample.output_text}}",
        },
    ),
    TestingCriterionAzureAIEvaluator(
        type="azure_ai_evaluator",
        name="violence",
        evaluator_name="builtin.violence",
        data_mapping={
            "query": "{{item.query}}",
            "response": "{{sample.output_text}}",
        },
    ),
]

Create evaluation and run

Model target

Generate synthetic queries and evaluate a model:

eval_object = openai_client.evals.create(
    name="Synthetic Data Evaluation",
    data_source_config=data_source_config,
    testing_criteria=testing_criteria,
)

data_source = {
    "type": "azure_ai_synthetic_data_gen_preview",
    "item_generation_params": {
        "type": "synthetic_data_gen_preview",
        "samples_count": 15,
        "prompt": "Generate customer service questions about returning defective products",
        "model_deployment_name": model_deployment_name,
        "output_dataset_name": "my-synthetic-dataset",
    },
    "target": {
        "type": "azure_ai_model",
        "model": model_deployment_name,
    },
}

eval_run = openai_client.evals.runs.create(
    eval_id=eval_object.id,
    name="synthetic-data-evaluation",
    data_source=data_source,
)

You can optionally add a system prompt to shape the target model's behavior. When you use input_messages with synthetic data generation, include only system role messages - the service provides the generated queries as user messages automatically.

data_source = {
    "type": "azure_ai_synthetic_data_gen_preview",
    "item_generation_params": {
        "type": "synthetic_data_gen_preview",
        "samples_count": 15,
        "prompt": "Generate customer service questions about returning defective products",
        "model_deployment_name": model_deployment_name,
    },
    "target": {
        "type": "azure_ai_model",
        "model": model_deployment_name,
    },
    "input_messages": {
        "type": "template",
        "template": [
            {
                "type": "message",
                "role": "system",
                "content": {
                    "type": "input_text",
                    "text": "You are a helpful customer service agent. Be empathetic and solution-oriented."
                }
            }
        ]
    },
}

Agent target

Generate synthetic queries and evaluate a Foundry agent:

data_source = {
    "type": "azure_ai_synthetic_data_gen_preview",
    "item_generation_params": {
        "type": "synthetic_data_gen_preview",
        "samples_count": 15,
        "prompt": "Generate questions about returning defective products",
        "model_deployment_name": model_deployment_name,
    },
    "target": {
        "type": "azure_ai_agent",
        "name": agent_name,
        "version": agent_version,
    },
}

eval_run = openai_client.evals.runs.create(
    eval_id=eval_object.id,
    name="synthetic-agent-evaluation",
    data_source=data_source,
)

The .NET client uses its protocol methods for this preview data source.

Model target

object[] testingCriteria =
[
  new
  {
    type = "azure_ai_evaluator",
    name = "coherence",
    evaluator_name = "builtin.coherence",
    initialization_parameters = new { model = modelDeploymentName },
    data_mapping = new
    {
      query = "{{item.query}}",
      response = "{{sample.output_text}}"
    }
  },
  new
  {
    type = "azure_ai_evaluator",
    name = "violence",
    evaluator_name = "builtin.violence",
    data_mapping = new
    {
      query = "{{item.query}}",
      response = "{{sample.output_text}}"
    }
  }
];
BinaryData evaluationData = BinaryData.FromObjectAsJson(new
{
  name = "Synthetic Data Evaluation",
  data_source_config = new
  {
    type = "azure_ai_source",
    scenario = "synthetic_data_gen_preview"
  },
  testing_criteria = testingCriteria
});
using BinaryContent evaluationContent = BinaryContent.Create(evaluationData);
ClientResult evaluation = await evaluationClient.CreateEvaluationAsync(
  evaluationContent);
string evaluationId = GetString(evaluation, "id");

BinaryData runData = BinaryData.FromObjectAsJson(new
{
  name = "synthetic-data-evaluation",
  data_source = new
  {
    type = "azure_ai_synthetic_data_gen_preview",
    item_generation_params = new
    {
      type = "synthetic_data_gen_preview",
      samples_count = 15,
      prompt = "Generate customer service questions about returning defective products",
      model_deployment_name = modelDeploymentName,
      output_dataset_name = "my-synthetic-dataset"
    },
    target = new
    {
      type = "azure_ai_model",
      model = modelDeploymentName
    }
  }
});
using BinaryContent runContent = BinaryContent.Create(runData);
ClientResult evaluationRun = await evaluationClient.CreateEvaluationRunAsync(
  evaluationId: evaluationId,
  content: runContent);
Console.WriteLine($"Evaluation run created: {GetString(evaluationRun, "id")}");

You can optionally add a system prompt to shape the target model's behavior. Include only system role messages because the service provides the generated queries as user messages automatically.

object modelDataSourceWithSystemPrompt = new
{
  type = "azure_ai_synthetic_data_gen_preview",
  item_generation_params = new
  {
    type = "synthetic_data_gen_preview",
    samples_count = 15,
    prompt = "Generate customer service questions about returning defective products",
    model_deployment_name = modelDeploymentName
  },
  target = new
  {
    type = "azure_ai_model",
    model = modelDeploymentName
  },
  input_messages = new
  {
    type = "template",
    template = new object[]
    {
      new
      {
        type = "message",
        role = "system",
        content = new
        {
          type = "input_text",
          text = "You are a helpful customer service agent. Be empathetic and solution-oriented."
        }
      }
    }
  }
};

Agent target

Generate synthetic queries and evaluate a Foundry agent:

BinaryData agentRunData = BinaryData.FromObjectAsJson(new
{
  name = "synthetic-agent-evaluation",
  data_source = new
  {
    type = "azure_ai_synthetic_data_gen_preview",
    item_generation_params = new
    {
      type = "synthetic_data_gen_preview",
      samples_count = 15,
      prompt = "Generate questions about returning defective products",
      model_deployment_name = modelDeploymentName
    },
    target = new
    {
      type = "azure_ai_agent",
      name = agentName,
      version = agentVersion
    }
  }
});
using BinaryContent agentRunContent = BinaryContent.Create(agentRunData);
ClientResult agentEvaluationRun =
  await evaluationClient.CreateEvaluationRunAsync(
    evaluationId: evaluationId,
    content: agentRunContent);
Console.WriteLine(
  $"Evaluation run created: {GetString(agentEvaluationRun, "id")}");

Reference: EvaluationClient protocol methods

The TypeScript examples use the OpenAI eval client from AIProjectClient. The as any casts allow the client to forward Azure-specific evaluation configurations and data sources that aren't currently represented by the OpenAI client types.

Model target

Generate synthetic queries and evaluate a model:

const dataSourceConfig = {
  type: "azure_ai_source",
  scenario: "synthetic_data_gen_preview",
};

const testingCriteria = [
  {
    type: "azure_ai_evaluator",
    name: "coherence",
    evaluator_name: "builtin.coherence",
    initialization_parameters: { model: modelDeploymentName },
    data_mapping: {
      query: "{{item.query}}",
      response: "{{sample.output_text}}",
    },
  },
  {
    type: "azure_ai_evaluator",
    name: "violence",
    evaluator_name: "builtin.violence",
    data_mapping: {
      query: "{{item.query}}",
      response: "{{sample.output_text}}",
    },
  },
];

const evalObject = await openAIClient.evals.create({
  name: "Synthetic Data Evaluation",
  data_source_config: dataSourceConfig as any,
  testing_criteria: testingCriteria as any,
});

const modelDataSource = {
  type: "azure_ai_synthetic_data_gen_preview",
  item_generation_params: {
    type: "synthetic_data_gen_preview",
    samples_count: 15,
    prompt:
      "Generate customer service questions about returning defective products",
    model_deployment_name: modelDeploymentName,
    output_dataset_name: "my-synthetic-dataset",
  },
  target: {
    type: "azure_ai_model",
    model: modelDeploymentName,
  },
};

const modelEvalRun = await openAIClient.evals.runs.create(evalObject.id, {
  name: "synthetic-data-evaluation",
  data_source: modelDataSource as any,
});

You can optionally add a system prompt to shape the target model's behavior. Include only system role messages because the service provides the generated queries as user messages automatically.

const modelDataSourceWithSystemPrompt = {
  ...modelDataSource,
  input_messages: {
    type: "template",
    template: [
      {
        type: "message",
        role: "system",
        content: {
          type: "input_text",
          text:
            "You are a helpful customer service agent. Be empathetic and solution-oriented.",
        },
      },
    ],
  },
};

Agent target

Generate synthetic queries and evaluate a Foundry agent:

const agentDataSource = {
  type: "azure_ai_synthetic_data_gen_preview",
  item_generation_params: {
    type: "synthetic_data_gen_preview",
    samples_count: 15,
    prompt: "Generate questions about returning defective products",
    model_deployment_name: modelDeploymentName,
  },
  target: {
    type: "azure_ai_agent",
    name: agentName,
    version: agentVersion,
  },
};

const agentEvalRun = await openAIClient.evals.runs.create(evalObject.id, {
  name: "synthetic-agent-evaluation",
  data_source: agentDataSource as any,
});

# Step 1: Create the evaluation
curl --request POST \
  --url "https://${ACCOUNT}.services.ai.azure.com/api/projects/${PROJECT}/openai/evals?api-version=v1" \
  --header "Authorization: Bearer ${TOKEN}" \
  --header "Content-Type: application/json" \
  --data '{
    "name": "Synthetic Data Evaluation",
    "data_source_config": {
      "type": "azure_ai_source",
      "scenario": "synthetic_data_gen_preview"
    },
    "testing_criteria": [
      {
        "type": "azure_ai_evaluator",
        "name": "coherence",
        "evaluator_name": "builtin.coherence",
        "initialization_parameters": {
          "model": "gpt-5-mini"
        },
        "data_mapping": {
          "query": "{{item.query}}",
          "response": "{{sample.output_text}}"
        }
      },
      {
        "type": "azure_ai_evaluator",
        "name": "violence",
        "evaluator_name": "builtin.violence",
        "data_mapping": {
          "query": "{{item.query}}",
          "response": "{{sample.output_text}}"
        }
      }
    ]
  }'

# Step 2: Create a run with synthetic data generation
curl --request POST \
  --url "https://${ACCOUNT}.services.ai.azure.com/api/projects/${PROJECT}/openai/evals/${EVAL_ID}/runs?api-version=v1" \
  --header "Authorization: Bearer ${TOKEN}" \
  --header "Content-Type: application/json" \
  --data '{
    "name": "synthetic-data-evaluation",
    "data_source": {
      "type": "azure_ai_synthetic_data_gen_preview",
      "item_generation_params": {
        "type": "synthetic_data_gen_preview",
        "samples_count": 15,
        "prompt": "Generate customer service questions about returning defective products",
        "model_deployment_name": "gpt-5-mini",
        "output_dataset_name": "my-synthetic-dataset"
      },
      "target": {
        "type": "azure_ai_model",
        "model": "gpt-5-mini"
      }
    }
  }'

Next steps

To poll for completion and interpret results, see Get cloud evaluation results. The response includes an output_dataset_id property that contains the ID of the generated dataset, which you can use to retrieve or reuse the synthetic data.
For complete runnable examples, see sample_synthetic_data_agent_evaluation.py and sample_synthetic_data_model_evaluation.py on GitHub.

คำติชม

หน้านี้มีประโยชน์หรือไม่

อัปเดตล่าสุดเมื่อ 2026-10-08

Generate synthetic evaluation data with Microsoft Foundry SDK

Prerequisites

Generate synthetic queries

How synthetic data evaluation works

Parameters

Set up evaluators and data mappings

Create evaluation and run

Model target

Agent target

Next steps

คำติชม

แหล่งทรัพยากรเพิ่มเติม