Usare lo strumento per l'uso del computer per gli agenti (anteprima)

Screenshot dell'inizializzazione per l'esecuzione dello strumento informatico

Gli estratti seguenti illustrano come creare una versione dell'agente con lo strumento di utilizzo del computer, inviare una richiesta iniziale con uno screenshot ed eseguire più iterazioni per completare un'attività. Gli estratti di codice di Prompt Agents dipendono dall'esempio Python aggiornato e dall'helper menzionati in precedenza. Selezionare Prompt Agents per usare Azure AI Projects SDK per creare un agente prompt sul lato server o Hosted Agents per usare Agent Framework FoundryChatClient per creare un agente temporaneo in-process.

Agenti rapidi

from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient
from azure.ai.projects.models import PromptAgentDefinition, ComputerUsePreviewTool

# Import shared helper functions
from computer_use_util import (
    SearchState,
    load_screenshot_assets,
    handle_computer_action_and_take_screenshot,
    print_final_output,
)

"""Main function to demonstrate Computer Use Agent functionality."""
# Initialize state machine
current_state = SearchState.INITIAL

# Load screenshot assets
try:
    screenshots = load_screenshot_assets()
    print("Successfully loaded screenshot assets")
except FileNotFoundError:
    print("Failed to load required screenshot assets. Use the maintained SDK sample on GitHub to get the helper file and images.")
    exit(1)

Creare una versione dell'agente con lo strumento

# Format: "https://resource_name.ai.azure.com/api/projects/project_name"
PROJECT_ENDPOINT = "your_project_endpoint"

project = AIProjectClient(
    endpoint=PROJECT_ENDPOINT,
    credential=DefaultAzureCredential(),
)

computer_use_tool = ComputerUsePreviewTool(display_width=1026, display_height=769, environment="windows")

agent = project.agents.create_version(
    agent_name="ComputerUseAgent",
    definition=PromptAgentDefinition(
        model="computer-use-preview",
        instructions="""
        You are a computer automation assistant. 

        Be direct and efficient. When you reach the search results page, read and describe the actual search result titles and descriptions you can see.
        """,
        tools=[computer_use_tool],
    ),
    description="Computer automation agent with screen interaction capabilities.",
)
print(f"Agent created (id: {agent.id}, name: {agent.name})")

Un'iterazione per lo strumento per elaborare lo screenshot ed eseguire il passaggio successivo

openai = project.get_openai_client()

# Initial request with screenshot - start with Bing search page
response = openai.responses.create(
    input=[
        {
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": "I need you to help me search for 'OpenAI news'. Please type 'OpenAI news' and submit the search. Once you see search results, the task is complete.",
                },
                {
                    "type": "input_image",
                    "image_url": screenshots["browser_search"]["url"],
                    "detail": "high",
                },  # Start with Bing search page
            ],
        }
    ],
    extra_body={"agent_reference": {"name": agent.name, "type": "agent_reference"}},
    truncation="auto",
)

print(f"Initial response received (ID: {response.id})")

Eseguire più iterazioni

Assicurarsi di esaminare ogni iterazione e azione. L'esempio di codice seguente illustra una richiesta API di base. Dopo aver inviato la richiesta API iniziale, eseguire un ciclo in cui il codice dell'applicazione esegue l'azione specificata. Inviare uno screenshot con ogni turno in modo che il modello possa valutare lo stato aggiornato dell'ambiente. L'esempio include un numero massimo di iterazioni per evitare cicli infiniti, ma è possibile modificarlo in base alle esigenze.

max_iterations = 10  # Allow enough iterations for completion
iteration = 0

while True:
    if iteration >= max_iterations:
        print(f"\nReached maximum iterations ({max_iterations}). Stopping.")
        break

    iteration += 1
    print(f"\n--- Iteration {iteration} ---")

    # Check for computer calls in the response
    computer_calls = [item for item in response.output if item.type == "computer_call"]

    if not computer_calls:
        print_final_output(response)
        break

    # Process the first computer call
    computer_call = computer_calls[0]
    action = computer_call.action
    call_id = computer_call.call_id

    # Never execute an action with pending safety checks without user approval.
    safety_checks = computer_call.pending_safety_checks or []
    if safety_checks:
      for check in safety_checks:
        print(f"Safety check: {check.code}: {check.message}")
      if input("Approve this action? Type yes to continue: ").lower() != "yes":
        print("Action rejected by the user.")
        break

    # Handle the action and get the screenshot info
    screenshot_info, current_state = handle_computer_action_and_take_screenshot(action, current_state, screenshots)

    # Regular response with just the screenshot
    response = openai.responses.create(
        previous_response_id=response.id,
        input=[
            {
                "call_id": call_id,
                "type": "computer_call_output",
                "acknowledged_safety_checks": safety_checks,
                "output": {
                    "type": "computer_screenshot",
                    "image_url": screenshot_info["url"],
                },
            }
        ],
        extra_body={"agent_reference": {"name": agent.name, "type": "agent_reference"}},
        truncation="auto",
    )

    print(f"Iteration {iteration}: response received (ID: {response.id})")

Eseguire la pulizia

project.agents.delete_version(agent_name=agent.name, agent_version=agent.version)
print("Agent deleted")

Output previsto

L'esempio seguente mostra l'output previsto durante l'esecuzione dell'esempio di codice precedente:

Successfully loaded screenshot assets
Agent created (id: ..., name: ComputerUseAgent, version: 1)
Starting computer automation session (initial screenshot: cua_browser_search.png)...
Initial response received (ID: ...)
--- Iteration 1 ---
Processing computer call (ID: ...)
  Typing text "OpenAI news" - Simulating keyboard input
  -> Action processed: type
Sending action result back to agent (using cua_search_typed.png)...
Follow-up response received (ID: ...)
--- Iteration 2 ---
Processing computer call (ID: ...)
    Click at (512, 384) - Simulating click on UI element
    -> Assuming click on Search button when search field was populated, displaying results.
    -> Action processed: click
Sending action result back to agent (using cua_search_results.png)...
Follow-up response received (ID: ...)
OpenAI news - Latest Updates
Agent deleted

Agenti ospitati

Questo esempio usa FoundryChatClient da Microsoft Agent Framework e chiama get_computer_use_tool() per collegare lo strumento di anteprima dell'uso del computer. Installa il pacchetto con pip install agent-framework-foundry aiohttp, configura FOUNDRY_PROJECT_ENDPOINT (indica a FOUNDRY_MODEL una distribuzione computer-use-preview ) e accedi con az login. Il ciclo di acquisizione degli screenshot è specifico dell'applicazione; consultare il file di supporto di esempio a monte menzionato di seguito.

import asyncio

from agent_framework import Agent
from agent_framework.foundry import FoundryChatClient
from azure.identity import AzureCliCredential


async def main() -> None:
    agent = Agent(
        client=FoundryChatClient(credential=AzureCliCredential()),
        instructions=(
            "You are a computer automation assistant. Be direct and efficient. "
            "When you reach the search results page, describe the actual result titles you can see."
        ),
        tools=[
            FoundryChatClient.get_computer_use_tool(
                environment="windows",
                display_width=1026,
                display_height=769,
            )
        ],
    )

    # Replace this with your screenshot capture + action handler loop.
    # See the upstream samples folder for a reference implementation.
    result = await agent.run(
        "Help me search for 'OpenAI news'. Type the query and submit the search."
    )
    print(f"Agent: {result.text}")


if __name__ == "__main__":
    asyncio.run(main())

Output previsto

L'agente rilascia azioni di utilizzo del computer (clic, sequenze di tasti, screenshot) fino al completamento dell'attività, quindi descrive la pagina raggiunta:

Agent: I searched for "OpenAI news" in the address bar. The top results include articles from OpenAI's blog, TechCrunch, and The Verge ...

Per un'implementazione completa del ciclo di screenshot, vedere gli esempi di provider Foundry.

Esempio per l'uso di un agente con lo strumento di utilizzo del computer

L'esempio di codice C# seguente illustra come creare un agente con lo strumento per l'uso del computer, inviare una richiesta iniziale con uno screenshot ed eseguire più iterazioni per completare un'attività. Selezionare Prompt Agents per usare Azure AI Projects SDK per creare un agente prompt sul lato server o Hosted Agents per usare Microsoft Agent Framework per creare un agente temporaneo e in-process.

Agenti rapidi

Per consentire all'agente di usare lo strumento per l'uso del computer, usare ResponseTool.CreateComputerTool() quando si configurano gli strumenti dell'agente. In questo esempio viene usato il codice sincrono. Per l'utilizzo asincrono, vedere l'esempio di codice sample nell'Azure SDK per .NET repository in GitHub.

using System;
using System.Runtime.CompilerServices;
using Azure.AI.Projects;
using Azure.AI.Extensions.OpenAI;
using Azure.Identity;

class ComputerUseDemo
{
    // Format: "https://resource_name.ai.azure.com/api/projects/project_name"
    private const string ProjectEndpoint = "your_project_endpoint";

    // Read image files using `ReadImageFile` method.
    private static BinaryData ReadImageFile(string name, [CallerFilePath] string pth = "")
    {
        var dirName = Path.GetDirectoryName(pth) ?? "";
        return new BinaryData(File.ReadAllBytes(Path.Combine(dirName, name)));
    }

    // Create a helper method to parse the ComputerTool outputs and to respond
    // to Agents queries with new screenshots. Note that throughout
    // this sample the media type for image is set. Agents support `image/jpeg`,
    // `image/png`, `image/gif` and `image/webp` media types.
    private static string ProcessComputerUseCall(ComputerCallResponseItem item, string oldScreenshot)
    {
        string currentScreenshot = "browser_search";
        switch (item.Action.Kind)
        {
            case ComputerCallActionKind.Type:
                Console.WriteLine($"  Typing text \"{item.Action.TypeText}\" - Simulating keyboard input");
                currentScreenshot = "search_typed";
                break;
            case ComputerCallActionKind.KeyPress:
                HashSet<string> codes = new(item.Action.KeyPressKeyCodes);
                if (codes.Contains("Return") || codes.Contains("ENTER"))
                {
                    // If we have typed the value to the search field, go to search results.
                    if (string.Equals(oldScreenshot, "search_typed"))
                    {
                        Console.WriteLine("  -> Detected ENTER key press, when search field was populated, displaying results.");
                        currentScreenshot = "search_results";
                    }
                    else
                    {
                        Console.WriteLine("  -> Detected ENTER key press, on results or unpopulated search, do nothing.");
                        currentScreenshot = oldScreenshot;
                    }
                }
                else
                {
                    Console.WriteLine($"  Key press: {item.Action.KeyPressKeyCodes.Aggregate("", (agg, next) => agg + "+" + next)} - Simulating key combination");
                }
                break;
            case ComputerCallActionKind.Click:
                Console.WriteLine($"  Click at ({item.Action.ClickCoordinates.Value.X}, {item.Action.ClickCoordinates.Value.Y}) - Simulating click on UI element");
                if (string.Equals(oldScreenshot, "search_typed"))
                {
                    Console.WriteLine("  -> Assuming click on Search button when search field was populated, displaying results.");
                    currentScreenshot = "search_results";
                }
                else
                {
                    Console.WriteLine("  -> Assuming click on Search on results or when search was not populated, do nothing.");
                    currentScreenshot = oldScreenshot;
                }
                break;
            case ComputerCallActionKind.Drag:
                string pathStr = item.Action.DragPath.ToArray().Select(p => $"{p.X}, {p.Y}").Aggregate("", (agg, next) => $"{agg} -> {next}");
                Console.WriteLine($"  Drag path: {pathStr} - Simulating drag operation");
                break;
            case ComputerCallActionKind.Scroll:
                Console.WriteLine($"  Scroll at ({item.Action.ScrollCoordinates.Value.X}, {item.Action.ScrollCoordinates.Value.Y}) - Simulating scroll action");
                break;
            case ComputerCallActionKind.Screenshot:
                Console.WriteLine("  Taking screenshot - Capturing current screen state");
                break;
            default:
                break;
        }
        Console.WriteLine($"  -> Action processed: {item.Action.Kind}");

        return currentScreenshot;
    }

    public static void Main()
    {
        // Create project client
        AIProjectClient projectClient = new(endpoint: new Uri(ProjectEndpoint), tokenProvider: new DefaultAzureCredential());

        // Read in three example screenshots and place them into a dictionary.
        Dictionary<string, BinaryData> screenshots = new() {
            { "browser_search", ReadImageFile("Assets/cua_browser_search.png")},
            { "search_typed", ReadImageFile("Assets/cua_search_typed.png")},
            { "search_results", ReadImageFile("Assets/cua_search_results.png")},
        };

        // Create a PromptAgentDefinition with ComputerTool.
        DeclarativeAgentDefinition agentDefinition = new(model: "computer-use-preview")
        {
            Instructions = "You are a computer automation assistant.\n\n" +
                            "Be direct and efficient. When you reach the search results page, read and describe the actual search result titles and descriptions you can see.",
            Tools = {
                ResponseTool.CreateComputerTool(
                    environment: new ComputerToolEnvironment("windows"),
                    displayWidth: 1026,
                    displayHeight: 769
                ),
            }
        };
        AgentVersion agentVersion = projectClient.AgentAdministrationClient.CreateAgentVersion(
            agentName: "myAgent",
            options: new(agentDefinition)
        );
        // Create an `ResponseResult` using `ResponseItem`, containing two `ResponseContentPart`:
        // one with the image and another with the text. In the loop, request Agent
        // while it is continuing to browse web. Finally, print the tool output message.
        ProjectResponsesClient responseClient = projectClient.ProjectOpenAIClient.GetProjectResponsesClientForAgent(agentVersion.Name);
        CreateResponseOptions responseOptions = new()
        {
            TruncationMode = ResponseTruncationMode.Auto,
            InputItems =
            {
                ResponseItem.CreateUserMessageItem(
                [
                    ResponseContentPart.CreateInputTextPart("I need you to help me search for 'OpenAI news'. Please type 'OpenAI news' and submit the search. Once you see search results, the task is complete."),
                    ResponseContentPart.CreateInputImagePart(imageBytes: screenshots["browser_search"], imageBytesMediaType: "image/png", imageDetailLevel: ResponseImageDetailLevel.High)
                ]),
            },
        };
        bool computerUseCalled = false;
        string currentScreenshot = "browser_search";
        int limitIteration = 10;
        ResponseResult response;
        do
        {
            response = responseClient.CreateResponse(responseOptions);
            computerUseCalled = false;
            responseOptions.InputItems.Clear();
            responseOptions.PreviousResponseId = response.Id;
            foreach (ResponseItem responseItem in response.OutputItems)
            {
                responseOptions.InputItems.Add(responseItem);
                if (responseItem is ComputerCallResponseItem computerCall)
                {
                  if (computerCall.PendingSafetyChecks.Count > 0)
                  {
                    throw new InvalidOperationException(
                      "Pause execution and obtain end-user approval before acknowledging safety checks."
                    );
                  }
                    currentScreenshot = ProcessComputerUseCall(computerCall, currentScreenshot);
                    responseOptions.InputItems.Add(ResponseItem.CreateComputerCallOutputItem(callId: computerCall.CallId, output: ComputerCallOutput.CreateScreenshotOutput(screenshotImageBytes: screenshots[currentScreenshot], screenshotImageBytesMediaType: "image/png")));
                    computerUseCalled = true;
                }
            }
            limitIteration--;
        } while (computerUseCalled && limitIteration > 0);
        Console.WriteLine(response.GetOutputText());

        // Clean up resources by deleting Agent.
        projectClient.AgentAdministrationClient.DeleteAgentVersion(agentName: agentVersion.Name, agentVersion: agentVersion.Version);
    }
}

Output previsto

L'esempio seguente mostra l'output previsto durante l'esecuzione dell'esempio di codice precedente:

Agent created (id: ..., name: myAgent, version: 1)
Starting computer automation session (initial screenshot: cua_browser_search.png)...
Initial response received (ID: ...)
--- Iteration 1 ---
Processing computer call (ID: ...)
  Typing text "OpenAI news" - Simulating keyboard input
  -> Action processed: Type
Sending action result back to agent (using cua_search_typed.png)...
Follow-up response received (ID: ...)
--- Iteration 2 ---
Processing computer call (ID: ...)
  Click at (512, 384) - Simulating click on UI element
  -> Assuming click on Search button when search field was populated, displaying results.
  -> Action processed: Click
Sending action result back to agent (using cua_search_results.png)...
Follow-up response received (ID: ...)
OpenAI news - Latest Updates
Agent deleted

Agenti ospitati

Questo esempio usa Microsoft Agent Framework e chiama AsAIAgent(...) in AIProjectClient insieme a FoundryAITool.CreateComputerTool(...) da Microsoft.Agents.AI.Foundry per assegnare all'agente lo strumento di utilizzo del computer. Installare i pacchetti Microsoft.Agents.AI.Foundry e Azure.AI.Projects, impostare le variabili di ambiente AZURE_AI_PROJECT_ENDPOINT e AZURE_AI_COMPUTER_USE_DEPLOYMENT_NAME e accedere con az login. Questo esempio omette le funzioni di supporto per gli screenshot — consulta l'esempio completo per il ciclo delle azioni e le utilità per le risorse.

using Azure.AI.Projects;
using Azure.Identity;
using Microsoft.Agents.AI;
using Microsoft.Agents.AI.Foundry;
using Microsoft.Extensions.AI;
using OpenAI.Responses;

string endpoint = Environment.GetEnvironmentVariable("AZURE_AI_PROJECT_ENDPOINT")
    ?? throw new InvalidOperationException("AZURE_AI_PROJECT_ENDPOINT is not set.");
string deploymentName = Environment.GetEnvironmentVariable("AZURE_AI_COMPUTER_USE_DEPLOYMENT_NAME") ?? "computer-use-preview";

AIProjectClient projectClient = new(new Uri(endpoint), new DefaultAzureCredential());
using IHostedFileClient fileClient = projectClient.GetProjectOpenAIClient().AsIHostedFileClient();

AIAgent agent = projectClient.AsAIAgent(
    model: deploymentName,
    name: "ComputerAgent",
    instructions: "You are a computer automation assistant.",
    tools: [FoundryAITool.CreateComputerTool(ComputerToolEnvironment.Browser, 1026, 769)]);

// Upload pre-captured screenshots that simulate browser state transitions.
// (See the full sample for ComputerUseUtil implementation.)
Dictionary<string, string> screenshots = await ComputerUseUtil.UploadScreenshotAssetsAsync(fileClient);

ChatClientAgentRunOptions runOptions = new()
{
    ChatOptions = new ChatOptions
    {
        RawRepresentationFactory = (_) => new CreateResponseOptions { TruncationMode = ResponseTruncationMode.Auto },
    }
};

ChatMessage message = new(ChatRole.User,
[
    new TextContent("Search for 'OpenAI news'. Type it and submit. Once you see results, the task is complete."),
    new AIContent { RawRepresentation = ResponseContentPart.CreateInputImagePart(imageFileId: screenshots["browser_search"], imageDetailLevel: ResponseImageDetailLevel.High) }
]);

AgentSession session = await agent.CreateSessionAsync();
AgentResponse response = await agent.RunAsync(message, session: session, options: runOptions);

// Loop: parse computer call actions from response, simulate them, return new screenshots.
for (int i = 0; i < 10; i++)
{
    ComputerCallResponseItem? computerCall = response.Messages
        .SelectMany(m => m.Contents)
        .Select(c => c.RawRepresentation as ComputerCallResponseItem)
        .FirstOrDefault(item => item is not null);

    if (computerCall is null) break;

    (_, string fileId) = await ComputerUseUtil.GetScreenshotAsync(computerCall.Action, default, screenshots);

    AIContent callOutput = new()
    {
        RawRepresentation = new ComputerCallOutputResponseItem(
            computerCall.CallId,
            output: ComputerCallOutput.CreateScreenshotOutput(screenshotImageFileId: fileId))
    };

    response = await agent.RunAsync([new ChatMessage(ChatRole.User, [callOutput])], session: session, options: runOptions);
}

await ComputerUseUtil.EnsureDeleteScreenshotAssetsAsync(fileClient, screenshots);
Console.WriteLine($"Response: {response.Text}");

Output previsto

Al termine del ciclo di azione, la risposta dell'agente finale descrive la pagina raggiunta:

Response: I searched for "OpenAI news" in the address bar. The top results include articles from OpenAI's blog, TechCrunch, and The Verge ...

Per l'implementazione completa dell'helper per screenshot e il ciclo d'azione end-to-end, consultare Agent_Step15_ComputerUse.

Esempio per l'uso di un agente con lo strumento di utilizzo del computer

L'estratto di TypeScript seguente illustra come creare una versione dell'agente con lo strumento di utilizzo del computer, inviare una richiesta iniziale con uno screenshot ed eseguire più iterazioni. Importa un helper locale computerUseUtil.js e prevede gli asset di screenshot che non sono inclusi in questo articolo. Tratta l'estratto come uno schema di integrazione e fornisci l'esecuzione delle azioni gestite dall'applicazione, l'acquisizione di schermate e l'approvazione esplicita in materia di sicurezza prima di confermare i controlli di sicurezza in sospeso.

import { DefaultAzureCredential } from "@azure/identity";
import { AIProjectClient } from "@azure/ai-projects";
import { createInterface } from "node:readline/promises";
import { stdin, stdout } from "node:process";
import {
  SearchState,
  loadScreenshotAssets,
  handleComputerActionAndTakeScreenshot,
  printFinalOutput,
  type ComputerAction,
} from "./computerUseUtil.js";

// Format: "https://resource_name.ai.azure.com/api/projects/project_name"
const PROJECT_ENDPOINT = "your_project_endpoint";

export async function main(): Promise<void> {
  // Initialize state machine
  let currentState = SearchState.INITIAL;

  // Load screenshot assets
  const screenshots = loadScreenshotAssets();
  console.log("Successfully loaded screenshot assets");

  // Create AI Project client
  const project = new AIProjectClient(PROJECT_ENDPOINT, new DefaultAzureCredential());
  const openai = project.getOpenAIClient();

  console.log("Creating Computer Use Agent...");
  const agent = await project.agents.createVersion("ComputerUseAgent", {
    kind: "prompt" as const,
    model: "computer-use-preview",
    instructions: `
You are a computer automation assistant.

Be direct and efficient. When you reach the search results page, read and describe the actual search result titles and descriptions you can see.
    `.trim(),
    tools: [
      {
        type: "computer_use_preview",
        display_width: 1026,
        display_height: 769,
        environment: "windows" as const,
      },
    ],
  });
  console.log(`Agent created (id: ${agent.id}, name: ${agent.name}, version: ${agent.version})`);

  // Initial request with screenshot - start with Bing search page
  console.log(
    "Starting computer automation session (initial screenshot: cua_browser_search.png)...",
  );
  let response = await openai.responses.create(
    {
      input: [
        {
          role: "user" as const,
          content: [
            {
              type: "input_text",
              text: "I need you to help me search for 'OpenAI news'. Please type 'OpenAI news' and submit the search. Once you see search results, the task is complete.",
            },
            {
              type: "input_image",
              image_url: screenshots.browser_search.url,
              detail: "high",
            },
          ],
        },
      ],
      truncation: "auto",
    },
    {
      body: { agent_reference: { name: agent.name, type: "agent_reference" } },
    },
  );

  console.log(`Initial response received (ID: ${response.id})`);

  // Main interaction loop with deterministic completion
  const maxIterations = 10; // Allow enough iterations for completion
  let iteration = 0;

  while (iteration < maxIterations) {
    iteration++;
    console.log(`\n--- Iteration ${iteration} ---`);

    // Check for computer calls in the response
    const computerCalls = response.output.filter((item) => item.type === "computer_call");

    if (computerCalls.length === 0) {
      printFinalOutput({
        output: response.output,
        status: response.status ?? "",
      });
      break;
    }

    // Process the first computer call
    const computerCall = computerCalls[0];
    const action: ComputerAction = computerCall.action;
    const callId: string = computerCall.call_id;

    // Never execute an action with pending safety checks without user approval.
    const safetyChecks = computerCall.pending_safety_checks ?? [];
    if (safetyChecks.length > 0) {
      for (const check of safetyChecks) {
        console.warn(`Safety check: ${check.code}: ${check.message}`);
      }
      const prompt = createInterface({ input: stdin, output: stdout });
      const answer = await prompt.question("Approve this action? Type yes to continue: ");
      prompt.close();
      if (answer.toLowerCase() !== "yes") {
        throw new Error("Action rejected by the user.");
      }
    }

    console.log(`Processing computer call (ID: ${callId})`);

    // Handle the action and get the screenshot info
    const [screenshotInfo, updatedState] = handleComputerActionAndTakeScreenshot(
      action,
      currentState,
      screenshots,
    );
    currentState = updatedState;

    console.log(`Sending action result back to agent (using ${screenshotInfo.filename})...`);
    // Regular response with just the screenshot
    response = await openai.responses.create(
      {
        previous_response_id: response.id,
        input: [
          {
            call_id: callId,
            type: "computer_call_output",
            acknowledged_safety_checks: safetyChecks,
            output: {
              type: "computer_screenshot",
              image_url: screenshotInfo.url,
            },
          },
        ],
        truncation: "auto",
      },
      {
        body: { agent_reference: { name: agent.name, type: "agent_reference" } },
      },
    );

    console.log(`Follow-up response received (ID: ${response.id})`);
  }

  if (iteration >= maxIterations) {
    console.log(`\nReached maximum iterations (${maxIterations}). Stopping.`);
  }

  // Clean up resources
  console.log("\nCleaning up...");
  await project.agents.deleteVersion(agent.name, agent.version);
  console.log("Agent deleted");

  console.log("\nComputer Use Agent sample completed!");
}

main().catch((err) => {
  console.error("The sample encountered an error:", err);
});

Output previsto

L'esempio seguente mostra l'output previsto durante l'esecuzione dell'esempio di codice precedente:

Successfully loaded screenshot assets
Creating Computer Use Agent...
Agent created (id: ..., name: ComputerUseAgent, version: 1)
Starting computer automation session (initial screenshot: cua_browser_search.png)...
Initial response received (ID: ...)
--- Iteration 1 ---
Processing computer call (ID: ...)
  Typing text "OpenAI news" - Simulating keyboard input
  -> Action processed: type
Sending action result back to agent (using cua_search_typed.png)...
Follow-up response received (ID: ...)
--- Iteration 2 ---
Processing computer call (ID: ...)
    Click at (512, 384) - Simulating click on UI element
    -> Assuming click on Search button when search field was populated, displaying results.
    -> Action processed: click
Sending action result back to agent (using cua_search_results.png)...
Follow-up response received (ID: ...)
OpenAI news - Latest Updates
Cleaning up...
Agent deleted
Computer Use Agent sample completed!

Utilizzare le funzionalità del computer in un agente Java

Aggiungi la dipendenza a pom.xml:

<dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-ai-agents</artifactId>
    <version>2.4.0</version>
</dependency>

Creare un agente per l'uso di un computer

import com.azure.ai.agents.AgentsClient;
import com.azure.ai.agents.AgentsClientBuilder;
import com.azure.ai.agents.ResponsesClient;
import com.azure.ai.agents.models.*;
import com.azure.identity.DefaultAzureCredentialBuilder;
import com.openai.models.responses.Response;
import com.openai.models.responses.ResponseCreateParams;

import java.util.Collections;

public class ComputerUseExample {
    // Format: "https://resource_name.ai.azure.com/api/projects/project_name"
    private static final String PROJECT_ENDPOINT = "your_project_endpoint";

    public static void main(String[] args) {

        AgentsClientBuilder builder = new AgentsClientBuilder()
            .credential(new DefaultAzureCredentialBuilder().build())
            .endpoint(PROJECT_ENDPOINT);

        AgentsClient agentsClient = builder.buildAgentsClient();
        ResponsesClient responsesClient = builder.buildResponsesClient();

        // Create computer use tool
        ComputerUsePreviewTool tool = new ComputerUsePreviewTool(
            ComputerEnvironment.WINDOWS,
            1024,
            768
        );

        // Create agent with computer use tool
        PromptAgentDefinition agentDefinition = new PromptAgentDefinition("computer-use-preview")
            .setInstructions("You are a computer automation assistant.")
            .setTools(Collections.singletonList(tool));

        AgentVersionDetails agent = agentsClient.createAgentVersion("computer-use-agent", agentDefinition);
        System.out.printf("Agent created: %s (version %s)%n", agent.getName(), agent.getVersion());

        // Create a response with initial screenshot
        AgentReference agentReference = new AgentReference(agent.getName())
            .setVersion(agent.getVersion());

        Response response = responsesClient.createAzureResponse(
            new AzureCreateResponseOptions().setAgentReference(agentReference),
            ResponseCreateParams.builder()
                .input("Open the browser and navigate to microsoft.com"));

        System.out.println("Response: " + response.output());

        // The response will contain computer_call items with actions
        // to execute. Process each action, take screenshots, and
        // send results back using responsesClient.createAzureResponse()
        // with the previousResponseId and computer call output.

        // Clean up
        agentsClient.deleteAgentVersion(agent.getName(), agent.getVersion());
    }
}

Per il ciclo simulato completo, usare l'esempio di ComputerUseSync.java gestito con il relativo helper ComputerUseUtil.java. Il componente helper associa le azioni richieste alle schermate acquisite in precedenza. Sostituisci la simulazione con l'esecutore di azioni della tua applicazione, l'acquisizione di schermate e il flusso di approvazione della sicurezza.

Usare il computer con l'API REST

Ottenere un token di accesso:

export AGENT_TOKEN=$(az account get-access-token --scope "https://ai.azure.com/.default" --query accessToken -o tsv)

Creare un agente con l'uso del computer

curl -X POST "$FOUNDRY_PROJECT_ENDPOINT/agents?api-version=v1" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AGENT_TOKEN" \
  -d '{
    "name": "computer-use-agent",
    "definition": {
      "kind": "prompt",
      "model": "computer-use-preview",
      "instructions": "You are a computer automation assistant.",
      "tools": [
        {
          "type": "computer_use_preview",
          "environment": "windows",
          "display_width": 1024,
          "display_height": 768
        }
      ]
    }
  }'

Generare una risposta

curl -X POST "$FOUNDRY_PROJECT_ENDPOINT/openai/v1/responses" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AGENT_TOKEN" \
  -d '{
    "agent_reference": {"type": "agent_reference", "name": "computer-use-agent"},
    "input": "Open the browser and navigate to microsoft.com"
  }'

La risposta include computer_call elementi di output con azioni da eseguire. Prima di eseguire un'azione, esaminare pending_safety_checks. Se la matrice non è vuota, sospendere e mostrare l'azione e i controlli di sicurezza all'utente finale. Continuare solo dopo che l'utente approva esplicitamente l'azione.

Inviare i risultati dell'azione con screenshot

Dopo che l'utente ha approvato eventuali controlli di sicurezza in sospeso e l'applicazione esegue l'azione sul computer, acquisisci uno screenshot e rimandalo. Includi ogni verifica approvata in acknowledged_safety_checks. Se non sono stati restituiti controlli, usare una matrice vuota.

curl -X POST "$FOUNDRY_PROJECT_ENDPOINT/openai/v1/responses" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AGENT_TOKEN" \
  -d '{
    "agent_reference": {"type": "agent_reference", "name": "computer-use-agent"},
    "previous_response_id": "<RESPONSE_ID>",
    "input": [
      {
        "type": "computer_call_output",
        "call_id": "<CALL_ID>",
        "acknowledged_safety_checks": [],
        "output": {
          "type": "computer_screenshot",
          "image_url": "data:image/png;base64,<BASE64_SCREENSHOT>"
        }
      }
    ]
  }'

Sostituire <RESPONSE_ID>, <CALL_ID>e <BASE64_SCREENSHOT> con i valori della risposta precedente. Ripetere questo ciclo fino a quando il modello non restituisce una risposta di testo anziché un oggetto computer_call.

Eseguire la pulizia

curl -X DELETE "$FOUNDRY_PROJECT_ENDPOINT/agents/computer-use-agent?api-version=v1" \
  -H "Authorization: Bearer $AGENT_TOKEN"