Usar a ferramenta de utilização do computador para agentes (pré-visualização)

Inicialização de capturas de ecrã para execução de ferramentas informáticas

Os excertos seguintes demonstram como criar uma versão agente com a ferramenta de utilização do computador, enviar um pedido inicial com uma captura de ecrã e realizar múltiplas iterações para completar uma tarefa. Os excertos de Prompt Agents dependem do exemplo em Python mantido e do auxiliar cuja ligação foi fornecida acima. Selecione Prompt Agents para usar o SDK Azure AI Projects para criar um agente de prompt do lado do servidor, ou Hosted Agents para usar o Agent Framework FoundryChatClient para construir um agente efémero em processo.

Agentes de comando

from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient
from azure.ai.projects.models import PromptAgentDefinition, ComputerUsePreviewTool

# Import shared helper functions
from computer_use_util import (
    SearchState,
    load_screenshot_assets,
    handle_computer_action_and_take_screenshot,
    print_final_output,
)

"""Main function to demonstrate Computer Use Agent functionality."""
# Initialize state machine
current_state = SearchState.INITIAL

# Load screenshot assets
try:
    screenshots = load_screenshot_assets()
    print("Successfully loaded screenshot assets")
except FileNotFoundError:
    print("Failed to load required screenshot assets. Use the maintained SDK sample on GitHub to get the helper file and images.")
    exit(1)

Crie uma versão de agente com a ferramenta

# Format: "https://resource_name.ai.azure.com/api/projects/project_name"
PROJECT_ENDPOINT = "your_project_endpoint"

project = AIProjectClient(
    endpoint=PROJECT_ENDPOINT,
    credential=DefaultAzureCredential(),
)

computer_use_tool = ComputerUsePreviewTool(display_width=1026, display_height=769, environment="windows")

agent = project.agents.create_version(
    agent_name="ComputerUseAgent",
    definition=PromptAgentDefinition(
        model="computer-use-preview",
        instructions="""
        You are a computer automation assistant. 

        Be direct and efficient. When you reach the search results page, read and describe the actual search result titles and descriptions you can see.
        """,
        tools=[computer_use_tool],
    ),
    description="Computer automation agent with screen interaction capabilities.",
)
print(f"Agent created (id: {agent.id}, name: {agent.name})")

Uma iteração para a ferramenta processar a captura de ecrã e dar o próximo passo

openai = project.get_openai_client()

# Initial request with screenshot - start with Bing search page
response = openai.responses.create(
    input=[
        {
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": "I need you to help me search for 'OpenAI news'. Please type 'OpenAI news' and submit the search. Once you see search results, the task is complete.",
                },
                {
                    "type": "input_image",
                    "image_url": screenshots["browser_search"]["url"],
                    "detail": "high",
                },  # Start with Bing search page
            ],
        }
    ],
    extra_body={"agent_reference": {"name": agent.name, "type": "agent_reference"}},
    truncation="auto",
)

print(f"Initial response received (ID: {response.id})")

Realizar múltiplas iterações

Certifica-te de rever cada iteração e ação. O exemplo de código seguinte mostra um pedido básico de API. Depois de enviar o pedido inicial da API, execute um ciclo onde o código da sua aplicação executa a ação especificada. Envie uma captura de ecrã em cada turno para que o modelo possa avaliar o estado atualizado do ambiente. A amostra inclui um número máximo de iterações para evitar ciclos infinitos, mas pode ajustá-lo conforme necessário.

max_iterations = 10  # Allow enough iterations for completion
iteration = 0

while True:
    if iteration >= max_iterations:
        print(f"\nReached maximum iterations ({max_iterations}). Stopping.")
        break

    iteration += 1
    print(f"\n--- Iteration {iteration} ---")

    # Check for computer calls in the response
    computer_calls = [item for item in response.output if item.type == "computer_call"]

    if not computer_calls:
        print_final_output(response)
        break

    # Process the first computer call
    computer_call = computer_calls[0]
    action = computer_call.action
    call_id = computer_call.call_id

    # Never execute an action with pending safety checks without user approval.
    safety_checks = computer_call.pending_safety_checks or []
    if safety_checks:
      for check in safety_checks:
        print(f"Safety check: {check.code}: {check.message}")
      if input("Approve this action? Type yes to continue: ").lower() != "yes":
        print("Action rejected by the user.")
        break

    # Handle the action and get the screenshot info
    screenshot_info, current_state = handle_computer_action_and_take_screenshot(action, current_state, screenshots)

    # Regular response with just the screenshot
    response = openai.responses.create(
        previous_response_id=response.id,
        input=[
            {
                "call_id": call_id,
                "type": "computer_call_output",
                "acknowledged_safety_checks": safety_checks,
                "output": {
                    "type": "computer_screenshot",
                    "image_url": screenshot_info["url"],
                },
            }
        ],
        extra_body={"agent_reference": {"name": agent.name, "type": "agent_reference"}},
        truncation="auto",
    )

    print(f"Iteration {iteration}: response received (ID: {response.id})")

Limpar

project.agents.delete_version(agent_name=agent.name, agent_version=agent.version)
print("Agent deleted")

Produção esperada

O exemplo seguinte mostra a saída esperada ao executar o exemplo de código anterior:

Successfully loaded screenshot assets
Agent created (id: ..., name: ComputerUseAgent, version: 1)
Starting computer automation session (initial screenshot: cua_browser_search.png)...
Initial response received (ID: ...)
--- Iteration 1 ---
Processing computer call (ID: ...)
  Typing text "OpenAI news" - Simulating keyboard input
  -> Action processed: type
Sending action result back to agent (using cua_search_typed.png)...
Follow-up response received (ID: ...)
--- Iteration 2 ---
Processing computer call (ID: ...)
    Click at (512, 384) - Simulating click on UI element
    -> Assuming click on Search button when search field was populated, displaying results.
    -> Action processed: click
Sending action result back to agent (using cua_search_results.png)...
Follow-up response received (ID: ...)
OpenAI news - Latest Updates
Agent deleted

Agentes alojados

Este exemplo utiliza FoundryChatClient do Microsoft Agent Framework e chama get_computer_use_tool() para associar a ferramenta de pré-visualização de utilização do computador. Instale o pacote com pip install agent-framework-foundry aiohttp, defina o FOUNDRY_PROJECT_ENDPOINT (aponte FOUNDRY_MODEL para uma computer-use-preview implantação) e inicie sessão com az login. O ciclo de captura de ecrã é específico da aplicação; veja o ficheiro auxiliar de exemplo original referido abaixo.

import asyncio

from agent_framework import Agent
from agent_framework.foundry import FoundryChatClient
from azure.identity import AzureCliCredential


async def main() -> None:
    agent = Agent(
        client=FoundryChatClient(credential=AzureCliCredential()),
        instructions=(
            "You are a computer automation assistant. Be direct and efficient. "
            "When you reach the search results page, describe the actual result titles you can see."
        ),
        tools=[
            FoundryChatClient.get_computer_use_tool(
                environment="windows",
                display_width=1026,
                display_height=769,
            )
        ],
    )

    # Replace this with your screenshot capture + action handler loop.
    # See the upstream samples folder for a reference implementation.
    result = await agent.run(
        "Help me search for 'OpenAI news'. Type the query and submit the search."
    )
    print(f"Agent: {result.text}")


if __name__ == "__main__":
    asyncio.run(main())

Produção esperada

O agente emite ações de utilização do computador (cliques, teclas de tecla, capturas de ecrã) até a tarefa ser concluída, descrevendo então a página a que chegou:

Agent: I searched for "OpenAI news" in the address bar. The top results include articles from OpenAI's blog, TechCrunch, and The Verge ...

Para uma implementação completa do ciclo de capturas de ecrã, consulte os exemplos do fornecedor Foundry.

Exemplo para utilização de um Agente com ferramenta de Utilização Informática

O seguinte exemplo de código C# demonstra como criar um agente com a ferramenta de utilização do computador, enviar um pedido inicial com uma captura de ecrã e realizar múltiplas iterações para completar uma tarefa. Selecione Prompt Agents para usar o SDK Azure AI Projects para criar um agente de prompt do lado do servidor, ou Hosted Agents para usar o Microsoft Agent Framework para construir um agente efémero em processo.

Agentes de comando

Para permitir que o seu agente utilize a ferramenta de uso do computador, use ResponseTool.CreateComputerTool() ao configurar as ferramentas do agente. Este exemplo utiliza código síncrono. Para uso assíncrono, veja o exemplo de código no repositório do SDK do Azure para .NET no GitHub.

using System;
using System.Runtime.CompilerServices;
using Azure.AI.Projects;
using Azure.AI.Extensions.OpenAI;
using Azure.Identity;

class ComputerUseDemo
{
    // Format: "https://resource_name.ai.azure.com/api/projects/project_name"
    private const string ProjectEndpoint = "your_project_endpoint";

    // Read image files using `ReadImageFile` method.
    private static BinaryData ReadImageFile(string name, [CallerFilePath] string pth = "")
    {
        var dirName = Path.GetDirectoryName(pth) ?? "";
        return new BinaryData(File.ReadAllBytes(Path.Combine(dirName, name)));
    }

    // Create a helper method to parse the ComputerTool outputs and to respond
    // to Agents queries with new screenshots. Note that throughout
    // this sample the media type for image is set. Agents support `image/jpeg`,
    // `image/png`, `image/gif` and `image/webp` media types.
    private static string ProcessComputerUseCall(ComputerCallResponseItem item, string oldScreenshot)
    {
        string currentScreenshot = "browser_search";
        switch (item.Action.Kind)
        {
            case ComputerCallActionKind.Type:
                Console.WriteLine($"  Typing text \"{item.Action.TypeText}\" - Simulating keyboard input");
                currentScreenshot = "search_typed";
                break;
            case ComputerCallActionKind.KeyPress:
                HashSet<string> codes = new(item.Action.KeyPressKeyCodes);
                if (codes.Contains("Return") || codes.Contains("ENTER"))
                {
                    // If we have typed the value to the search field, go to search results.
                    if (string.Equals(oldScreenshot, "search_typed"))
                    {
                        Console.WriteLine("  -> Detected ENTER key press, when search field was populated, displaying results.");
                        currentScreenshot = "search_results";
                    }
                    else
                    {
                        Console.WriteLine("  -> Detected ENTER key press, on results or unpopulated search, do nothing.");
                        currentScreenshot = oldScreenshot;
                    }
                }
                else
                {
                    Console.WriteLine($"  Key press: {item.Action.KeyPressKeyCodes.Aggregate("", (agg, next) => agg + "+" + next)} - Simulating key combination");
                }
                break;
            case ComputerCallActionKind.Click:
                Console.WriteLine($"  Click at ({item.Action.ClickCoordinates.Value.X}, {item.Action.ClickCoordinates.Value.Y}) - Simulating click on UI element");
                if (string.Equals(oldScreenshot, "search_typed"))
                {
                    Console.WriteLine("  -> Assuming click on Search button when search field was populated, displaying results.");
                    currentScreenshot = "search_results";
                }
                else
                {
                    Console.WriteLine("  -> Assuming click on Search on results or when search was not populated, do nothing.");
                    currentScreenshot = oldScreenshot;
                }
                break;
            case ComputerCallActionKind.Drag:
                string pathStr = item.Action.DragPath.ToArray().Select(p => $"{p.X}, {p.Y}").Aggregate("", (agg, next) => $"{agg} -> {next}");
                Console.WriteLine($"  Drag path: {pathStr} - Simulating drag operation");
                break;
            case ComputerCallActionKind.Scroll:
                Console.WriteLine($"  Scroll at ({item.Action.ScrollCoordinates.Value.X}, {item.Action.ScrollCoordinates.Value.Y}) - Simulating scroll action");
                break;
            case ComputerCallActionKind.Screenshot:
                Console.WriteLine("  Taking screenshot - Capturing current screen state");
                break;
            default:
                break;
        }
        Console.WriteLine($"  -> Action processed: {item.Action.Kind}");

        return currentScreenshot;
    }

    public static void Main()
    {
        // Create project client
        AIProjectClient projectClient = new(endpoint: new Uri(ProjectEndpoint), tokenProvider: new DefaultAzureCredential());

        // Read in three example screenshots and place them into a dictionary.
        Dictionary<string, BinaryData> screenshots = new() {
            { "browser_search", ReadImageFile("Assets/cua_browser_search.png")},
            { "search_typed", ReadImageFile("Assets/cua_search_typed.png")},
            { "search_results", ReadImageFile("Assets/cua_search_results.png")},
        };

        // Create a PromptAgentDefinition with ComputerTool.
        DeclarativeAgentDefinition agentDefinition = new(model: "computer-use-preview")
        {
            Instructions = "You are a computer automation assistant.\n\n" +
                            "Be direct and efficient. When you reach the search results page, read and describe the actual search result titles and descriptions you can see.",
            Tools = {
                ResponseTool.CreateComputerTool(
                    environment: new ComputerToolEnvironment("windows"),
                    displayWidth: 1026,
                    displayHeight: 769
                ),
            }
        };
        AgentVersion agentVersion = projectClient.AgentAdministrationClient.CreateAgentVersion(
            agentName: "myAgent",
            options: new(agentDefinition)
        );
        // Create an `ResponseResult` using `ResponseItem`, containing two `ResponseContentPart`:
        // one with the image and another with the text. In the loop, request Agent
        // while it is continuing to browse web. Finally, print the tool output message.
        ProjectResponsesClient responseClient = projectClient.ProjectOpenAIClient.GetProjectResponsesClientForAgent(agentVersion.Name);
        CreateResponseOptions responseOptions = new()
        {
            TruncationMode = ResponseTruncationMode.Auto,
            InputItems =
            {
                ResponseItem.CreateUserMessageItem(
                [
                    ResponseContentPart.CreateInputTextPart("I need you to help me search for 'OpenAI news'. Please type 'OpenAI news' and submit the search. Once you see search results, the task is complete."),
                    ResponseContentPart.CreateInputImagePart(imageBytes: screenshots["browser_search"], imageBytesMediaType: "image/png", imageDetailLevel: ResponseImageDetailLevel.High)
                ]),
            },
        };
        bool computerUseCalled = false;
        string currentScreenshot = "browser_search";
        int limitIteration = 10;
        ResponseResult response;
        do
        {
            response = responseClient.CreateResponse(responseOptions);
            computerUseCalled = false;
            responseOptions.InputItems.Clear();
            responseOptions.PreviousResponseId = response.Id;
            foreach (ResponseItem responseItem in response.OutputItems)
            {
                responseOptions.InputItems.Add(responseItem);
                if (responseItem is ComputerCallResponseItem computerCall)
                {
                  if (computerCall.PendingSafetyChecks.Count > 0)
                  {
                    throw new InvalidOperationException(
                      "Pause execution and obtain end-user approval before acknowledging safety checks."
                    );
                  }
                    currentScreenshot = ProcessComputerUseCall(computerCall, currentScreenshot);
                    responseOptions.InputItems.Add(ResponseItem.CreateComputerCallOutputItem(callId: computerCall.CallId, output: ComputerCallOutput.CreateScreenshotOutput(screenshotImageBytes: screenshots[currentScreenshot], screenshotImageBytesMediaType: "image/png")));
                    computerUseCalled = true;
                }
            }
            limitIteration--;
        } while (computerUseCalled && limitIteration > 0);
        Console.WriteLine(response.GetOutputText());

        // Clean up resources by deleting Agent.
        projectClient.AgentAdministrationClient.DeleteAgentVersion(agentName: agentVersion.Name, agentVersion: agentVersion.Version);
    }
}

Produção esperada

O exemplo seguinte mostra a saída esperada ao executar o exemplo de código anterior:

Agent created (id: ..., name: myAgent, version: 1)
Starting computer automation session (initial screenshot: cua_browser_search.png)...
Initial response received (ID: ...)
--- Iteration 1 ---
Processing computer call (ID: ...)
  Typing text "OpenAI news" - Simulating keyboard input
  -> Action processed: Type
Sending action result back to agent (using cua_search_typed.png)...
Follow-up response received (ID: ...)
--- Iteration 2 ---
Processing computer call (ID: ...)
  Click at (512, 384) - Simulating click on UI element
  -> Assuming click on Search button when search field was populated, displaying results.
  -> Action processed: Click
Sending action result back to agent (using cua_search_results.png)...
Follow-up response received (ID: ...)
OpenAI news - Latest Updates
Agent deleted

Agentes alojados

Este exemplo utiliza o Microsoft Agent Framework e invoca AsAIAgent(...) em AIProjectClient juntamente com FoundryAITool.CreateComputerTool(...) de Microsoft.Agents.AI.Foundry para disponibilizar ao agente a ferramenta de utilização do computador. Instala os pacotes Microsoft.Agents.AI.Foundry e Azure.AI.Projects, define as variáveis de ambiente AZURE_AI_PROJECT_ENDPOINT e AZURE_AI_COMPUTER_USE_DEPLOYMENT_NAME e inicia sessão com az login. Este exemplo omite as funções auxiliares para capturas de ecrã — veja o exemplo completo para o ciclo de ações e os utilitários de recursos.

using Azure.AI.Projects;
using Azure.Identity;
using Microsoft.Agents.AI;
using Microsoft.Agents.AI.Foundry;
using Microsoft.Extensions.AI;
using OpenAI.Responses;

string endpoint = Environment.GetEnvironmentVariable("AZURE_AI_PROJECT_ENDPOINT")
    ?? throw new InvalidOperationException("AZURE_AI_PROJECT_ENDPOINT is not set.");
string deploymentName = Environment.GetEnvironmentVariable("AZURE_AI_COMPUTER_USE_DEPLOYMENT_NAME") ?? "computer-use-preview";

AIProjectClient projectClient = new(new Uri(endpoint), new DefaultAzureCredential());
using IHostedFileClient fileClient = projectClient.GetProjectOpenAIClient().AsIHostedFileClient();

AIAgent agent = projectClient.AsAIAgent(
    model: deploymentName,
    name: "ComputerAgent",
    instructions: "You are a computer automation assistant.",
    tools: [FoundryAITool.CreateComputerTool(ComputerToolEnvironment.Browser, 1026, 769)]);

// Upload pre-captured screenshots that simulate browser state transitions.
// (See the full sample for ComputerUseUtil implementation.)
Dictionary<string, string> screenshots = await ComputerUseUtil.UploadScreenshotAssetsAsync(fileClient);

ChatClientAgentRunOptions runOptions = new()
{
    ChatOptions = new ChatOptions
    {
        RawRepresentationFactory = (_) => new CreateResponseOptions { TruncationMode = ResponseTruncationMode.Auto },
    }
};

ChatMessage message = new(ChatRole.User,
[
    new TextContent("Search for 'OpenAI news'. Type it and submit. Once you see results, the task is complete."),
    new AIContent { RawRepresentation = ResponseContentPart.CreateInputImagePart(imageFileId: screenshots["browser_search"], imageDetailLevel: ResponseImageDetailLevel.High) }
]);

AgentSession session = await agent.CreateSessionAsync();
AgentResponse response = await agent.RunAsync(message, session: session, options: runOptions);

// Loop: parse computer call actions from response, simulate them, return new screenshots.
for (int i = 0; i < 10; i++)
{
    ComputerCallResponseItem? computerCall = response.Messages
        .SelectMany(m => m.Contents)
        .Select(c => c.RawRepresentation as ComputerCallResponseItem)
        .FirstOrDefault(item => item is not null);

    if (computerCall is null) break;

    (_, string fileId) = await ComputerUseUtil.GetScreenshotAsync(computerCall.Action, default, screenshots);

    AIContent callOutput = new()
    {
        RawRepresentation = new ComputerCallOutputResponseItem(
            computerCall.CallId,
            output: ComputerCallOutput.CreateScreenshotOutput(screenshotImageFileId: fileId))
    };

    response = await agent.RunAsync([new ChatMessage(ChatRole.User, [callOutput])], session: session, options: runOptions);
}

await ComputerUseUtil.EnsureDeleteScreenshotAssetsAsync(fileClient, screenshots);
Console.WriteLine($"Response: {response.Text}");

Produção esperada

Após a conclusão do ciclo de ação, a resposta final do agente descreve a página a que chegou:

Response: I searched for "OpenAI news" in the address bar. The top results include articles from OpenAI's blog, TechCrunch, and The Verge ...

Para a implementação completa do ajudante de captura de ecrã e o ciclo de ações de ponta a ponta, veja Agent_Step15_ComputerUse.

Exemplo para utilização de um Agente com ferramenta de Utilização Informática

O excerto seguinte do TypeScript demonstra como criar uma versão do agente com a ferramenta de uso do computador, enviar um pedido inicial com uma captura de ecrã e realizar múltiplas iterações. Importa um assistente local computerUseUtil.js e espera recursos de captura de ecrã que não estão incluídos neste artigo. Trate o excerto como um esquema de integração e forneça a execução de ações da responsabilidade da aplicação, a captura de ecrã e a aprovação de segurança explícita antes de confirmar as verificações de segurança pendentes.

import { DefaultAzureCredential } from "@azure/identity";
import { AIProjectClient } from "@azure/ai-projects";
import { createInterface } from "node:readline/promises";
import { stdin, stdout } from "node:process";
import {
  SearchState,
  loadScreenshotAssets,
  handleComputerActionAndTakeScreenshot,
  printFinalOutput,
  type ComputerAction,
} from "./computerUseUtil.js";

// Format: "https://resource_name.ai.azure.com/api/projects/project_name"
const PROJECT_ENDPOINT = "your_project_endpoint";

export async function main(): Promise<void> {
  // Initialize state machine
  let currentState = SearchState.INITIAL;

  // Load screenshot assets
  const screenshots = loadScreenshotAssets();
  console.log("Successfully loaded screenshot assets");

  // Create AI Project client
  const project = new AIProjectClient(PROJECT_ENDPOINT, new DefaultAzureCredential());
  const openai = project.getOpenAIClient();

  console.log("Creating Computer Use Agent...");
  const agent = await project.agents.createVersion("ComputerUseAgent", {
    kind: "prompt" as const,
    model: "computer-use-preview",
    instructions: `
You are a computer automation assistant.

Be direct and efficient. When you reach the search results page, read and describe the actual search result titles and descriptions you can see.
    `.trim(),
    tools: [
      {
        type: "computer_use_preview",
        display_width: 1026,
        display_height: 769,
        environment: "windows" as const,
      },
    ],
  });
  console.log(`Agent created (id: ${agent.id}, name: ${agent.name}, version: ${agent.version})`);

  // Initial request with screenshot - start with Bing search page
  console.log(
    "Starting computer automation session (initial screenshot: cua_browser_search.png)...",
  );
  let response = await openai.responses.create(
    {
      input: [
        {
          role: "user" as const,
          content: [
            {
              type: "input_text",
              text: "I need you to help me search for 'OpenAI news'. Please type 'OpenAI news' and submit the search. Once you see search results, the task is complete.",
            },
            {
              type: "input_image",
              image_url: screenshots.browser_search.url,
              detail: "high",
            },
          ],
        },
      ],
      truncation: "auto",
    },
    {
      body: { agent_reference: { name: agent.name, type: "agent_reference" } },
    },
  );

  console.log(`Initial response received (ID: ${response.id})`);

  // Main interaction loop with deterministic completion
  const maxIterations = 10; // Allow enough iterations for completion
  let iteration = 0;

  while (iteration < maxIterations) {
    iteration++;
    console.log(`\n--- Iteration ${iteration} ---`);

    // Check for computer calls in the response
    const computerCalls = response.output.filter((item) => item.type === "computer_call");

    if (computerCalls.length === 0) {
      printFinalOutput({
        output: response.output,
        status: response.status ?? "",
      });
      break;
    }

    // Process the first computer call
    const computerCall = computerCalls[0];
    const action: ComputerAction = computerCall.action;
    const callId: string = computerCall.call_id;

    // Never execute an action with pending safety checks without user approval.
    const safetyChecks = computerCall.pending_safety_checks ?? [];
    if (safetyChecks.length > 0) {
      for (const check of safetyChecks) {
        console.warn(`Safety check: ${check.code}: ${check.message}`);
      }
      const prompt = createInterface({ input: stdin, output: stdout });
      const answer = await prompt.question("Approve this action? Type yes to continue: ");
      prompt.close();
      if (answer.toLowerCase() !== "yes") {
        throw new Error("Action rejected by the user.");
      }
    }

    console.log(`Processing computer call (ID: ${callId})`);

    // Handle the action and get the screenshot info
    const [screenshotInfo, updatedState] = handleComputerActionAndTakeScreenshot(
      action,
      currentState,
      screenshots,
    );
    currentState = updatedState;

    console.log(`Sending action result back to agent (using ${screenshotInfo.filename})...`);
    // Regular response with just the screenshot
    response = await openai.responses.create(
      {
        previous_response_id: response.id,
        input: [
          {
            call_id: callId,
            type: "computer_call_output",
            acknowledged_safety_checks: safetyChecks,
            output: {
              type: "computer_screenshot",
              image_url: screenshotInfo.url,
            },
          },
        ],
        truncation: "auto",
      },
      {
        body: { agent_reference: { name: agent.name, type: "agent_reference" } },
      },
    );

    console.log(`Follow-up response received (ID: ${response.id})`);
  }

  if (iteration >= maxIterations) {
    console.log(`\nReached maximum iterations (${maxIterations}). Stopping.`);
  }

  // Clean up resources
  console.log("\nCleaning up...");
  await project.agents.deleteVersion(agent.name, agent.version);
  console.log("Agent deleted");

  console.log("\nComputer Use Agent sample completed!");
}

main().catch((err) => {
  console.error("The sample encountered an error:", err);
});

Produção esperada

O exemplo seguinte mostra a saída esperada ao executar o exemplo de código anterior:

Successfully loaded screenshot assets
Creating Computer Use Agent...
Agent created (id: ..., name: ComputerUseAgent, version: 1)
Starting computer automation session (initial screenshot: cua_browser_search.png)...
Initial response received (ID: ...)
--- Iteration 1 ---
Processing computer call (ID: ...)
  Typing text "OpenAI news" - Simulating keyboard input
  -> Action processed: type
Sending action result back to agent (using cua_search_typed.png)...
Follow-up response received (ID: ...)
--- Iteration 2 ---
Processing computer call (ID: ...)
    Click at (512, 384) - Simulating click on UI element
    -> Assuming click on Search button when search field was populated, displaying results.
    -> Action processed: click
Sending action result back to agent (using cua_search_results.png)...
Follow-up response received (ID: ...)
OpenAI news - Latest Updates
Cleaning up...
Agent deleted
Computer Use Agent sample completed!

Utilização de um computador num agente Java

Adicione a dependência ao seu pom.xml:

<dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-ai-agents</artifactId>
    <version>2.4.0</version>
</dependency>

Criar um agente de uso de computador

import com.azure.ai.agents.AgentsClient;
import com.azure.ai.agents.AgentsClientBuilder;
import com.azure.ai.agents.ResponsesClient;
import com.azure.ai.agents.models.*;
import com.azure.identity.DefaultAzureCredentialBuilder;
import com.openai.models.responses.Response;
import com.openai.models.responses.ResponseCreateParams;

import java.util.Collections;

public class ComputerUseExample {
    // Format: "https://resource_name.ai.azure.com/api/projects/project_name"
    private static final String PROJECT_ENDPOINT = "your_project_endpoint";

    public static void main(String[] args) {

        AgentsClientBuilder builder = new AgentsClientBuilder()
            .credential(new DefaultAzureCredentialBuilder().build())
            .endpoint(PROJECT_ENDPOINT);

        AgentsClient agentsClient = builder.buildAgentsClient();
        ResponsesClient responsesClient = builder.buildResponsesClient();

        // Create computer use tool
        ComputerUsePreviewTool tool = new ComputerUsePreviewTool(
            ComputerEnvironment.WINDOWS,
            1024,
            768
        );

        // Create agent with computer use tool
        PromptAgentDefinition agentDefinition = new PromptAgentDefinition("computer-use-preview")
            .setInstructions("You are a computer automation assistant.")
            .setTools(Collections.singletonList(tool));

        AgentVersionDetails agent = agentsClient.createAgentVersion("computer-use-agent", agentDefinition);
        System.out.printf("Agent created: %s (version %s)%n", agent.getName(), agent.getVersion());

        // Create a response with initial screenshot
        AgentReference agentReference = new AgentReference(agent.getName())
            .setVersion(agent.getVersion());

        Response response = responsesClient.createAzureResponse(
            new AzureCreateResponseOptions().setAgentReference(agentReference),
            ResponseCreateParams.builder()
                .input("Open the browser and navigate to microsoft.com"));

        System.out.println("Response: " + response.output());

        // The response will contain computer_call items with actions
        // to execute. Process each action, take screenshots, and
        // send results back using responsesClient.createAzureResponse()
        // with the previousResponseId and computer call output.

        // Clean up
        agentsClient.deleteAgentVersion(agent.getName(), agent.getVersion());
    }
}

Para o ciclo simulado completo, utilize o exemplo ComputerUseSync.java disponibilizado com o ficheiro auxiliar ComputerUseUtil.java. O assistente associa as ações solicitadas a capturas de ecrã previamente obtidas. Substitua essa simulação pelo executor de ações da sua aplicação, captura de ecrã e fluxo de aprovação de segurança.

Utilize o uso de computador com a API REST

Obtenha um token de acesso:

export AGENT_TOKEN=$(az account get-access-token --scope "https://ai.azure.com/.default" --query accessToken -o tsv)

Criar um agente com uso informático

curl -X POST "$FOUNDRY_PROJECT_ENDPOINT/agents?api-version=v1" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AGENT_TOKEN" \
  -d '{
    "name": "computer-use-agent",
    "definition": {
      "kind": "prompt",
      "model": "computer-use-preview",
      "instructions": "You are a computer automation assistant.",
      "tools": [
        {
          "type": "computer_use_preview",
          "environment": "windows",
          "display_width": 1024,
          "display_height": 768
        }
      ]
    }
  }'

Gerar uma resposta

curl -X POST "$FOUNDRY_PROJECT_ENDPOINT/openai/v1/responses" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AGENT_TOKEN" \
  -d '{
    "agent_reference": {"type": "agent_reference", "name": "computer-use-agent"},
    "input": "Open the browser and navigate to microsoft.com"
  }'

A resposta inclui computer_call itens de saída com ações a executar. Antes de executar uma ação, inspecione pending_safety_checks. Se o array não estiver vazio, pausa e mostra as verificações de ação e segurança ao utilizador final. Continue apenas depois de o utilizador aprovar explicitamente a ação.

Enviar resultados de ações com captura de ecrã

Depois de o utilizador aprovar quaisquer verificações de segurança pendentes e a sua aplicação executar a ação do computador, capture uma captura de ecrã e envie-a de volta. Inclua todas as verificações aprovadas em acknowledged_safety_checks. Se não houver verificações, use um array vazio.

curl -X POST "$FOUNDRY_PROJECT_ENDPOINT/openai/v1/responses" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AGENT_TOKEN" \
  -d '{
    "agent_reference": {"type": "agent_reference", "name": "computer-use-agent"},
    "previous_response_id": "<RESPONSE_ID>",
    "input": [
      {
        "type": "computer_call_output",
        "call_id": "<CALL_ID>",
        "acknowledged_safety_checks": [],
        "output": {
          "type": "computer_screenshot",
          "image_url": "data:image/png;base64,<BASE64_SCREENSHOT>"
        }
      }
    ]
  }'

Substitua <RESPONSE_ID>, <CALL_ID>, e <BASE64_SCREENSHOT> por valores da resposta anterior. Repete este ciclo até que o modelo devolva uma resposta de texto em vez de um computer_call.

Limpar

curl -X DELETE "$FOUNDRY_PROJECT_ENDPOINT/agents/computer-use-agent?api-version=v1" \
  -H "Authorization: Bearer $AGENT_TOKEN"