Använda datoranvändningsverktyget för agenter (förhandsversion)

Skärmbild av initiering för körning av verktyg för datoranvändning

Följande utdrag visar hur du skapar en agentversion med datoranvändningsverktyget, skickar en första begäran med en skärmbild och utför flera iterationer för att slutföra en uppgift. Utdrag ur Prompt Agents är beroende av det underhållna Python-exemplet och hjälpfunktionen som tidigare länkades. Välj Prompt Agents för att använda Azure AI Projects SDK för att skapa en agent på serversidan, eller Hosted Agents för att använda Agent Framework FoundryChatClient för att skapa en tillfällig, processbaserad agent.

Aktivera agenter

from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient
from azure.ai.projects.models import PromptAgentDefinition, ComputerUsePreviewTool

# Import shared helper functions
from computer_use_util import (
    SearchState,
    load_screenshot_assets,
    handle_computer_action_and_take_screenshot,
    print_final_output,
)

"""Main function to demonstrate Computer Use Agent functionality."""
# Initialize state machine
current_state = SearchState.INITIAL

# Load screenshot assets
try:
    screenshots = load_screenshot_assets()
    print("Successfully loaded screenshot assets")
except FileNotFoundError:
    print("Failed to load required screenshot assets. Use the maintained SDK sample on GitHub to get the helper file and images.")
    exit(1)

Skapa en agentversion med verktyget

# Format: "https://resource_name.ai.azure.com/api/projects/project_name"
PROJECT_ENDPOINT = "your_project_endpoint"

project = AIProjectClient(
    endpoint=PROJECT_ENDPOINT,
    credential=DefaultAzureCredential(),
)

computer_use_tool = ComputerUsePreviewTool(display_width=1026, display_height=769, environment="windows")

agent = project.agents.create_version(
    agent_name="ComputerUseAgent",
    definition=PromptAgentDefinition(
        model="computer-use-preview",
        instructions="""
        You are a computer automation assistant. 

        Be direct and efficient. When you reach the search results page, read and describe the actual search result titles and descriptions you can see.
        """,
        tools=[computer_use_tool],
    ),
    description="Computer automation agent with screen interaction capabilities.",
)
print(f"Agent created (id: {agent.id}, name: {agent.name})")

En iteration för verktyget för att bearbeta skärmbilden och ta nästa steg

openai = project.get_openai_client()

# Initial request with screenshot - start with Bing search page
response = openai.responses.create(
    input=[
        {
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": "I need you to help me search for 'OpenAI news'. Please type 'OpenAI news' and submit the search. Once you see search results, the task is complete.",
                },
                {
                    "type": "input_image",
                    "image_url": screenshots["browser_search"]["url"],
                    "detail": "high",
                },  # Start with Bing search page
            ],
        }
    ],
    extra_body={"agent_reference": {"name": agent.name, "type": "agent_reference"}},
    truncation="auto",
)

print(f"Initial response received (ID: {response.id})")

Utföra flera iterationer

Kontrollera att du granskar varje iteration och åtgärd. Följande kodexempel visar en grundläggande API-begäran. När du har skickat den första API-begäran utför du en loop där programkoden utför den angivna åtgärden. Skicka en skärmbild med varje tur så att modellen kan utvärdera det uppdaterade tillståndet för miljön. Exemplet innehåller ett maximalt iterationsantal för att förhindra oändliga loopar, men du kan justera detta efter behov.

max_iterations = 10  # Allow enough iterations for completion
iteration = 0

while True:
    if iteration >= max_iterations:
        print(f"\nReached maximum iterations ({max_iterations}). Stopping.")
        break

    iteration += 1
    print(f"\n--- Iteration {iteration} ---")

    # Check for computer calls in the response
    computer_calls = [item for item in response.output if item.type == "computer_call"]

    if not computer_calls:
        print_final_output(response)
        break

    # Process the first computer call
    computer_call = computer_calls[0]
    action = computer_call.action
    call_id = computer_call.call_id

    # Never execute an action with pending safety checks without user approval.
    safety_checks = computer_call.pending_safety_checks or []
    if safety_checks:
      for check in safety_checks:
        print(f"Safety check: {check.code}: {check.message}")
      if input("Approve this action? Type yes to continue: ").lower() != "yes":
        print("Action rejected by the user.")
        break

    # Handle the action and get the screenshot info
    screenshot_info, current_state = handle_computer_action_and_take_screenshot(action, current_state, screenshots)

    # Regular response with just the screenshot
    response = openai.responses.create(
        previous_response_id=response.id,
        input=[
            {
                "call_id": call_id,
                "type": "computer_call_output",
                "acknowledged_safety_checks": safety_checks,
                "output": {
                    "type": "computer_screenshot",
                    "image_url": screenshot_info["url"],
                },
            }
        ],
        extra_body={"agent_reference": {"name": agent.name, "type": "agent_reference"}},
        truncation="auto",
    )

    print(f"Iteration {iteration}: response received (ID: {response.id})")

Rensa

project.agents.delete_version(agent_name=agent.name, agent_version=agent.version)
print("Agent deleted")

Förväntade utdata

I följande exempel visas förväntade utdata när du kör föregående kodexempel:

Successfully loaded screenshot assets
Agent created (id: ..., name: ComputerUseAgent, version: 1)
Starting computer automation session (initial screenshot: cua_browser_search.png)...
Initial response received (ID: ...)
--- Iteration 1 ---
Processing computer call (ID: ...)
  Typing text "OpenAI news" - Simulating keyboard input
  -> Action processed: type
Sending action result back to agent (using cua_search_typed.png)...
Follow-up response received (ID: ...)
--- Iteration 2 ---
Processing computer call (ID: ...)
    Click at (512, 384) - Simulating click on UI element
    -> Assuming click on Search button when search field was populated, displaying results.
    -> Action processed: click
Sending action result back to agent (using cua_search_results.png)...
Follow-up response received (ID: ...)
OpenAI news - Latest Updates
Agent deleted

Hostade agenter

Det här exemplet använder FoundryChatClient från Microsoft Agent Framework och anropar get_computer_use_tool() för att koppla förhandsversionsverktyget för datoranvändning. Installera paketet med pip install agent-framework-foundry aiohttp, ange FOUNDRY_PROJECT_ENDPOINT (punkt FOUNDRY_MODEL vid en computer-use-preview distribution) och logga in med az login. Loopen för skärmbildstagning är programspecifik; se den ursprungliga hjälpfilen med exempel som det hänvisas till nedan.

import asyncio

from agent_framework import Agent
from agent_framework.foundry import FoundryChatClient
from azure.identity import AzureCliCredential


async def main() -> None:
    agent = Agent(
        client=FoundryChatClient(credential=AzureCliCredential()),
        instructions=(
            "You are a computer automation assistant. Be direct and efficient. "
            "When you reach the search results page, describe the actual result titles you can see."
        ),
        tools=[
            FoundryChatClient.get_computer_use_tool(
                environment="windows",
                display_width=1026,
                display_height=769,
            )
        ],
    )

    # Replace this with your screenshot capture + action handler loop.
    # See the upstream samples folder for a reference implementation.
    result = await agent.run(
        "Help me search for 'OpenAI news'. Type the query and submit the search."
    )
    print(f"Agent: {result.text}")


if __name__ == "__main__":
    asyncio.run(main())

Förväntade utdata

Agenten utfärdar åtgärder för datoranvändning (klick, tangenttryckningar, skärmbilder) tills uppgiften har slutförts och beskriver sedan sidan som den nådde:

Agent: I searched for "OpenAI news" in the address bar. The top results include articles from OpenAI's blog, TechCrunch, and The Verge ...

För en fullständig implementering av en skärmbildsloop, se providerexemplen för Foundry.

Exempel för användning av en agent med datoranvändningsverktyget

Följande C#-kodexempel visar hur du skapar en agent med datoranvändningsverktyget, skickar en första begäran med en skärmbild och utför flera iterationer för att slutföra en uppgift. Välj Prompt Agents om du vill använda Azure AI Projects SDK för att skapa en agent på serversidan eller Hosted Agents för att använda Microsoft Agent Framework för att skapa en tillfällig agent i processen.

Aktivera agenter

Om du vill att agenten ska kunna använda datoranvändningsverktyget använder du ResponseTool.CreateComputerTool() när du konfigurerar agentens verktyg. I det här exemplet används synkron kod. För asynkron användning, se exempelkod i Azure SDKs för .NET repository på GitHub.

using System;
using System.Runtime.CompilerServices;
using Azure.AI.Projects;
using Azure.AI.Extensions.OpenAI;
using Azure.Identity;

class ComputerUseDemo
{
    // Format: "https://resource_name.ai.azure.com/api/projects/project_name"
    private const string ProjectEndpoint = "your_project_endpoint";

    // Read image files using `ReadImageFile` method.
    private static BinaryData ReadImageFile(string name, [CallerFilePath] string pth = "")
    {
        var dirName = Path.GetDirectoryName(pth) ?? "";
        return new BinaryData(File.ReadAllBytes(Path.Combine(dirName, name)));
    }

    // Create a helper method to parse the ComputerTool outputs and to respond
    // to Agents queries with new screenshots. Note that throughout
    // this sample the media type for image is set. Agents support `image/jpeg`,
    // `image/png`, `image/gif` and `image/webp` media types.
    private static string ProcessComputerUseCall(ComputerCallResponseItem item, string oldScreenshot)
    {
        string currentScreenshot = "browser_search";
        switch (item.Action.Kind)
        {
            case ComputerCallActionKind.Type:
                Console.WriteLine($"  Typing text \"{item.Action.TypeText}\" - Simulating keyboard input");
                currentScreenshot = "search_typed";
                break;
            case ComputerCallActionKind.KeyPress:
                HashSet<string> codes = new(item.Action.KeyPressKeyCodes);
                if (codes.Contains("Return") || codes.Contains("ENTER"))
                {
                    // If we have typed the value to the search field, go to search results.
                    if (string.Equals(oldScreenshot, "search_typed"))
                    {
                        Console.WriteLine("  -> Detected ENTER key press, when search field was populated, displaying results.");
                        currentScreenshot = "search_results";
                    }
                    else
                    {
                        Console.WriteLine("  -> Detected ENTER key press, on results or unpopulated search, do nothing.");
                        currentScreenshot = oldScreenshot;
                    }
                }
                else
                {
                    Console.WriteLine($"  Key press: {item.Action.KeyPressKeyCodes.Aggregate("", (agg, next) => agg + "+" + next)} - Simulating key combination");
                }
                break;
            case ComputerCallActionKind.Click:
                Console.WriteLine($"  Click at ({item.Action.ClickCoordinates.Value.X}, {item.Action.ClickCoordinates.Value.Y}) - Simulating click on UI element");
                if (string.Equals(oldScreenshot, "search_typed"))
                {
                    Console.WriteLine("  -> Assuming click on Search button when search field was populated, displaying results.");
                    currentScreenshot = "search_results";
                }
                else
                {
                    Console.WriteLine("  -> Assuming click on Search on results or when search was not populated, do nothing.");
                    currentScreenshot = oldScreenshot;
                }
                break;
            case ComputerCallActionKind.Drag:
                string pathStr = item.Action.DragPath.ToArray().Select(p => $"{p.X}, {p.Y}").Aggregate("", (agg, next) => $"{agg} -> {next}");
                Console.WriteLine($"  Drag path: {pathStr} - Simulating drag operation");
                break;
            case ComputerCallActionKind.Scroll:
                Console.WriteLine($"  Scroll at ({item.Action.ScrollCoordinates.Value.X}, {item.Action.ScrollCoordinates.Value.Y}) - Simulating scroll action");
                break;
            case ComputerCallActionKind.Screenshot:
                Console.WriteLine("  Taking screenshot - Capturing current screen state");
                break;
            default:
                break;
        }
        Console.WriteLine($"  -> Action processed: {item.Action.Kind}");

        return currentScreenshot;
    }

    public static void Main()
    {
        // Create project client
        AIProjectClient projectClient = new(endpoint: new Uri(ProjectEndpoint), tokenProvider: new DefaultAzureCredential());

        // Read in three example screenshots and place them into a dictionary.
        Dictionary<string, BinaryData> screenshots = new() {
            { "browser_search", ReadImageFile("Assets/cua_browser_search.png")},
            { "search_typed", ReadImageFile("Assets/cua_search_typed.png")},
            { "search_results", ReadImageFile("Assets/cua_search_results.png")},
        };

        // Create a PromptAgentDefinition with ComputerTool.
        DeclarativeAgentDefinition agentDefinition = new(model: "computer-use-preview")
        {
            Instructions = "You are a computer automation assistant.\n\n" +
                            "Be direct and efficient. When you reach the search results page, read and describe the actual search result titles and descriptions you can see.",
            Tools = {
                ResponseTool.CreateComputerTool(
                    environment: new ComputerToolEnvironment("windows"),
                    displayWidth: 1026,
                    displayHeight: 769
                ),
            }
        };
        AgentVersion agentVersion = projectClient.AgentAdministrationClient.CreateAgentVersion(
            agentName: "myAgent",
            options: new(agentDefinition)
        );
        // Create an `ResponseResult` using `ResponseItem`, containing two `ResponseContentPart`:
        // one with the image and another with the text. In the loop, request Agent
        // while it is continuing to browse web. Finally, print the tool output message.
        ProjectResponsesClient responseClient = projectClient.ProjectOpenAIClient.GetProjectResponsesClientForAgent(agentVersion.Name);
        CreateResponseOptions responseOptions = new()
        {
            TruncationMode = ResponseTruncationMode.Auto,
            InputItems =
            {
                ResponseItem.CreateUserMessageItem(
                [
                    ResponseContentPart.CreateInputTextPart("I need you to help me search for 'OpenAI news'. Please type 'OpenAI news' and submit the search. Once you see search results, the task is complete."),
                    ResponseContentPart.CreateInputImagePart(imageBytes: screenshots["browser_search"], imageBytesMediaType: "image/png", imageDetailLevel: ResponseImageDetailLevel.High)
                ]),
            },
        };
        bool computerUseCalled = false;
        string currentScreenshot = "browser_search";
        int limitIteration = 10;
        ResponseResult response;
        do
        {
            response = responseClient.CreateResponse(responseOptions);
            computerUseCalled = false;
            responseOptions.InputItems.Clear();
            responseOptions.PreviousResponseId = response.Id;
            foreach (ResponseItem responseItem in response.OutputItems)
            {
                responseOptions.InputItems.Add(responseItem);
                if (responseItem is ComputerCallResponseItem computerCall)
                {
                  if (computerCall.PendingSafetyChecks.Count > 0)
                  {
                    throw new InvalidOperationException(
                      "Pause execution and obtain end-user approval before acknowledging safety checks."
                    );
                  }
                    currentScreenshot = ProcessComputerUseCall(computerCall, currentScreenshot);
                    responseOptions.InputItems.Add(ResponseItem.CreateComputerCallOutputItem(callId: computerCall.CallId, output: ComputerCallOutput.CreateScreenshotOutput(screenshotImageBytes: screenshots[currentScreenshot], screenshotImageBytesMediaType: "image/png")));
                    computerUseCalled = true;
                }
            }
            limitIteration--;
        } while (computerUseCalled && limitIteration > 0);
        Console.WriteLine(response.GetOutputText());

        // Clean up resources by deleting Agent.
        projectClient.AgentAdministrationClient.DeleteAgentVersion(agentName: agentVersion.Name, agentVersion: agentVersion.Version);
    }
}

Förväntade utdata

I följande exempel visas förväntade utdata när du kör föregående kodexempel:

Agent created (id: ..., name: myAgent, version: 1)
Starting computer automation session (initial screenshot: cua_browser_search.png)...
Initial response received (ID: ...)
--- Iteration 1 ---
Processing computer call (ID: ...)
  Typing text "OpenAI news" - Simulating keyboard input
  -> Action processed: Type
Sending action result back to agent (using cua_search_typed.png)...
Follow-up response received (ID: ...)
--- Iteration 2 ---
Processing computer call (ID: ...)
  Click at (512, 384) - Simulating click on UI element
  -> Assuming click on Search button when search field was populated, displaying results.
  -> Action processed: Click
Sending action result back to agent (using cua_search_results.png)...
Follow-up response received (ID: ...)
OpenAI news - Latest Updates
Agent deleted

Hostade agenter

Det här exemplet använder Microsoft Agent Framework och anropar AsAIAgent(...) på AIProjectClient tillsammans med FoundryAITool.CreateComputerTool(...) från Microsoft.Agents.AI.Foundry för att ge agenten datoranvändningsverktyget. Installera paketen Microsoft.Agents.AI.Foundry och Azure.AI.Projects, ange miljövariablerna AZURE_AI_PROJECT_ENDPOINT och AZURE_AI_COMPUTER_USE_DEPLOYMENT_NAME och logga in med az login. Det här exemplet utelämnar skärmbildshjälparna – se det fullständiga exemplet för åtgärdsloopen och tillgångsverktygen.

using Azure.AI.Projects;
using Azure.Identity;
using Microsoft.Agents.AI;
using Microsoft.Agents.AI.Foundry;
using Microsoft.Extensions.AI;
using OpenAI.Responses;

string endpoint = Environment.GetEnvironmentVariable("AZURE_AI_PROJECT_ENDPOINT")
    ?? throw new InvalidOperationException("AZURE_AI_PROJECT_ENDPOINT is not set.");
string deploymentName = Environment.GetEnvironmentVariable("AZURE_AI_COMPUTER_USE_DEPLOYMENT_NAME") ?? "computer-use-preview";

AIProjectClient projectClient = new(new Uri(endpoint), new DefaultAzureCredential());
using IHostedFileClient fileClient = projectClient.GetProjectOpenAIClient().AsIHostedFileClient();

AIAgent agent = projectClient.AsAIAgent(
    model: deploymentName,
    name: "ComputerAgent",
    instructions: "You are a computer automation assistant.",
    tools: [FoundryAITool.CreateComputerTool(ComputerToolEnvironment.Browser, 1026, 769)]);

// Upload pre-captured screenshots that simulate browser state transitions.
// (See the full sample for ComputerUseUtil implementation.)
Dictionary<string, string> screenshots = await ComputerUseUtil.UploadScreenshotAssetsAsync(fileClient);

ChatClientAgentRunOptions runOptions = new()
{
    ChatOptions = new ChatOptions
    {
        RawRepresentationFactory = (_) => new CreateResponseOptions { TruncationMode = ResponseTruncationMode.Auto },
    }
};

ChatMessage message = new(ChatRole.User,
[
    new TextContent("Search for 'OpenAI news'. Type it and submit. Once you see results, the task is complete."),
    new AIContent { RawRepresentation = ResponseContentPart.CreateInputImagePart(imageFileId: screenshots["browser_search"], imageDetailLevel: ResponseImageDetailLevel.High) }
]);

AgentSession session = await agent.CreateSessionAsync();
AgentResponse response = await agent.RunAsync(message, session: session, options: runOptions);

// Loop: parse computer call actions from response, simulate them, return new screenshots.
for (int i = 0; i < 10; i++)
{
    ComputerCallResponseItem? computerCall = response.Messages
        .SelectMany(m => m.Contents)
        .Select(c => c.RawRepresentation as ComputerCallResponseItem)
        .FirstOrDefault(item => item is not null);

    if (computerCall is null) break;

    (_, string fileId) = await ComputerUseUtil.GetScreenshotAsync(computerCall.Action, default, screenshots);

    AIContent callOutput = new()
    {
        RawRepresentation = new ComputerCallOutputResponseItem(
            computerCall.CallId,
            output: ComputerCallOutput.CreateScreenshotOutput(screenshotImageFileId: fileId))
    };

    response = await agent.RunAsync([new ChatMessage(ChatRole.User, [callOutput])], session: session, options: runOptions);
}

await ComputerUseUtil.EnsureDeleteScreenshotAssetsAsync(fileClient, screenshots);
Console.WriteLine($"Response: {response.Text}");

Förväntade utdata

När åtgärdsslingan har slutförts beskriver agentens slutliga svar den sida som nåddes:

Response: I searched for "OpenAI news" in the address bar. The top results include articles from OpenAI's blog, TechCrunch, and The Verge ...

För den fullständiga implementeringen av skärmbildshjälpen och den heltäckande åtgärdsloopen, se Agent_Step15_ComputerUse.

Exempel för användning av en agent med datoranvändningsverktyget

Följande TypeScript-utdrag visar hur du skapar en agentversion med datoranvändningsverktyget, skickar en första begäran med en skärmbild och utför flera iterationer. Den importerar en lokal computerUseUtil.js hjälpfunktion och förutsätter skärmbildsfiler som inte ingår i den här artikeln. Behandla utdraget som en integrationsöversikt och tillhandahåll applikationsstyrd körning av åtgärder, skärmbildstagning och ett uttryckligt säkerhetsgodkännande innan du bekräftar utestående säkerhetskontroller.

import { DefaultAzureCredential } from "@azure/identity";
import { AIProjectClient } from "@azure/ai-projects";
import { createInterface } from "node:readline/promises";
import { stdin, stdout } from "node:process";
import {
  SearchState,
  loadScreenshotAssets,
  handleComputerActionAndTakeScreenshot,
  printFinalOutput,
  type ComputerAction,
} from "./computerUseUtil.js";

// Format: "https://resource_name.ai.azure.com/api/projects/project_name"
const PROJECT_ENDPOINT = "your_project_endpoint";

export async function main(): Promise<void> {
  // Initialize state machine
  let currentState = SearchState.INITIAL;

  // Load screenshot assets
  const screenshots = loadScreenshotAssets();
  console.log("Successfully loaded screenshot assets");

  // Create AI Project client
  const project = new AIProjectClient(PROJECT_ENDPOINT, new DefaultAzureCredential());
  const openai = project.getOpenAIClient();

  console.log("Creating Computer Use Agent...");
  const agent = await project.agents.createVersion("ComputerUseAgent", {
    kind: "prompt" as const,
    model: "computer-use-preview",
    instructions: `
You are a computer automation assistant.

Be direct and efficient. When you reach the search results page, read and describe the actual search result titles and descriptions you can see.
    `.trim(),
    tools: [
      {
        type: "computer_use_preview",
        display_width: 1026,
        display_height: 769,
        environment: "windows" as const,
      },
    ],
  });
  console.log(`Agent created (id: ${agent.id}, name: ${agent.name}, version: ${agent.version})`);

  // Initial request with screenshot - start with Bing search page
  console.log(
    "Starting computer automation session (initial screenshot: cua_browser_search.png)...",
  );
  let response = await openai.responses.create(
    {
      input: [
        {
          role: "user" as const,
          content: [
            {
              type: "input_text",
              text: "I need you to help me search for 'OpenAI news'. Please type 'OpenAI news' and submit the search. Once you see search results, the task is complete.",
            },
            {
              type: "input_image",
              image_url: screenshots.browser_search.url,
              detail: "high",
            },
          ],
        },
      ],
      truncation: "auto",
    },
    {
      body: { agent_reference: { name: agent.name, type: "agent_reference" } },
    },
  );

  console.log(`Initial response received (ID: ${response.id})`);

  // Main interaction loop with deterministic completion
  const maxIterations = 10; // Allow enough iterations for completion
  let iteration = 0;

  while (iteration < maxIterations) {
    iteration++;
    console.log(`\n--- Iteration ${iteration} ---`);

    // Check for computer calls in the response
    const computerCalls = response.output.filter((item) => item.type === "computer_call");

    if (computerCalls.length === 0) {
      printFinalOutput({
        output: response.output,
        status: response.status ?? "",
      });
      break;
    }

    // Process the first computer call
    const computerCall = computerCalls[0];
    const action: ComputerAction = computerCall.action;
    const callId: string = computerCall.call_id;

    // Never execute an action with pending safety checks without user approval.
    const safetyChecks = computerCall.pending_safety_checks ?? [];
    if (safetyChecks.length > 0) {
      for (const check of safetyChecks) {
        console.warn(`Safety check: ${check.code}: ${check.message}`);
      }
      const prompt = createInterface({ input: stdin, output: stdout });
      const answer = await prompt.question("Approve this action? Type yes to continue: ");
      prompt.close();
      if (answer.toLowerCase() !== "yes") {
        throw new Error("Action rejected by the user.");
      }
    }

    console.log(`Processing computer call (ID: ${callId})`);

    // Handle the action and get the screenshot info
    const [screenshotInfo, updatedState] = handleComputerActionAndTakeScreenshot(
      action,
      currentState,
      screenshots,
    );
    currentState = updatedState;

    console.log(`Sending action result back to agent (using ${screenshotInfo.filename})...`);
    // Regular response with just the screenshot
    response = await openai.responses.create(
      {
        previous_response_id: response.id,
        input: [
          {
            call_id: callId,
            type: "computer_call_output",
            acknowledged_safety_checks: safetyChecks,
            output: {
              type: "computer_screenshot",
              image_url: screenshotInfo.url,
            },
          },
        ],
        truncation: "auto",
      },
      {
        body: { agent_reference: { name: agent.name, type: "agent_reference" } },
      },
    );

    console.log(`Follow-up response received (ID: ${response.id})`);
  }

  if (iteration >= maxIterations) {
    console.log(`\nReached maximum iterations (${maxIterations}). Stopping.`);
  }

  // Clean up resources
  console.log("\nCleaning up...");
  await project.agents.deleteVersion(agent.name, agent.version);
  console.log("Agent deleted");

  console.log("\nComputer Use Agent sample completed!");
}

main().catch((err) => {
  console.error("The sample encountered an error:", err);
});

Förväntade utdata

I följande exempel visas förväntade utdata när du kör föregående kodexempel:

Successfully loaded screenshot assets
Creating Computer Use Agent...
Agent created (id: ..., name: ComputerUseAgent, version: 1)
Starting computer automation session (initial screenshot: cua_browser_search.png)...
Initial response received (ID: ...)
--- Iteration 1 ---
Processing computer call (ID: ...)
  Typing text "OpenAI news" - Simulating keyboard input
  -> Action processed: type
Sending action result back to agent (using cua_search_typed.png)...
Follow-up response received (ID: ...)
--- Iteration 2 ---
Processing computer call (ID: ...)
    Click at (512, 384) - Simulating click on UI element
    -> Assuming click on Search button when search field was populated, displaying results.
    -> Action processed: click
Sending action result back to agent (using cua_search_results.png)...
Follow-up response received (ID: ...)
OpenAI news - Latest Updates
Cleaning up...
Agent deleted
Computer Use Agent sample completed!

Använd datoranvändning i en Java-agent

Lägg till beroendet till din pom.xml.

<dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-ai-agents</artifactId>
    <version>2.4.0</version>
</dependency>

Skapa en datoranvändningsagent

import com.azure.ai.agents.AgentsClient;
import com.azure.ai.agents.AgentsClientBuilder;
import com.azure.ai.agents.ResponsesClient;
import com.azure.ai.agents.models.*;
import com.azure.identity.DefaultAzureCredentialBuilder;
import com.openai.models.responses.Response;
import com.openai.models.responses.ResponseCreateParams;

import java.util.Collections;

public class ComputerUseExample {
    // Format: "https://resource_name.ai.azure.com/api/projects/project_name"
    private static final String PROJECT_ENDPOINT = "your_project_endpoint";

    public static void main(String[] args) {

        AgentsClientBuilder builder = new AgentsClientBuilder()
            .credential(new DefaultAzureCredentialBuilder().build())
            .endpoint(PROJECT_ENDPOINT);

        AgentsClient agentsClient = builder.buildAgentsClient();
        ResponsesClient responsesClient = builder.buildResponsesClient();

        // Create computer use tool
        ComputerUsePreviewTool tool = new ComputerUsePreviewTool(
            ComputerEnvironment.WINDOWS,
            1024,
            768
        );

        // Create agent with computer use tool
        PromptAgentDefinition agentDefinition = new PromptAgentDefinition("computer-use-preview")
            .setInstructions("You are a computer automation assistant.")
            .setTools(Collections.singletonList(tool));

        AgentVersionDetails agent = agentsClient.createAgentVersion("computer-use-agent", agentDefinition);
        System.out.printf("Agent created: %s (version %s)%n", agent.getName(), agent.getVersion());

        // Create a response with initial screenshot
        AgentReference agentReference = new AgentReference(agent.getName())
            .setVersion(agent.getVersion());

        Response response = responsesClient.createAzureResponse(
            new AzureCreateResponseOptions().setAgentReference(agentReference),
            ResponseCreateParams.builder()
                .input("Open the browser and navigate to microsoft.com"));

        System.out.println("Response: " + response.output());

        // The response will contain computer_call items with actions
        // to execute. Process each action, take screenshots, and
        // send results back using responsesClient.createAzureResponse()
        // with the previousResponseId and computer call output.

        // Clean up
        agentsClient.deleteAgentVersion(agent.getName(), agent.getVersion());
    }
}

För den fullständiga simulerade loopen använder du det underhållna ComputerUseSync.java exemplet med hjälpverktyget ComputerUseUtil.java. Hjälpen mappar begärda åtgärder till förinspelade skärmbilder. Ersätt simuleringen med programmets åtgärdsexekutor, skärmbildsavbildning och säkerhetsgodkännandeflöde.

Använda datorn genom REST API:et

Hämta en åtkomsttoken:

export AGENT_TOKEN=$(az account get-access-token --scope "https://ai.azure.com/.default" --query accessToken -o tsv)

Skapa en agent med datoranvändning

curl -X POST "$FOUNDRY_PROJECT_ENDPOINT/agents?api-version=v1" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AGENT_TOKEN" \
  -d '{
    "name": "computer-use-agent",
    "definition": {
      "kind": "prompt",
      "model": "computer-use-preview",
      "instructions": "You are a computer automation assistant.",
      "tools": [
        {
          "type": "computer_use_preview",
          "environment": "windows",
          "display_width": 1024,
          "display_height": 768
        }
      ]
    }
  }'

Generera ett svar

curl -X POST "$FOUNDRY_PROJECT_ENDPOINT/openai/v1/responses" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AGENT_TOKEN" \
  -d '{
    "agent_reference": {"type": "agent_reference", "name": "computer-use-agent"},
    "input": "Open the browser and navigate to microsoft.com"
  }'

Svaret innehåller computer_call utdataobjekt med åtgärder som ska köras. Innan du utför en åtgärd, kontrollera pending_safety_checks. Om matrisen inte är tom pausar du och visar åtgärden och säkerhetskontrollerna för slutanvändaren. Fortsätt först när användaren uttryckligen har godkänt åtgärden.

Skicka åtgärdsresultat med skärmbild

När användaren har godkänt eventuella väntande säkerhetskontroller och programmet kör datoråtgärden tar du en skärmbild och skickar tillbaka den. Inkludera varje godkänd kontroll i acknowledged_safety_checks. Om inga kontroller returnerades använder du en tom matris.

curl -X POST "$FOUNDRY_PROJECT_ENDPOINT/openai/v1/responses" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AGENT_TOKEN" \
  -d '{
    "agent_reference": {"type": "agent_reference", "name": "computer-use-agent"},
    "previous_response_id": "<RESPONSE_ID>",
    "input": [
      {
        "type": "computer_call_output",
        "call_id": "<CALL_ID>",
        "acknowledged_safety_checks": [],
        "output": {
          "type": "computer_screenshot",
          "image_url": "data:image/png;base64,<BASE64_SCREENSHOT>"
        }
      }
    ]
  }'

Ersätt <RESPONSE_ID>, <CALL_ID>och <BASE64_SCREENSHOT> med värden från föregående svar. Upprepa den här cykeln tills modellen returnerar ett textsvar i stället för en computer_call.

Rensa

curl -X DELETE "$FOUNDRY_PROJECT_ENDPOINT/agents/computer-use-agent?api-version=v1" \
  -H "Authorization: Bearer $AGENT_TOKEN"