Agentic retrieval in Azure AI Search

In Azure AI Search, agentic retrieval is a multi-query pipeline designed for complex questions posed by users or agents in chat and copilot apps. It's intended for retrieval-augmented generation (RAG) patterns and agent-to-agent workflows.

Here's what agentic retrieval does:

• Can use LLM-based query planning (preview) to break down a complex query into smaller, focused subqueries for better coverage across proprietary and external content. Query planning can use chat history for additional context.

• Runs subqueries in parallel. Each subquery is semantically reranked to promote the most relevant matches.

• Combines the best results into a unified response that an LLM can use to generate grounded answers.

• Can return source references and an activity log alongside the merged content, so you can use just the grounding data or pass it to an LLM for a full answer.

This high-performance pipeline helps you generate high-quality grounding data or answers for your chat application, with the ability to answer complex questions quickly.

Why use agentic retrieval?

Agentic retrieval supports both managed and custom experiences for agents and apps. In the Microsoft Foundry portal, it powers Foundry IQ as a managed knowledge layer for agents. You can also build custom agentic retrieval solutions by using the Azure portal, Search Service REST API, or a supported Azure SDK.

Use agentic retrieval when you want to provide agents and apps with the most relevant content for answering harder questions, drawing on chat context, your proprietary content, and external sources.

Agentic retrieval adds latency compared to a single-query pipeline, but it handles query complexity that a single query can't. For example, it can handle:

• Questions with multiple asks, such as "find me a hotel near the beach, with airport transportation, and that's within walking distance of vegetarian restaurants."

• Questions that depend on earlier context in the conversation.

• Queries that benefit from rewriting, using synonym maps and LLM-generated paraphrasing to expand coverage across your content.

• Spelling mistakes.

Architecture and workflow

The agentic retrieval process works as follows:

1. Workflow initiation: Your application calls a knowledge base with a retrieve action that provides a query and conversation history.

2. Query planning: At low and medium retrieval reasoning effort, the knowledge base sends your query and conversation history to an LLM, which generates focused subqueries. At minimal effort, this step is skipped and queries are issued directly to knowledge sources. Reasoning effort defaults to low and is configured on the knowledge base.

3. Query execution: The knowledge base sends the subqueries to your knowledge sources. All subqueries run simultaneously and can be keyword, vector, or hybrid search. Each subquery undergoes semantic reranking to find the most relevant matches. References are extracted and retained for citation purposes.

4. Result synthesis: The system combines all results into a unified response. Merged content is always returned. Source references and an execution activity log are optional.

Components

For all agentic retrieval scenarios, a knowledge base and at least one knowledge source are required. Other components are optional and depend on your configuration.

Integration requirements

Your application drives the pipeline by calling the knowledge base and handling the response. The pipeline returns grounding data that you can pass to an LLM for answer generation or use directly in your conversation interface. For implementation details, see Tutorial: Build an end-to-end agentic retrieval solution.

Feature availability

Agentic retrieval supports both generally available and preview capabilities. Choose the Search Service REST API version that matches your agentic retrieval experience:

• Use the 2026-04-01 REST API for production workloads that use generally available knowledge source types with minimal, extractive retrieval.

• Use the 2026-08-01-preview REST API for preview knowledge source types and capabilities such as LLM-based query planning, answer synthesis, non-minimal retrieval reasoning effort, and multi-turn messages. See Azure AI Search preview terms.

The Azure portal and Microsoft Foundry portal provide preview-only access to all agentic retrieval capabilities. Objects created in either portal might use preview schemas and require migration when you move to the generally available REST API version. For a version-by-version breakdown and migration guidance, see Migrate agentic retrieval code to the latest version.

Region availability, limits, and billing

Before you use agentic retrieval, review its regional availability, service limits, and billing model.

Region availability

Agentic retrieval is available in select regions.

Limits

Knowledge sources and knowledge bases have maximum limits that vary by pricing tier and retrieval reasoning effort.

Billing

Agentic retrieval incurs charges from two services:

• Azure AI Search bills for retrieval tokens consumed during subquery execution and semantic ranking. The free plan (default) provides a monthly token allowance. The standard plan enables pay-as-you-go pricing after the free allowance is consumed. For more information, see Enable or disable agentic retrieval billing.

• Azure OpenAI bills for input and output tokens used in LLM-based query planning and answer synthesis (preview). Pricing is always pay-as-you-go and based on the model you assign to the knowledge base. Charges appear on your Azure OpenAI bill. For rates, see Azure OpenAI pricing.

The following table compares billing between the classic single-query pipeline and the agentic retrieval multi-query pipeline. In the classic pipeline, the billable component is semantic ranker.

Example: Estimate costs

This example helps illustrate the cost estimation process for query planning and query execution, but not answer synthesis. Your costs could be lower. For current rates, see Azure AI Search pricing and Azure OpenAI pricing.

To estimate the query plan costs as pay-as-you-go in Azure OpenAI, let's assume gpt-4o-mini:

• 15 cents for 1 million input tokens.
• 60 cents for 1 million output tokens.
• 2,000 input tokens for average chat conversation size.
• 350 tokens for average output plan size.

Estimated billing costs for query execution

To estimate agentic retrieval token counts, start with an idea of what an average document in your index looks like. For example, you might approximate:

• 10,000 chunks, where each chunk is one to two paragraphs of a PDF.
• 500 tokens per chunk.
• Each subquery reranks up to 50 chunks.
• On average, there are three subqueries per query plan.

Calculating price of execution

1. Assume we make 2,000 agentic retrievals with three subqueries per plan. This gives us about 6,000 total queries.

2. Rerank 50 chunks per subquery, which is 300,000 total chunks.

3. Average chunk is 500 tokens, so the total tokens for reranking is 150 million.

4. Given a hypothetical price of 0.022 per token, $3.30 is the total cost for reranking in US dollars.

5. Moving on to query plan costs: 2,000 input tokens multiplied by 2,000 agentic retrievals equal 4 million input tokens for a total of 60 cents.

6. Estimate the output costs based on an average of 350 tokens. If we multiply 350 by 2,000 agentic retrievals, we get 700,000 output tokens total for a total of 42 cents.

Putting it all together, you'd pay about $3.30 for agentic retrieval in Azure AI Search, 60 cents for input tokens in Azure OpenAI, and 42 cents for output tokens in Azure OpenAI, for $1.02 for query planning total. The combined cost for the full execution is $4.32.

Tips for controlling costs

• Review the activity log in the response to find out what queries were issued to which sources and the parameters used. You can reissue those queries against your indexes and use a public tokenizer to estimate tokens and compare to API-reported usage. Precise reconstruction of a query or response isn't guaranteed however. Factors include the type of knowledge source, such as public web data or a remote SharePoint knowledge source that's predicated on a user identity, which can affect query reproduction.

• Reduce the number of knowledge sources (indexes); consolidating content can lower fan-out and token volume.

• Lower the reasoning effort to reduce LLM usage during query planning and query expansion (iterative search).

• Organize content so the most relevant information can be found with fewer sources and documents (for example, curated summaries or tables).

How to get started

To create an agentic retrieval solution, you can use the Azure portal, Microsoft Foundry (new) portal, REST APIs, or an equivalent Azure SDK package.

Next step