Scale AI Search endpoint throughput with high QPS

By default, standard endpoints support 20–200 QPS depending on index size. Real-time applications such as search bars, recommendation systems, and entity matching often require 100–1000+ QPS. On standard endpoints only, you can set a target QPS. Databricks provisions the infrastructure to best match that throughput level (best-effort, not guaranteed).

Important

Setting a target QPS provisions additional capacity, which increases the cost of the endpoint. You are charged for this additional capacity regardless of actual query traffic. Throughput scaling is best-effort and not guaranteed.

Use high QPS when:

  • Your application requires more than 50 QPS of sustained throughput.
  • You receive 429 (Too Many Requests) errors under normal load.
  • Latency degrades as traffic ramps up, even when average utilization appears low.

Requirements

  • High QPS is available for standard endpoints only. Storage-optimized endpoints are not supported.
  • Use service principal OAuth authentication and the index URL for high-QPS production workloads. Personal access tokens (PATs) and the workspace query URL are appropriate for prototyping, but they do not use the optimized query route and are capped at a few tens of QPS.
  • Delta Sync indexes that use managed embedding models can't use the optimized query route if the workspace has the IP access list feature enabled or has Public Network Access set to Disabled, which limits front-end access to Azure Private Link.
    • Using Azure Private Link with public network access enabled doesn't affect the route.
    • When either condition applies, queries to the index use the workspace query URL, which is capped at a few tens of QPS, so the endpoint can't reach a higher target QPS. This applies to all queries to the index, including queries that pass query_vector.
    • To reach your target QPS, use self-managed embeddings: a Delta Sync index with an embedding vector column, or a Direct Vector Access index. See options for providing embeddings. If you can't use self-managed embeddings, contact your Azure Databricks account team.

Configure target QPS

Set a target QPS when creating a new endpoint or updating an existing one. The additional capacity needed to best match the target throughput is provisioned automatically. Throughput scaling is best-effort and not guaranteed: actual QPS depends on your index size, vector dimensionality, query complexity, and filter usage.

Databricks UI

When creating a new endpoint:

  1. In the left sidebar, click Compute.

  2. Click the AI Search tab and click Create endpoint.

    Create AI Search compute.

  3. Under Advanced Settings, enter the Target QPS value.

    Create AI Search endpoint dialog.

When updating an existing endpoint:

  1. Navigate to the endpoint detail page.

  2. In the right panel, click the pencil icon Pencil icon. next to Target QPS.

    Edit target QPS.

  3. Enter the new value and click Save.

    Enter target QPS value.

While the change is in progress, the endpoint page shows its progress under Target QPS. See Track scaling progress.

Python SDK

from databricks.ai_search.client import AISearchClient

client = AISearchClient()

# Create a new endpoint with target QPS
endpoint = client.create_endpoint(
    name="my-high-qps-endpoint",
    endpoint_type="STANDARD",
    target_qps=500,
)

# Update an existing endpoint's target QPS
client.update_endpoint(name="my-endpoint", target_qps=500)

# Check scaling status
scaling_info = client.get_endpoint("my-endpoint").get("scaling_info", {})
print(f"Requested target QPS: {scaling_info.get('requested_target_qps')}")
print(f"State: {scaling_info.get('state')}")
# State is "SCALING_CHANGE_IN_PROGRESS" while capacity is being provisioned,
# then transitions to "SCALING_CHANGE_APPLIED".
# While the change is in progress, scaling_info also reports its progress.
# See "Track scaling progress" below.

REST API

Create an endpoint with target QPS:

POST /api/2.0/vector-search/endpoints
{
  "name": "my-high-qps-endpoint",
  "endpoint_type": "STANDARD",
  "target_qps": 500
}

Update target QPS on an existing endpoint:

PATCH /api/2.0/vector-search/endpoints/<ENDPOINT_NAME>
{
  "target_qps": 500
}

Check scaling status:

GET /api/2.0/vector-search/endpoints/<ENDPOINT_NAME>

The response scaling_info field shows the requested_target_qps and scaling state. The state is SCALING_CHANGE_IN_PROGRESS while capacity is being provisioned, then transitions to SCALING_CHANGE_APPLIED. To track a change that's in progress, see Track scaling progress.

Set up a service principal

High-QPS queries need an OAuth token with authorization_details for the index. Use a service principal for production. To set up the service principal:

  1. Add a service principal and assign it to the workspace that contains the index.

    These steps use an Azure Databricks managed service principal. For a Microsoft Entra ID managed service principal, see Authenticate with Microsoft Entra service principals.

  2. Create an OAuth secret for the service principal:

    • Follow the steps in Create an OAuth secret.
    • Under Scopes, select the narrowest scopes that your client supports. A token can't exceed its secret's scopes.
      • Run load test on the AI Search endpoint page, the load test notebook, and databricks-ai-search 0.81 and above: Select vector-search, plus model-serving if the index uses a Databricks-managed embedding model. These clients request all-apis first and fall back to narrower scopes if the secret doesn't allow it. To make the SDK request exactly the scopes you choose, pass them in the oauth_scopes parameter of AISearchClient, for example oauth_scopes=["vector-search", "model-serving"]. If the index embeds through a Unity Catalog model service, also select ai-gateway and include it in oauth_scopes if you set that parameter. The notebook and the SDK fall back to scopes that include ai-gateway, but Run load test on the endpoint page doesn't request it, so use a secret with all-apis for that page.
      • Earlier versions of databricks-ai-search and the REST token request in How to query an AI Search index: Select all APIs. These clients request the all-apis scope, so a secret limited to narrower scopes can't get a token for them. This scope makes the secret broad, so use a dedicated service principal that has only the grants on this page.
    • Store the client ID and secret in a secret scope instead of in your application code.
  3. Grant the service principal the following Unity Catalog privileges:

    • USE CATALOG on the catalog that contains the index.
    • USE SCHEMA on the schema that contains the index.
    • SELECT on the index.
    • SELECT and MODIFY on the index, if the service principal upserts or deletes data in a Direct Vector Access index.

    If the service principal owns the index, it doesn't need SELECT or MODIFY, but it still needs USE CATALOG and USE SCHEMA.

  4. For an index that uses a Databricks-managed embedding model, grant permissions on the embedding model serving endpoint and, if the index sets model_endpoint_name_for_query, on that endpoint too. The required permission depends on the endpoint type:

    • Pay-per-token Foundation Model APIs endpoints, such as databricks-gte-large-en: No grant is needed, unless your account restricts foundation models with Unity Catalog permissions, in which case grant EXECUTE on the model.
    • Provisioned throughput and custom serving endpoints: Grant Can Query to query with query_text. Grant at least Can View even if you only query with query_vector and use the Python SDK, because get_index() reads these serving endpoints and fails without permission.

Querying an index doesn't require any permission on the AI Search endpoint itself.

The service principal authenticates with OAuth machine-to-machine (M2M) authorization. The OAuth token must include authorization_details that allow ReadVectorIndex on the index, or WriteVectorIndex for upserts and deletes.

If the embedding model serving endpoint or the model_endpoint_name_for_query endpoint is route-optimized, the token also needs a workspace_permission entry that allows query_inference_endpoint on each one. To check whether a serving endpoint is route-optimized, see the route_optimized field returned by GET /api/2.0/serving-endpoints/<name>.

  • Python SDK: Pass the client ID and secret to AISearchClient, as shown in Query the index URL. The SDK requests a token with the required authorization_details and adds the serving endpoint entries automatically. For a route-optimized model_endpoint_name_for_query endpoint, this requires databricks-ai-search 0.79 and above.
  • REST API: Request the token yourself. See the service principal example in the REST API tab of How to query an AI Search index.

For testing, you can also use a token obtained through the notebook token exchange. Use a service principal for production. To query a Delta Sync index that uses a Databricks-managed embedding model with query_text, use a service principal token, because the embedding model serving endpoint might reject a token from the notebook token exchange with a 403 error.

Note

Privilege changes apply only to newly issued tokens. Revoking a privilege doesn't invalidate a token that was already issued. That token stays valid until it expires, which is up to one hour. With the Python SDK, call get_index() again after you change privileges or the embedding serving endpoint's configuration. REST callers must request a new token.

Query the index URL

After the endpoint scaling state is SCALING_CHANGE_APPLIED, send queries to the index URL using an OAuth token with authorization_details for the index, such as a service principal token. This URL is required to use the additional query capacity provisioned by target_qps.

For Python applications, call get_index() one time and reuse the returned index object. The Python SDK sends queries to the index URL. To query an index from a Databricks app, see Add an AI Search index resource to a Databricks app.

from databricks.ai_search.client import AISearchClient

client = AISearchClient(
    service_principal_client_id="...",
    service_principal_client_secret="...",
    workspace_url="https://<workspace-url>",
)

index = client.get_index(endpoint_name="my-high-qps-endpoint", index_name="catalog.schema.index")

# Reuse this index object for every query.
index.similarity_search(query_vector=[...], columns=["id", "text"], num_results=10)

For REST or non-Python applications, first get the index URL, then send query requests to that URL. The token must be an OAuth token with authorization_details for the index. Use a service principal for production.

export WORKSPACE_URL=https://<workspace-url>
export INDEX_NAME=catalog.schema.index
export TOKEN=<oauth-token>

export INDEX_URL=$(curl -X GET \
  -H "Authorization: Bearer $TOKEN" \
  "$WORKSPACE_URL/api/2.0/vector-search/indexes/$INDEX_NAME" \
  | jq -r '.status.index_url')

case "$INDEX_URL" in
  http://*|https://*) ;;
  *) INDEX_URL="https://$INDEX_URL" ;;
esac

curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  "$INDEX_URL/query" \
  --data '{"num_results": 10, "query_vector": [...], "columns": ["id", "text"]}'

Do not use the workspace query URL, such as /api/2.0/vector-search/indexes/<index_name>/query, for high-QPS production traffic. That URL does not use the optimized query route and might return 429 errors before the endpoint reaches the configured target QPS.

How scaling applies

After you set a target QPS, the required capacity is provisioned automatically. The new throughput level applies after provisioning completes. You don't need to sync indexes to trigger the change.

Provisioning can take from minutes to hours, depending on the index size and how much the target QPS changes. To track progress, check the endpoint's scaling_info.state until it is SCALING_CHANGE_APPLIED. While the change is in progress, scaling_info also shows an estimated completion time and the current phase. See Track scaling progress.

An endpoint state of ONLINE doesn't mean its indexes are ready to query. When you create the first index on an endpoint, the index stays in PROVISIONING_ENDPOINT while the endpoint's compute is created. Check the index's status.ready and status.detailed_state before you send traffic.

Note

Attempting to update target QPS while a scaling operation is in progress returns a RESOURCE_CONFLICT error. Wait for the current operation to complete before retrying.

Track scaling progress

While a target QPS scaling change is in progress (scaling_info.state is SCALING_CHANGE_IN_PROGRESS), the endpoint's scaling_info reports the estimated completion time, the current phase, the index initialization percentage, and any delay. These progress fields are in Beta.

Every progress field is optional. If a field is missing from the response, that information isn't available, so check for each field before you use it.

Field Description
estimated_completion_timestamp The estimated time the scaling change completes, in milliseconds since the Unix epoch. The estimate can change as scaling progresses. It isn't returned while the completion time is being estimated or while the change is delayed.
phase The current phase of the scaling change. See the phases below.
index_initialization_percentage The percentage of index initialization completed, from 0 to 100. Returned only in the INITIALIZING_INDEX phase, when progress can be measured.
delayed_reason Why the scaling change is delayed. When this field is set, estimated_completion_timestamp isn't returned. See the delay reasons below.
target_qps_change_allowed_after_timestamp The earliest time a new target QPS change is accepted, in milliseconds since the Unix epoch. Before this time, a target QPS update returns a RESOURCE_CONFLICT error. The value can be in the past.

The phase field has one of these values:

  • ESTIMATING: The completion time is being estimated, for example just after you request or update a change. This phase usually lasts a few minutes.
  • ADJUSTING_CAPACITY: Capacity is being added to or removed from the endpoint.
  • INITIALIZING_INDEX: The new capacity is in place, and index replicas are being initialized on it.

The delayed_reason field has one of these values:

  • CAPACITY_UNAVAILABLE: The additional capacity that the change needs isn't available.
  • SCALE_DOWN_BLOCKED: The endpoint can't scale down safely, so it stays at its current size.
  • TAKING_LONGER_THAN_EXPECTED: The change is taking longer than expected.

For large scaling changes, the estimated completion time might appear only after scaling has made measurable progress. If the estimated completion time passes while the state is still SCALING_CHANGE_IN_PROGRESS, keep checking until the state is SCALING_CHANGE_APPLIED.

In the UI, the endpoint page shows the estimated ready time under Target QPS, or Scaling is taking longer than expected when the change is delayed. To see the current phase and the index initialization percentage, hover over the in-progress icon next to the value.

The following example reads the progress fields with the Python SDK:

import datetime

from databricks.ai_search.client import AISearchClient

client = AISearchClient()

scaling_info = client.get_endpoint("my-endpoint").get("scaling_info", {})
if scaling_info.get("state") == "SCALING_CHANGE_IN_PROGRESS":
    eta_ms = scaling_info.get("estimated_completion_timestamp")
    if scaling_info.get("delayed_reason"):
        print(f"Delayed: {scaling_info['delayed_reason']}")
    elif eta_ms:
        eta = datetime.datetime.fromtimestamp(eta_ms / 1000, tz=datetime.timezone.utc)
        print(f"Phase: {scaling_info.get('phase')}, estimated completion: {eta.isoformat()}")
    else:
        print(f"Phase: {scaling_info.get('phase')}, no estimate yet")

Troubleshoot 429 errors

For high-QPS workloads, use these checks to find the bottleneck:

  • To see whether errors come from the embedding model endpoint or from AI Search, and the cause of each 429 error, check the endpoint metrics. See Troubleshooting.

  • If you use a PAT or the workspace query URL, switch to service principal OAuth authentication and the index URL.

  • If scaling_info.state is SCALING_CHANGE_IN_PROGRESS, wait until the state changes to SCALING_CHANGE_APPLIED.

  • If your application sends vector queries with query_vector, the embedding model is not in the query path. If 429 errors continue after scaling completes, reduce request concurrency or set a higher target_qps. Target QPS assumes vector queries that return about 10 results. Hybrid queries and larger num_results values use more capacity per query, so set a higher target_qps for them and load-test your workload.

  • If your application sends text queries to a Delta Sync index with Databricks-managed embedding models, the embedding model might be the bottleneck. Use a smaller embedding model, such as databricks-qwen3-embedding-0-6b, instead of databricks-gte-large-en, or use a provisioned throughput Foundation Model APIs endpoint or another dedicated Model Serving endpoint for embeddings.

    An index's embedding endpoints are set when you create the index, so to use a different embedding endpoint, such as a provisioned throughput endpoint, create a new index. When you create it, you can keep the ingestion embedding model and set only a separate query endpoint that serves the same model with model_endpoint_name_for_query.

Limitations

  • No autoscaling: You must set target QPS manually based on expected traffic. If traffic exceeds the provisioned level, 429 errors occur. See Plan for query spikes.
  • Standard endpoints only: Storage-optimized endpoints do not support target_qps.
  • Optimized route required: The configured target QPS applies to traffic that uses service principal OAuth authentication and the index URL. PAT traffic and workspace query URL traffic are capped at a few tens of QPS.
  • Managed embedding models can add a second limit: For Delta Sync indexes that use a managed embedding model for text queries, query throughput also depends on the embedding model serving endpoint. Target QPS doesn't change the capacity of that serving endpoint. Increase model serving capacity, use provisioned throughput, or use self-managed embeddings for predictable query throughput.