NVIDIA NIM embeddings

The alpha agent-framework-nvidia package provides text embeddings through NVIDIA NIM. Use the hosted endpoint or set a base URL for a self-hosted NIM container.

Warning

Treat model or embedding output, and model-generated tool calls when supported, as untrusted application input. Validate them before code performs side effects or renders content. If you configure a custom endpoint, verify that you trust its URL and operator.

The agent-framework-nvidia integration documented on this page is available for Python. Use another .NET embedding generator for C# applications.

Install the package

pip install agent-framework-nvidia --pre

Configure the client

Set the API key and embedding model:

NVIDIA_API_KEY="<api-key>"
NVIDIA_EMBEDDING_MODEL="nvidia/nemotron-3-embed-1b"

NVIDIA_BASE_URL optionally replaces the hosted endpoint with a self-hosted NIM endpoint.

Create NvidiaEmbeddingClient, generate embeddings, and close the HTTP client that the instance created:

from agent_framework_nvidia import NvidiaEmbeddingClient

client = NvidiaEmbeddingClient()
try:
    embeddings = await client.get_embeddings(
        ["Agent Framework", "NVIDIA NIM"],
    )
    print(embeddings[0].dimensions)
finally:
    await client.close()

A preconfigured async_client remains caller-owned and stays open when NvidiaEmbeddingClient.close() runs.

Configure retrieval embeddings

Asymmetric retrieval models embed stored documents and search queries differently. Set input_type="passage" for indexed content and input_type="query" for search text. Use truncate to control over-long inputs:

await collection.upsert(
    notes,
    embeddings_options={
        "input_type": "passage",
        "truncate": "END",
    },
)

results = await collection.search(
    "How do I call a model hosted on NIM?",
    embeddings_options={"input_type": "query"},
)

input_type accepts "passage" or "query". truncate accepts "NONE", "START", or "END". Pass the same query options to create_vector_search_tool(..., embeddings_options=...) when an agent performs the search.

The hosted endpoint accepts at most 256 inputs and a 2-MiB request body. The client splits larger batches into consecutive requests and returns vectors in the original order. Model availability depends on the account associated with the API key.

For a complete retrieval example, see the NVIDIA embedding sample.

The agent-framework-nvidia integration documented on this page is available for Python. Use another Go embedding generator for Go applications.

Next steps

Vector stores

  • Sidst opdateret den