Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Document Parsing uses state-of-the-art research techniques to extract and visualize structured data from a wide range of document types, including but not limited to PDFs, images, Word documents (DOC/DOCX), and PowerPoint files (PPT/PPTX). It's designed to handle complex layouts such as tables, charts, and mixed text-image content.
Document parsing is available through the Agent Bricks UI, in SQL, with the REST API, and with the Asynchronous API (Beta).
Create a document parsing agent in the UI
Requirements
- A workspace that includes the following:
- Serverless compute enabled. See Serverless compute requirements.
- Unity Catalog enabled. See Enable a workspace for Unity Catalog.
- Access to a serverless usage policy with a nonzero budget.
- This function is only available in some regions, see AI function availability.
- The
ai_parse_documentfunction is also available to workspaces with the Enhanced Security and Compliance add-on.
- The
Use Document Parsing to parse your documents and visualize their structure.
- Go to
Agents in the left navigation pane of your workspace.
- Click Create Agent > Document Parsing.
- Select your source document. You can choose to upload a file or select one from an existing Unity Catalog catalog. Supported formats include: PDF, images, DOC/DOCX, and PPT/PPTX.
- Click Parse document.
Parsing your document can take a few minutes. When complete, Document Parsing shows the source document on the left and the parsed document on the right. You can choose to view the parsed document as Formatted text or Raw JSON.

Process and query results
To view the ai_parse_document query and run it on more documents, click Use Agent and choose either to run the query from the SQL Editor or Notebook. You can edit the query to point to the volume or table your documents live in.
To prepare the parsed output for retrieval (RAG), use ai_prep_search (Beta) downstream.
Parse documents in SQL
Call ai_parse_document in SQL to parse documents at scale:
SELECT
ai_parse_document(
content,
map('version', '2.0')
) AS parsed
FROM READ_FILES('/Volumes/my_catalog/my_schema/my_documents', format => 'binaryFile');
See the ai_parse_document SQL reference.
Parse documents with the REST API
Use the REST API to parse documents from applications, services, and automation workflows. See the REST API docs for reference.
ai_parse_document REST API requests are limited to 100 pages per document.
The default rate limit for ai_parse_document REST API requests is 250 requests per minute per workspace. Contact your Databricks account team to request a higher limit.
ai_parse_document reads its input from a Unity Catalog volume. The following example uploads a local PDF to a volume with the Files API, then parses it.
First, upload the document to a volume:
curl -X PUT "$DATABRICKS_HOST/api/2.0/fs/files/Volumes/my_catalog/my_schema/my_volume/invoice.pdf?overwrite=true" \
-H "Authorization: Bearer $DATABRICKS_TOKEN" \
-H "Content-Type: application/octet-stream" \
--data-binary @invoice.pdf
Then parse the uploaded document by passing its volume path as content:
curl -X POST "$DATABRICKS_HOST/api/2.0/ai-functions/ai-parse-document" \
-H "Authorization: Bearer $DATABRICKS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"content": "/Volumes/my_catalog/my_schema/my_volume/invoice.pdf"
}'
Parse documents with the Asynchronous API (Beta)
For batch processing, Databricks recommends AI Function tasks in Lakeflow Jobs. To parse documents in a job, use an ai_parse_document task, which does the following:
- Runs the processing in the background and writes the results to a Unity Catalog table, so you don't submit batches, check their status, or download output files.
- Processes only the documents that are new or changed since the previous run.
- Shows the progress of each run, and how many inputs succeeded and failed.
Use the AI Functions Asynchronous API (Beta) instead when you want to integrate the processing into an existing application or an external orchestrator, and manage the submissions and results yourself.
Important
This feature is in Beta. To use it, a workspace admin must turn on AI Functions Asynchronous API from the Previews page. See Manage Azure Databricks previews.
For the request and response fields, and for the requests to cancel and list batches, see the Asynchronous API reference.
Input directory or file
The input of an ai_parse_document batch is one of the following:
- A directory in a Unity Catalog volume. The batch parses every file in the directory and in its subdirectories, so put only the documents to parse there. The API makes one request for each file, with the path of the file as its
custom_id. - A JSONL input file in a volume that lists the documents to parse. Use an input file to parse only some of the documents in a directory, or to give each document your own
custom_id.
Each line of an input file is one request, a JSON object with exactly the following fields:
| Field | Description |
|---|---|
custom_id |
A string that identifies the request. Each line of the output file has the custom_id of its request, so use a value that you can match the results to. |
body |
The document to parse, in the form {"path": "/Volumes/..."}, where path is the volume path of the document. |
The following input file has two requests:
{"custom_id": "invoice-001", "body": {"path": "/Volumes/my_catalog/my_schema/my_volume/invoices/invoice-001.pdf"}}
{"custom_id": "invoice-002", "body": {"path": "/Volumes/my_catalog/my_schema/my_volume/invoices/invoice-002.pdf"}}
ai_parse_document needs no parameters other than the document. To set function options, such as version or pageRange, put them in options in params when you create the batch, for example "params": {"options": {"version": "2.0"}}. Write each option name in camelCase, for example imageOutputPath rather than image_output_path. The options apply to every document in the batch. For what each option does, see the AI Parse Document REST API reference.
When you create a batch, the API checks the input and params first. If the input or params breaks one of the following rules, the request fails with HTTP 400 and no batch is created:
- A directory has from 1 to 50,000 files.
- To process more documents, split them into several directories or input files and create a batch for each.
- An input file is UTF-8 encoded, and each line is one JSON object with exactly the fields
custom_idandbody. The file has no blank lines. - Each
custom_idis a non-empty string, and no two lines have the samecustom_id. - Each
bodyhas exactly one key,path.- If a
pathvalue isn't a string, the batch is still created, and that request fails with the error typeinvalid_request.
- If a
- An input file has from 1 to 50,000 lines and is at most 200 MiB.
paramshas no parameters thatai_parse_documentdoesn't take.
For a problem in a line of the input file, the error message gives the line number and the rule, for example Batch input line 3 custom_id must be unique. If you can't read the input or a volume that a path names, or can't write to output_ref, the request fails with HTTP 403.
The API lists the directory, or copies the input file, when it creates the batch. Files that you add to the directory afterward aren't part of the batch.
Run a batch
To parse documents with the Asynchronous API:
Upload the documents to a directory in a Unity Catalog volume. To parse only some of them, also create an input file that lists them, and upload it to a volume.
Create a batch that runs
ai_parse_document. Setinput_refto the directory or to the input file, andoutput_refto a volume directory or a new file path for the output file. The following example parses a directory:curl -X POST "$DATABRICKS_HOST/api/2.0/ai-functions/batches" \ -H "Authorization: Bearer $DATABRICKS_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "endpoint": "ai_parse_document", "input_ref": "/Volumes/my_catalog/my_schema/my_volume/invoices/", "output_ref": "/Volumes/my_catalog/my_schema/my_volume/output/" }'The response contains the batch
name. Use it in place of<batch-name>in the next steps. By default, a workspace can have up to 20 unfinished batches at a time, across all users. If it's at the limit, the request fails with HTTP 409. Create the batch again after another batch finishes or is cancelled.Check the status of the batch. The batch is finished when
statusisCOMPLETED,CANCELLED, orFAILED:curl -X GET "$DATABRICKS_HOST/api/2.0/ai-functions/batches/<batch-name>" \ -H "Authorization: Bearer $DATABRICKS_TOKEN"When the output file is ready,
output_refin the response contains its path.Download the output file with the Files API:
curl -X GET "$DATABRICKS_HOST/api/2.0/fs/files/Volumes/my_catalog/my_schema/my_volume/output/<batch-name>_output.jsonl" \ -H "Authorization: Bearer $DATABRICKS_TOKEN" \ -o output.jsonl
Output file
The Asynchronous API writes the output file of an ai_parse_document batch to the location in output_ref:
- If
output_refends with/or names an existing directory, the API writes the output file to<batch-name>_output.jsonlin that directory. Several batches can share one output directory. - Otherwise,
output_refis the path of the output file. The file must not exist when you create the batch.
The output file has one line for each document. Each line has the custom_id of its request and one of the following fields:
response: The result of the request. It has the same fields as the AI Parse Document REST API response.error: Thetypeandmessageof the error, if the request failed. If the batch was cancelled before the request ran, the type iscancelled.
For an input file, the lines are in the same order as the input file. For a directory, the custom_id of each line is the path of the document, and the lines can be in any order:
{"custom_id": "/Volumes/my_catalog/my_schema/my_volume/invoices/invoice-001.pdf", "response": {...}}
{"custom_id": "/Volumes/my_catalog/my_schema/my_volume/invoices/invoice-002.pdf", "error": {"type": "request_failed", "message": "..."}}