Batch inference

These examples run large-scale batch inference on AI Runtime serverless GPUs. They cover a range of modalities, such as LLM text generation and speech-to-text, using frameworks like Ray Data, vLLM, and Transformers.

Example Interface Description
Qwen2.5-32B batch inference with Ray Data and vLLM Notebook Run multilingual batch inference with eight persistent vLLM replicas of Qwen2.5-32B-Instruct on 8 H100 GPUs and save the results to Unity Catalog.
Batch speech-to-text with Whisper Notebook Transcribe audio in batches with OpenAI Whisper large-v3-turbo on serverless GPUs, using libraries pre-installed in AI Runtime.
Batch inference with Qwen2.5-7B-Instruct and vLLM CLI Use Ray Data and vLLM to generate responses on an 8x H100 node and write the results to a Unity Catalog volume as Parquet.
Audio transcription and diarization with Whisper and pyannote CLI Use Ray Data to distribute speech transcription and speaker diarization across a GPU cluster and write the results to a Unity Catalog volume as Parquet.
Protein embeddings with BioNeMo ESM-2 CLI Use Ray Data and BioNeMo ESM-2 to embed proteins across an 8x H100 node, then train and evaluate a linear membrane-protein classifier.
Batch image understanding with Qwen3-VL-8B and vLLM CLI Use Ray Data and vLLM to caption and classify images on an 8x H100 node and write structured results to Parquet.
Batch image classification with ViT and parallel writes CLI Classify pet photos with ViT-Base on one A10 GPU and save an annotated JPEG for each prediction to a Unity Catalog volume, using parallel writes so the GPU doesn't wait on storage.
Speculative decoding with SGLang and EAGLE-3 CLI Compare SGLang batch inference throughput for Qwen3-8B on one H100 GPU with and without an EAGLE-3 draft head for speculative decoding.