Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
These examples run large-scale batch inference on AI Runtime serverless GPUs. They cover a range of modalities, such as LLM text generation and speech-to-text, using frameworks like Ray Data, vLLM, and Transformers.
| Example | Interface | Description |
|---|---|---|
| Qwen2.5-32B batch inference with Ray Data and vLLM | Notebook | Run multilingual batch inference with eight persistent vLLM replicas of Qwen2.5-32B-Instruct on 8 H100 GPUs and save the results to Unity Catalog. |
| Batch speech-to-text with Whisper | Notebook | Transcribe audio in batches with OpenAI Whisper large-v3-turbo on serverless GPUs, using libraries pre-installed in AI Runtime. |
| Batch inference with Qwen2.5-7B-Instruct and vLLM | CLI | Use Ray Data and vLLM to generate responses on an 8x H100 node and write the results to a Unity Catalog volume as Parquet. |
| Audio transcription and diarization with Whisper and pyannote | CLI | Use Ray Data to distribute speech transcription and speaker diarization across a GPU cluster and write the results to a Unity Catalog volume as Parquet. |
| Protein embeddings with BioNeMo ESM-2 | CLI | Use Ray Data and BioNeMo ESM-2 to embed proteins across an 8x H100 node, then train and evaluate a linear membrane-protein classifier. |
| Batch image understanding with Qwen3-VL-8B and vLLM | CLI | Use Ray Data and vLLM to caption and classify images on an 8x H100 node and write structured results to Parquet. |
| Batch image classification with ViT and parallel writes | CLI | Classify pet photos with ViT-Base on one A10 GPU and save an annotated JPEG for each prediction to a Unity Catalog volume, using parallel writes so the GPU doesn't wait on storage. |
| Speculative decoding with SGLang and EAGLE-3 | CLI | Compare SGLang batch inference throughput for Qwen3-8B on one H100 GPU with and without an EAGLE-3 draft head for speculative decoding. |