Real-time video analysis in Azure AI Video Indexer - Preview

Azure AI Video Indexer enabled by Arc (VI) provides real-time video analysis. It helps you extract insights from live video so that you can detect events and act at the edge. VI real-time video analysis includes built-in insights and support for custom AI insights that identify objects or states relevant to your scenario. You can view insights directly on your video stream, with bounding boxes that highlight detected objects. You can also save streams and insights as files, and upload and index external media files. Azure AI Video Indexer can also generate concise summaries for segments of recorded video so that you can catch up on key events without watching the entire recording.

Use the web portal or REST API to integrate Azure AI Video Indexer into your workflows and systems.

• You don't pay extra for real-time video analysis while it's in preview.
• Real-time analysis was validated on Azure Local but is compatible with any Kubernetes infrastructure.

What can I do with real-time analysis?

You can integrate VI real-time analysis with camera streams to use AI-based detection across locations. This service analyzes live video and turns raw footage into actionable insights.

Customer scenarios and use cases

• Retail: Optimize store layouts and improve customer experience and safety. By using real-time analysis, you can monitor the number of customers in checkout lines in real time, helping retailers optimize staffing and reduce wait times.

• Manufacturing: Ensure quality control and worker safety through video analysis. For example, identify workers who aren't wearing protective gear by using real-time detection of critical events and locating specific moments in video streams.

• Modern safety: Detect security and safety issues before they create risk.

Limitations

The following limitations apply to the real-time analysis feature:

Camera limitations

• Only static cameras are supported. PTZ cameras aren't supported.
• Cameras must support RTCP sender reports.
• Only continuous video streaming is supported. Motion-triggered video isn't supported.
• The frame rate should be between 28 and 32 FPS. All cameras connected to the same extension should have the same FPS.
• The minimum supported resolution is 640 x 480 pixels. The recommended resolution for optimal performance is 1,280 x 720 pixels.
• When you use ultra-high-resolution video (frame width equal to or greater than 3,840 pixels, or frame height equal to or greater than 2,160 pixels), the system automatically uses a solution that detects small objects and covers the whole frame. However, the recorded video is saved in lower resolution, Full HD (1,920 x 1,080 pixels), while preserving the aspect ratio. The system can't accept ultra-high-resolution video and lower-resolution video on the same extension.
• Color should be RGB.
• Fisheye lenses aren't supported.

Detection limitations

• The system detects only objects larger than 35 x 35 pixels.
• The detector might not detect objects in dark areas and in bad weather conditions. Extreme weather conditions can reduce results quality. For example, heavy rain and fog.
• Occlusions might reduce results quality and cause fragmentation in object tracking.
• The detector can miss or misclassify objects when viewed from an unusual point of view or extreme angles.
• The confidence score for a detection, as shown in the UI, represents its first appearance. Along the track, the confidence can change and is shown only in the API.

Hardware requirements

The following tables show the hardware requirements.

Minimum:

Recommended:

Azure AI Video Indexer requires at least one supported GPU for real-time analysis. You can add GPUs to increase the camera capacity of one extension. The total extension capacity is the aggregate capacity of its configured GPU-backed real-time analysis instances.

GPU capacity and queued cameras

The following camera capacity values apply to each GPU-backed real-time analysis instance:

Using ultra-high resolution might reduce the number of supported cameras per GPU.

• Supported GPUs are NVIDIA A2, A10, V100, A100, and H100.
• Camera models consume different streaming-unit weights from each GPU instance's configured capacity. When a camera requires more capacity than any GPU instance has available, the camera status is Queued. Video Indexer admits the camera automatically when capacity becomes available.
• Video Indexer returns an error if a camera model has an unknown or missing streaming-unit weight. It doesn't assign a default weight.
• The minimum required shared storage is 50 GB for one camera per day. This estimation is based on the scenario where both recordings and insights are saved.
• To upload video media files to the live extension, add another virtual machine (VM) without a GPU, as specified in Minimum hardware requirements for VI enabled by Arc.
• Using event summary for recorded media files requires an extra VM with GPU.

Minimum software requirements

You can see which Azure regions Azure AI Video Indexer is available on the regions page.

Related content

• Manage real-time analysis extensions
• AI Agents for real-time video analysis in Azure AI Video Indexer