Azure AI Video Indexer Bring Your Own (BYO) AI Model overview

Combine insights from other sources, including third-party, classification, and detection models, to produce a detailed analysis of your media data. You can use one or more of any models offered by Microsoft, an external custom model, or a customized Person, Brand, Speech, or Language model offered by Azure Video Indexer.

The feature is also available with Azure AI Video Indexer enabled by Arc.

Pricing

With the Video Indexer BYO model, you can add custom insights to video insight objects without incurring any extra costs beyond the listed cost of the indexing process. However, any costs related to the external environment and model shouldn't be considered part of Video Indexer's billing price. We strongly recommend reviewing our best practices section to optimize the external logic and reduce costs.

General workflow

1. Video is uploaded and indexed with Azure AI Video Indexer.
2. When the indexing process is completed, an event is created.
3. Your custom code listens to the event and starts the video post-processing process.
   1. Retrieve insights extracted by Azure AI Video Indexer.
   2. Get keyframes for a video section.
   3. Send the keyframe to the custom AI model.
   4. Patch the custom insights back to Video Indexer.

Prerequisites

Before you can start using the BYO model feature with Azure AI Video Indexer, you must:

1. Train or bring an external AI model that receives video assets and returns insights.
2. Create custom code that:
   1. Listens for Event Hubs events.
   2. Extracts the video id from the events.
   3. Retrieves the relevant assets by calling Azure AI Video Indexer APIs. In this scenario, request Get Video Index and Get frames SAS URLs.
   4. Sends the assets to the external AI model.
   5. Creates a JSON object based on the insights retrieved from the custom AI model.
   6. Requests Patch Update Video Index.

Schema

The values for populating the custom data are as follows:

Framerate

Azure AI Video Indexer supports one FPS for the Basic/Standard video level and four FPS for the advanced level. Higher frame rates aren't supported. You can optimize indexing by:

• Process only specific segments that interest you, such as frames that include a detected sound, object, or person.
• Sample a lower FPS, for example, every 5 seconds.

Frame selection

Use the skip frames and page size parameters for time selection. Multiply the skip frames value by the FPS, then add the page size value multiplied by the FPS to determine the time range.

URL: https://api.videoindexer.ai/{location}/Accounts/{accountId}/Videos/{videoId}/FramesFilePaths[?urlsLifetimeSeconds][&pageSize][&skip][&accessToken]

Parameters:

Response: FrameFilePathsResult

FrameFilePathData

Sample data sent from custom application in schema format

"customInsights": [
    {
        "Name": "tattoo",  
        "displayName": "Tattoo’s model",
        "displayType": "CapsuleAndTag",
        "Results": [   
            {   
                "id": 1,   
                "Type": "Dragon",   
                "WikiDataId": "57F",   
                "SubType": "Leg tattoo",   
                "Metadata": "",   
                "Instances": [
                    {
                        "Confidence": 0.49,
                        "AdjustedStart": "0:00:32.72", 
                        "AdjustedEnd": "0:00:42.72",
              ‾         "start": "0:00:32.72",
              ‾         "end": "0:00:42.72",
              ‾     }
            ‾   ]
          ‾ }
        ]
    }... 

Bring Your Own model samples

• Bring your own model samples

Related content

• Use the Azure AI Video Indexer API