Gain valuable insights from your video and audio files

Automatically extract metadata—such as spoken words, written text, faces, speakers, celebrities, emotions, topics, brands and scenes from video and audio files. Access the data within your application or infrastructure, make it more discoverable, and use it to create new over-the-top (OTT) experiences and monetisation opportunities.

Features

The following list shows the insights you can retrieve from your videos using Video Indexer video and audio models:

Video insights

• Face detection: Detects and groups faces appearing in the video.
• Celebrity identification: Video Indexer automatically identifies over 1 million celebrities—like world leaders, actors, actresses, athletes, researchers, business, and tech leaders across the globe. The data about these celebrities can also be found on various websites (IMDB, Wikipedia, and so on).
• Account-based face identification: Video Indexer trains a model for a specific account. It then recognizes faces in the video based on the trained model. For more information, see Customize a Person model from the Video Indexer website and Customize a Person model with the Video Indexer API.
• Thumbnail extraction for faces("best face"): Automatically identifies the best captured face in each group of faces (based on quality, size, and frontal position) and extracts it as an image asset.
• Visual text recognition(OCR): Extracts text that's visually displayed in the video.
• Visual content moderation: Detects adult and/or racy visuals.
• Labels identification: Identifies visual objects and actions displayed.
• Scene segmentation: Determines when a scene changes in video based on visual cues. A scene depicts a single event and it's composed by a series of consecutive shots, which are semantically related.
• Shot detection: Determines when a shot changes in video based on visual cues. A shot is a series of frames taken from the same motion-picture camera. For more information, see Scenes, shots, and keyframes.
• Black frame detection: Identifies black frames presented in the video.
• Keyframe extraction: Detects stable keyframes in a video.
• Rolling credits: Identifies the beginning and end of the rolling credits in the end of TV shows and movies.
• Animated characters detection(preview): Detection, grouping, and recognition of characters in animated content via integration with Cognitive Services custom vision. For more information, see Animated character detection.
• Editorial shot type detection: Tagging shots based on their type (like wide shot, medium shot, close up, extreme close up, two shot, multiple people, outdoor and indoor, and so on). For more information, see Editorial shot type detection.

Audio insights

• Automatic language detection: Automatically identifies the dominant spoken language. Supported languages include English, Spanish, French, German, Italian, Chinese (Simplified), Japanese, Russian, and Brazilian Portuguese. If the language can't be identified with confidence, Video Indexer assumes the spoken language is English. For more information, see Language identification model.
• Multi-language speech identification and transcription(preview): Automatically identifies the spoken language in different segments from audio. It sends each segment of the media file to be transcribed and then combines the transcription back to one unified transcription. For more information, see Automatically identify and transcribe multi-language content.
• Audio transcription: Converts speech to text in 12 languages and allows extensions. Supported languages include English, Spanish, French, German, Italian, Chinese (Simplified), Japanese, Arabic, Russian, Brazilian Portuguese, Hindi, and Korean.
• Closed captioning: Creates closed captioning in three formats: VTT, TTML, SRT.
• Two channel processing: Auto detects separate transcript and merges to single timeline.
• Noise reduction: Clears up telephony audio or noisy recordings (based on Skype filters).
• Transcript customization(CRIS): Trains custom speech to text models to create industry-specific transcripts. For more information, see Customize a Language model from the Video Indexer website and Customize a Language model with the Video Indexer APIs.
• Speaker enumeration: Maps and understands which speaker spoke which words and when.
• Speaker statistics: Provides statistics for speakers' speech ratios.
• Textual content moderation: Detects explicit text in the audio transcript.
• Audio effects: Identifies audio effects like hand claps, speech, and silence.
• Emotion detection: Identifies emotions based on speech (what's being said) and voice tonality (how it's being said). The emotion could be joy, sadness, anger, or fear.
• Translation: Creates translations of the audio transcript to 54 different languages.

Audio and video insights (multi-channels)

When indexing by one channel, partial result for those models will be available.

• Keywords extraction: Extracts keywords from speech and visual text.
• Named entities extraction: Extracts brands, locations, and people from speech and visual text via natural language processing (NLP).
• Topic inference: Makes inference of main topics from transcripts. The 2nd-level IPTC taxonomy is included.
• Artifacts: Extracts rich set of "next level of details" artifacts for each of the models.
• Sentiment analysis: Identifies positive, negative, and neutral sentiments from speech and visual text.