Enhanced On-Device Video Summarization Using Audio and Visual Features

Video Summarization is gaining popularity, as there is lot of content available. Video Summarization is about determining the key or primary moments in the video. Hence, the most important step in a video summarization technique is identifying the key moments. While most of the related works are based on generating video summaries using only visual features, few recent studies have explored a better approach that utilizes the audio of the video along with the visual features. While considering the use of audio, it is very crucial to understand the correlation between the visual and audio features. However, while dealing with a constrained environment like embedded devices and smartphones, one has to consider the limitations of the resources available as well. Though using audio is found to improve the summary generated of the video, the resources of a mobile environment are hard constraints and one has to leverage the resources wisely. In this paper, we present a novel approach which demonstrates a different method to make use of audio features along with visual cues in video summarization. The proposed approach uses different models for inferencing from audio and visual features and provides an exhaustive approach on engineering both the features intelligently. Using different neural network models for audio and visual features makes it feasible to use in a mobile environment.