Automatic Video Summarization Using HEVC and CNN Features

Abstract

With the incredible surge of the internet and surveillance footage, there is a vast number of digital videos. The need to summarize these videos within databases is very crucial. This is where video summarization comes in handy. Video summarization can be achieved by a number of techniques. This study proposes a novel solution for the detection of key-frames for static video summarization. We preprocessed the well-known video datasets by coding them using the HEVC video coding standard. During coding, 64 proposed features were generated from the coder for each frame. Additionally, we extracted RGB frames from the original raw videos and fed them into pre-trained CNN networks for feature extraction. These include GoogleNet, AlexNet, Inception-ResNet-v2, and VGG16. The modified datasets are made publicly available to the research community. A subset of the proposed HEVC feature set was used to identify duplicate or similar frames and eliminate them from the video. We also propose an elimination solution based on the sum of the absolute differences between a frame and its motion-compensated predecessor. The proposed solutions are compared with existing works based on an SIFT flow algorithm that uses CNN features. Subsequently, an optional dimensionality reduction based on stepwise regression was applied to the feature vectors prior to detecting key-frames. The proposed solution is compared with existing studies that use sparse autoencoders with CNN features for dimensionality reduction. The accuracy of the proposed key-frame detection system was assessed using the Positive Predictive Values, Sensitivity, and F-score metrics. Combining the proposed solution with Multi-CNN features and using a Random Forests classifier, it was shown that the proposed solution achieved an average F-score of 0.98.