Neural network based multi-label semantic video concept detection using novel mixed-hybrid-fusion approach

Abstract

The performance of the semantic concept detection method depends on, the selection of the low-level visual features used to represent key-frames of a shot and the selection of the feature-fusion method. This paper proposes a set of low-level visual features of considerably smaller size and also proposes novel 'hybrid-fusion' and 'mixed-hybrid-fusion' approaches which are formulated by combining contemporary early and late-fusion strategies. In the proposed hybrid-fusion approach, the features from the same feature group are combined using early-fusion before classifier training; and the concept probability scores from multiple classifiers are merged using late-fusion approach, to get final detection scores. A feature group is defined as the features from the same feature family like color moments. The hybrid-fusion approach is refined and the 'mixed-hybrid-fusion' approach is proposed additionally to further improve the detection rate. Neural Network is used to build classifiers that produce concept probabilities for a test frame. The proposed approaches are evaluated on TRECVID development dataset which contains multi-labeled key-frames. Results show that, the proposed approaches outperform early-fusion and late-fusion approaches by large margins with respect to feature set dimensionality and mean Average Precision (mAP) values.