Multi-Reference Evaluation of Dynamic Video Summaries Using Granule-Aware F-Measure

A novel measure to evaluate a dynamic video summary against multiple reference summaries is proposed in this paper. To this end, concepts of rough set and granular computing are leveraged to theoretically design the measure that captures the inherent (dis)agreement among the multiple references and the resulting clustering tendency. In our design, multiple F-measures are used to represent the similarities between the dynamic video summary being evaluated and the multiple references. The clustering tendency among the multiple references induces granulation, which allows the computation of degrees of appropriateness of the multiple F-measures. These degrees of appropriateness are then used to combine the multiple F-measures resulting in our novel measure, which we refer to as the granule-aware F-measure or the GF-measure. Along with a few attributes of our proposed evaluation measure, it is theoretically shown that the average F-measure is a special case of our GF-measure. Two specific GF-measures called the GF(mad)-measure and GF(sat)-measure corresponding to judicious parameter choices are also discussed. Experiments including statistical, correlation and user studies are performed on the GF-measure to demonstrate its significance, distinguishing it from the popular average and maximum F-measures. The experiments are performed on summaries generated by multiple dynamic video summarization approaches for videos from a few standard datasets.