Multi-Modal Object Tracking using Dynamic Performance Metrics

Intelligent surveillance systems typically use a single visualspectrum modality for their input. These systems workwell in controlled conditions, but often fail when lightingis poor, or environmental effects such as shadows, dust orsmoke are present. Thermal spectrum imagery is not as susceptibleto environmental effects, however thermal imagingsensors are more sensitive to noise and they are onlygray scale, making distinguishing between objects difficult.Several approaches to combining the visual and thermalmodalities have been proposed, however they are limited byassuming that both modalities are perfuming equally well.When one modality fails, existing approaches are unable todetect the drop in performance and disregard the under performingmodality. In this paper, a novel middle fusion approachfor combining visual and thermal spectrum imagesfor object tracking is proposed. Motion and object detectionis performed on each modality and the object detectionresults for each modality are fused base on the currentperformance of each modality. Modality performance is determinedby comparing the number of objects tracked by thesystem with the number detected by each mode, with a smallallowance made for objects entering and exiting the scene.The tracking performance of the proposed fusion schemeis compared with performance of the visual and thermalmodes individually, and a baseline middle fusion scheme.Improvement in tracking performance using the proposedfusion approach is demonstrated. The proposed approachis also shown to be able to detect the failure of an individualmodality and disregard its results, ensuring performance isnot degraded in such situations.