Minding the Gaps in a Video Action Analysis Pipeline

Abstract

We present an event detection system, which shares many similarities with standard object detection pipelines. It is composed of four modules: feature extraction, event proposal generation, event classification and event localization. We developed and assessed each module separately by evaluating several candidate options given oracle input using intermediate evaluation metric. This particular process results in a mismatch gap between training and testing when we integrate the module into the complete system pipeline. This results from the fact that each module is trained on clean oracle input, but during testing the module can only receive system generated input, which can be significantly different from the oracle data. Furthermore, we discovered that all the gaps between the different modules can contribute to a decrease in accuracy and they represent the major bottleneck for a system developed in this way. Fortunately, we were able to develop a set of relatively simple fixes in our final system to address and mitigate some of the gaps.