A 46.1 fps Global Matching Optical Flow Estimation Processor for Action Recognition in Mobile Devices

A real-time global matching optical flow estimation (OFE) processor is proposed for action recognition in mobile devices. The global OFE requires a large number of external memory accesses (EMAs) and matrix computations, thus it is incompatible on mobile devices with real-time constraints. For real-time OFE on mobile devices, this paper proposes two key features, both of which to reduce the required memory bandwidth and a number of computations: 1) Tile-based hierarchical OFE enables intermediate data to be processed within 328 KB on-chip memory without external memory access. 2) Background skipping eliminates redundant matrix computation for zero optical flow region. Therefore, the proposed features reduce external memory bandwidth and computation by 99.7 % and 50.7 %, respectively. The proposed 4 mm2 OFE processor is implemented in 65 nm CMOS technology, and it achieves real-time OFE of 46.1 frames-per-second (fps) throughput for an image resolution of QVGA (320×240) and the resulting optical flow can be successfully used for action recognition.