Hands Tracking over Steering Wheel based on Multi-Bernoulli Filter Framework with Hand Landmarks Detector and Semantic Segmentation

Abstract

Car driver’s hands motion operating the steering-wheel in a car cabin has been tracked based on multi-Bernoulli filter framework with the aid of semantic segmentation for body region extraction and hand landmarks detection over camera image, according to the newly proposed method in this paper. Real-time implementation of the method has been proposed as a hybrid system of python and C/C++ languages with CUDA GPU computation for particle filtering, DeepLab v3 for semantic segmentation, and MediaPipe for hand landmarks detector. Experimental results demonstrate improved performance of hands tracking especially by newly incorporated part of the hand landmarks detector.