A Low-cost Driver and Passenger Activity Detection System based on Deep Learning and Multiple Sensor Fusion

Abstract

There are many existing research efforts focusing on detecting the status of the driver, with a special focus on driver distraction. In addition, existing solutions require expensive hardware to detect various driver statues. This paper proposes a low-cost passenger activity detection system which uses common sensors in mobile devices. The proposed system detects various human activities (e.g., chatting, silence) and traffic environments (e.g., clear traffic, crowded) using three types of sensors. First, the human conversation can be recorded with the microphone to infer activities of each individual. To address the context of the audio, a CNN based deep learning model is developed. Second, outside traffic context information can be extracted from motion sensors and GPS. Lastly, we use data fusion to combine the multiple sensors data associated with human activity preferences and traffic environments. Meanwhile, we use data collected in real world environments to evaluate the accuracy and effectiveness of our system.