LiDAR + Camera Sensor Data Fusion On Mobiles With AI-based Virtual Sensors To Provide Situational Awareness For The Visually Impaired

Autonomy of the blind and visually impaired can be achieved through technological means and thereby empowering them with a sense of independence. Mobile phones are ubiquitous and can access artificial intelligence capabilities locally and in the Cloud. Navigational sensors, such as Light Detection and Ranging (LiDAR), and wide angle cameras, typically found in self-driving cars, are beginning to be incorporated into mobile phones. In this paper, we propose techniques for using mobile phone LiDAR + camera sensor data fusion along with edge + Cloud split AI to create an indoor situational awareness and navigational aid for the visually impaired. In addition to physical sensors, the system uses AI models as virtual sensors to provide the required functionality. The system enhances the image of a scene captured by a camera using distance information from the LiDAR and directional information computed by the device to provide a rich 3-D description of the space in front of the user. The system also uses a combination of sensor data fusion and geometric formulas to provide step-by-step walking instructions for the user in order to reach destinations. The user-centric system proposed here can be a valuable assistive technology for the blind and visually imnpired.