Augmented Reality for the Visually Impaired: Navigation Aid and Scene Semantics for Indoor Use Cases

With the Augmented Reality (AR) technology avail-able today, it is quite feasible to accommodate the needs of the visually impaired (VI) via AR. In this paper, a framework is introduced to help the VI navigate and explore unfamiliar indoor environments. In contrast to commonly used AR applications focused on visual augmentation, the proposed framework em-ploys auditory three-dimensional feedback (A3DF) for guiding the VI. Concretely, the current framework reads the pose of the user and helps the VI reach a target location via A3DF. The A3DF is implemented with the Unity game engine to provide the optimal user experience. After acquiring the environment mesh (EM), the optimal path from the user's location to the target location is calculated, while avoiding obstacles using Unity's navigation system. Moreover, the user is provided with semantic information about the unknown environment whilst exploring via auditory information. This framework is implemented on Microsoft HoloLens 2 and tested at an office environment with different locations of interest. Additionally, this framework potentially accelerates the learning curve since the user can be trained using Unity's simulation environment. Lastly, given different design parameters of the framework, the proposed method can be fine-tuned to fit the specific needs of the individual VI.