Real-time Powered Wheelchair Assistive Navigation System Based on Intelligent Semantic Segmentation for Visually Impaired Users

Abstract

People with movement disabilities may find powered wheelchair driving a challenging task due to their comorbidities. Certain visually impaired persons with mobility disabilities are not prescribed a powered wheelchair because of their sight condition. However, powered wheelchairs are essential to the majority of these disabled users for commuting and social interaction. It is vital for their independence and wellbeing. In this paper, we propose to use a semantic segmentation (SS) system based on deep learning algorithms to provide environmental cues and information to visually impaired wheelchair users to aid with the navigation process. The system classifies the objects of the indoor environment and presents the annotated output on a display customised to the user's condition. The user can select a target object, for which the system can display the estimated distance from the current position of the wheelchair. The system runs in real-time, using a depth camera installed on the wheelchair, and it displays the scene in front of the wheelchair with every pixel annotated with distinguishable colour to represent the different components of the environment along with the distance to the target object. Our system has been designed, implemented and deployed on a real powered wheelchair for practical evaluation. The proposed system helped the users to estimate more accurately the distance to the target objects with a relative error of 19.8% and 18.4% for the conditions of a) semi-neglect and b) short-sightedness, respectively, compared to errors of 47.8% and 5.6% without the SS system. In our experiments, healthy participants were put in simulated conditions representing the above visual impairments using instruments commonly used in medical research for this purpose. Finally, our system helps to visualise, on the display, hidden areas of the environment and blind spots that visually impaired users would not be able to see without it.