Development and Mobile Deployment of a Stair Recognition System for Human–Robot Locomotion

Abstract

Environment sensing and recognition can improve the safety and autonomy of human-robot locomotion, especially during transitions between environmental states such as walking to and from stairs. However, accurate and real-time perception on edge devices with limited computational resources is an open problem. Here we present the development and mobile deployment of StairNet, a vision-based stair recognition system powered by deep learning. Building on ExoNet, the largest open-source dataset of egocentric images of real-world walking environments, we designed a new dataset for stair recognition with over 515,000 images. We trained a lightweight and efficient convolutional neural network for image classification, which predicted complex stair environments with 98.4% accuracy. We also studied different model compression optimization methods and deployed our system on several mobile devices running a custom-designed iOS application with onboard accelerators using CPU, GPU, and/or NPU backend computing. Of the designs that we studied, our highest performing system showed negligible reductions in classification accuracy due to model conversion for mobile deployment and achieved an inference time of 2.75 ms. The high speed and accuracy of StairNet on edge devices opens new opportunities for environment-adaptive control of robotic prosthetic legs, exoskeletons, and other assistive technologies for human locomotion.