Mobile Touch-Free Interaction for Global Health

Health workers in remote settings are increasingly using mobile devices to assist with a range of medical tasks that may require them to handle potentially infectious biological material, and touching their mobile device in these scenarios is undesirable or potentially harmful. To overcome this challenge, we present Maestro, a software-only gesture detection system that enables touch-free interaction on commodity mobile devices. Maestro uses the built-in, forward-facing camera on the device and computer vision to recognize users' in-air gestures. Our key design criteria are high gesture recognition rates and low power consumption. We describe Maestro's design and implementation and show that the system is able to detect and respond to users' gestures in real-time with acceptable energy consumption and memory overheads. We also evaluate Maestro through a controlled user study that provides insight into the performance of touch-free interaction, finding that participants were able to make gestures quickly and accurately enough to be useful for a variety of motivating global health applications. Finally, we describe the programming effort required to integrate touch-free interaction into several open-source mobile applications so that it can be used on commodity devices without requiring changes to the operating system. Taken together, our findings suggest that Maestro is a simple and practical tool that could allow health workers in remote settings to interact with their devices touch-free in demanding settings.