NailSense: fingertip force as a new input modality

Abstract

In this paper, we propose a new interaction technique, called NailSense, which allows users to control a mobile device by hovering and slightly bending/extending fingers behind the device. NailSense provides basic interactions equivalent to that of touchscreen interactions; 2-D locations and binary states (i.e., touch or released) are tracked and used for input, but without any need of touching on the screen. The proposed technique tracks the user's fingertip in real-time and triggers event on color change in the fingernail area. It works with conventional smartphone cameras, which means no additional hardware is needed for its utilization. This novel technique allows users to use mobile devices without occlusion which was a crucial problem in touchscreens, also promising extended interaction space in the air, on desktop, or in everywhere. This new interaction technique is tested with example applications: a drawing app and a web browser.