Connecting and Controlling Appliances Through Wearable Augmented Reality

Abstract

The number of interconnected devices around us is constantly growing, and it may become challenging for users to control all these devices when control interfaces are distributed over mechanical elements, apps, and configuration webpages. If devices are even supposed to be connected and work together, this challenge is intensified. An alternative way for configuring and controlling devices in situ is enabled by wearable technologies. In this paper, we investigate interaction methods for smart appliances in augmented reality from an egocentric perspective. We examine how users can control appliances through augmented reality directly. The physical objects are augmented with interaction widgets, which are generated on demand and represent the connected devices along with their adjustable parameters. For example, a widget can be overlaid on a loudspeaker to control its volume. We explore three ways of manipulating the virtual widgets: (1) in-air finger pinching and sliding, (2) whole-arm gestures rotating and waving, (3) incorporating physical objects in the surrounding and mapping their movements to the interaction primitives. We compare these methods in a user study with 25 participants and find significant differences in the preference of the users, the speed of executing commands, and the granularity of the type of control. While these methods only apply to controlling a single device at a time, in a second part, we create a method to also take potential connections between devices into account. Users can view and configure connections between smart devices in augmented reality and furthermore can manipulate them or create new device connections using simple gestures. This facilitates the understanding of existing connections and their modification.