IDIoT: Multimodal Framework for Ubiquitous Identification and Assignment of Human-carried Wearable Devices

Abstract

IoT (Internet of Things) devices, such as network-enabled wearables, are carried by increasingly more people throughout daily life. Information from multiple devices can be aggregated to gain insights into a person’s behavior or status. For example, an elderly care facility could monitor patients for falls by combining fitness bracelet data with video of the entire class. For this aggregated data to be useful to each person, we need a multi-modality association of the devices’ physical ID (i.e., location, the user holding it, visual appearance) with a virtual ID (e.g., IP address/available services). Existing approaches for multi-modality association often require intentional interaction or direct line-of-sight to the device, which is infeasible for a large number of users or when the device is obscured by clothing. We present IDIoT, a calibration-free passive sensing approach that fuses motion sensor information with camera footage of an area to estimate the body location of motion sensors carried by a user. We characterize results across three baselines to highlight how different fusing methodology results better than earlier IMU-vision fusion algorithms. From this characterization, we determine IDIoT is more robust to errors such as missing frames or miscalibration that frequently occur in IMU-vision matching systems.