Armand K. Koupai, Mohammud J. Bocus, Raul Santos-Rodriguez, Robert J. Piechocki, Ryan McConville
{"title":"Self-supervised multimodal fusion transformer for passive activity recognition","authors":"Armand K. Koupai, Mohammud J. Bocus, Raul Santos-Rodriguez, Robert J. Piechocki, Ryan McConville","doi":"10.1049/wss2.12044","DOIUrl":null,"url":null,"abstract":"<p>The pervasiveness of Wi-Fi signals provides significant opportunities for human sensing and activity recognition in fields such as healthcare. The sensors most commonly used for passive Wi-Fi sensing are based on passive Wi-Fi radar (PWR) and channel state information (CSI) data, however current systems do not effectively exploit the information acquired through multiple sensors to recognise the different activities. In this study, new properties of the Transformer architecture for multimodal sensor fusion are explored. Different signal processing techniques are used to extract multiple image-based features from PWR and CSI data such as spectrograms, scalograms and Markov transition field (MTF). The Fusion Transformer, an attention-based model for multimodal and multi-sensor fusion is first proposed. Experimental results show that the Fusion Transformer approach can achieve competitive results compared to a ResNet architecture but with much fewer resources. To further improve the model, a simple and effective framework for multimodal and multi-sensor self-supervised learning (SSL) is proposed. The self-supervised Fusion Transformer outperforms the baselines, achieving a macro F1-score of 95.9%. Finally, this study shows how this approach significantly outperforms the others when trained with as little as 1% (2 min) of labelled training data to 20% (40 min) of labelled training data.</p>","PeriodicalId":51726,"journal":{"name":"IET Wireless Sensor Systems","volume":null,"pages":null},"PeriodicalIF":1.5000,"publicationDate":"2022-11-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ietresearch.onlinelibrary.wiley.com/doi/epdf/10.1049/wss2.12044","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"IET Wireless Sensor Systems","FirstCategoryId":"1085","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1049/wss2.12044","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"TELECOMMUNICATIONS","Score":null,"Total":0}
引用次数: 0
Abstract
The pervasiveness of Wi-Fi signals provides significant opportunities for human sensing and activity recognition in fields such as healthcare. The sensors most commonly used for passive Wi-Fi sensing are based on passive Wi-Fi radar (PWR) and channel state information (CSI) data, however current systems do not effectively exploit the information acquired through multiple sensors to recognise the different activities. In this study, new properties of the Transformer architecture for multimodal sensor fusion are explored. Different signal processing techniques are used to extract multiple image-based features from PWR and CSI data such as spectrograms, scalograms and Markov transition field (MTF). The Fusion Transformer, an attention-based model for multimodal and multi-sensor fusion is first proposed. Experimental results show that the Fusion Transformer approach can achieve competitive results compared to a ResNet architecture but with much fewer resources. To further improve the model, a simple and effective framework for multimodal and multi-sensor self-supervised learning (SSL) is proposed. The self-supervised Fusion Transformer outperforms the baselines, achieving a macro F1-score of 95.9%. Finally, this study shows how this approach significantly outperforms the others when trained with as little as 1% (2 min) of labelled training data to 20% (40 min) of labelled training data.
期刊介绍:
IET Wireless Sensor Systems is aimed at the growing field of wireless sensor networks and distributed systems, which has been expanding rapidly in recent years and is evolving into a multi-billion dollar industry. The Journal has been launched to give a platform to researchers and academics in the field and is intended to cover the research, engineering, technological developments, innovative deployment of distributed sensor and actuator systems. Topics covered include, but are not limited to theoretical developments of: Innovative Architectures for Smart Sensors;Nano Sensors and Actuators Unstructured Networking; Cooperative and Clustering Distributed Sensors; Data Fusion for Distributed Sensors; Distributed Intelligence in Distributed Sensors; Energy Harvesting for and Lifetime of Smart Sensors and Actuators; Cross-Layer Design and Layer Optimisation in Distributed Sensors; Security, Trust and Dependability of Distributed Sensors. The Journal also covers; Innovative Services and Applications for: Monitoring: Health, Traffic, Weather and Toxins; Surveillance: Target Tracking and Localization; Observation: Global Resources and Geological Activities (Earth, Forest, Mines, Underwater); Industrial Applications of Distributed Sensors in Green and Agile Manufacturing; Sensor and RFID Applications of the Internet-of-Things ("IoT"); Smart Metering; Machine-to-Machine Communications.