Integrating infrared facial thermal imaging and tabular data for multimodal prediction of occupants' thermal sensation

Abstract

Developing robust thermal comfort models is essential for occupant-centric control (OCC) to optimize the indoor thermal environment while minimizing energy consumption. Conventional single-modal machine learning models, relying solely on either tabular or image data, often suffer from limited prediction accuracy and versatility. To address these challenges, this study proposes a multimodal framework that integrates both data types. A dataset of 610 paired records, encompassing environmental data, individual attributes, thermal sensation votes (TSV), and occupants’ facial thermal images, was collected. Separate single-modal models were trained on tabular and image data to identify the best-performing model for each modality. These were subsequently integrated using a self-attention mechanism to develop a unified multimodal predictive model. Results demonstrate that the artificial neural network (ANN), utilizing only tabular data, achieved an accuracy of 69.67% without incorporating temperature variables from facial regions of interest (ROIs), increasing to 72.46% when these variables were included. Conversely, the Inception-V3 model, trained solely on facial thermal images, achieved 63.44% accuracy. By integrating these approaches, the ANN+Inception-V3 multimodal model achieved a significantly improved accuracy of 81.48%, effectively capturing interaction effects from both data types. This study presents a robust framework and methodological reference for advancing multimodal thermal comfort prediction models, enabling scalable, personalized, and energy-efficient management strategies for indoor environments.