Remote heart rate measurement based on video color magnification and spatiotemporal self-attention

Abstract

Remote photoplethysmography (rPPG) for heart rate measurement has garnered significant attention due to its non-contact advantages. The challenge in video-based remote heart rate measurement lies in accurately capturing subtle changes in facial color. We propose an end-to-end deep learning model named Video Color Magnification and Spatiotemporal Feature Extraction Network (VS-Net). VS-Net comprises three main modules: video color magnification, spatiotemporal self-attention feature extraction, and contrastive learning. The video color magnification module, implemented using a deep neural network, initially magnifies subtle facial color changes in the input video. The magnified color features are then fed into the spatiotemporal self-attention feature extraction module. This module utilizes a multi-head self-attention mechanism along with convolutional neural networks to locally and globally model information exchange across magnified video frames, capturing long-term dependencies and extracting spatiotemporal features. Additionally, the model incorporates a contrastive learning module designed to improve weak signal detection in facial videos. By generating positive and negative samples based on video frequency resampling, the model captures similarities and differences among input samples, thereby learning more robust semantic feature representations. Comprehensive experiments were conducted on three public datasets: UBFC-RPPG, PURE, and MAHNOB-HCI. The results demonstrate that VS-Net effectively extracts rPPG signals from facial videos and outperforms state-of-the-art methods in heart rate measurement.