Temporal Relation Modeling and Multimodal Adversarial Alignment Network for Pilot Workload Evaluation

Abstract

Pilots face complex working environments during flight missions, which can easily lead to excessive workload and affect flight safety. Physiological signals are commonly used to evaluate a pilot’s workload because they are objective and can directly reflect physiological mental states. However, existing methods have shortcomings in temporal modeling, making it challenging to fully capture the dynamic characteristics of physiological signals over time. Moreover, fusing features of data from different modalities is also difficult.To address these problems, we proposed a temporal relation modeling and multimodal adversarial alignment network (TRM-MAAN) for pilot workload evaluation. Specifically, a Transformer-based temporal relationship modeling module was used to learn complex temporal relationships for better feature extraction. In addition, an adversarial alignment-based multi-modal fusion module was applied to capture and integrate multi-modal information, reducing distribution shifts between different modalities. The performance of the proposed TRM-MAAN method was evaluated via experiments of classifying three workload states using electroencephalogram (EEG) and electromyography (EMG) recordings of eight healthy pilots.Experimental results showed that the classification accuracy and F1 score of the proposed method were significantly better than the baseline model across different subjects, with an average recognition accuracy of $91.90~\pm ~1.72\%$ and an F1 score of $91.86~\pm ~1.75\%$ .This work provides essential technical support for improving the accuracy and robustness of pilot workload evaluation and introduces a promising way for enhancing flight safety, offering broad application prospects. Clinical and Translational Impact Statement: The proposed scheme provides a promising solution for workload evaluation based on electrophysiological signals, with potential applications in aiding the clinical monitoring of fatigue, mental status, cognitive psychology, and other disorders.