fNIRS Based Multi-Class Mental Workload Classification Using Recurrence Plots and CNN-LSTM

Abstract

The ever-increasing human-machine interaction and advancement in socio-technical systems have made it essential to analyze the vital human factors such as mental workload, vigilance, fatigue, stress, etc., via monitoring brain states. Similarly, brain signals are becoming paramount for rehabilitation and assistive purposes in fields such as brain- computer interface (BCI), closed-loop neuromodulation for neurological disorders, etc. The complex, non-stationary, and very low signal-to-noise ratio of brain signals poses a significant challenge for researchers to design robust and reliable BCI systems outside the laboratory environment. In this study, we present novel recurrence plots (RPs) based on convolutional neural network and long short term memory (CNN-LSTM) algorithm for four class functional near-infrared spectroscopy (fNIRS) BCI. The acquired brain signals are projected into a non-linear dimension with RPs and fed into the CNN, which extracts the important features. Then LSTM learns the chronological and time-dependent relations. The average accuracy achieved with the proposed model is 77.7%, while the maximum accuracy is 85.9%. The results confirm the viability of RPs based deep learning algorithms for successful BCI systems.