Pub Date : 2024-09-11DOI: 10.1109/TNSRE.2024.3457830
Jianglai Yu;Lei Zhang;Dongzhou Cheng;Wenbo Huang;Hao Wu;Aiguo Song
Early-exiting has recently provided an ideal solution for accelerating activity inference by attaching internal classifiers to deep neural networks. It allows easy activity samples to be predicted at shallower layers, without executing deeper layers, hence leading to notable adaptiveness in terms of accuracy-speed trade-off under varying resource demands. However, prior most works typically optimize all the classifiers equally on all types of activity data. As a result, deeper classifiers will only see hard samples during test phase, which renders the model suboptimal due to the training-test data distribution mismatch. Such issue has been rarely explored in the context of activity recognition. In this paper, to close the gap, we propose to organize all these classifiers as a dynamic-depth network and jointly optimize them in a similar gradient-boosting manner. Specifically, a gradient-rescaling is employed to bound the gradients of parameters at different depths, that makes such training procedure more stable. Particularly, we perform a prediction reweighting to emphasize current deep classifier while weakening the ensemble of its previous classifiers, so as to relieve the shortage of training data at deeper classifiers. Comprehensive experiments on multiple HAR benchmarks including UCI-HAR, PAMAP2, UniMiB-SHAR, and USC-HAD verify that it is state-of-the-art in accuracy and speed. A real implementation is measured on an ARM-based mobile device.
最近，Early-exiting 通过在深度神经网络中附加内部分类器，为加速活动推理提供了一种理想的解决方案。它允许在较浅的层预测简单的活动样本，而无需执行较深的层，因此在不同的资源需求下，在准确性-速度权衡方面具有显著的适应性。然而，之前的大多数研究通常会在所有类型的活动数据上对所有分类器进行同等优化。因此，较深的分类器在测试阶段只能看到较难样本，这使得模型因训练-测试数据分布不匹配而无法达到最佳状态。在活动识别中，很少有人探讨过这个问题。在本文中，为了缩小这一差距，我们建议将所有这些分类器组织成一个动态深度网络，并以类似梯度提升的方式对它们进行联合优化。具体来说，我们采用梯度缩放来约束不同深度参数的梯度，从而使这种训练过程更加稳定。特别是，我们进行了预测重权，在强调当前深度分类器的同时，弱化了其之前分类器的集合，从而缓解了深度分类器训练数据的不足。在多个 HAR 基准（包括 UCI-HAR、PAMAP2、UniMiB-SHAR 和 USC-HAD）上进行的综合实验验证了它在准确性和速度方面的先进性。在基于 ARM 的移动设备上测量了实际实施情况。
{"title":"Improving Human Activity Recognition With Wearable Sensors Through BEE: Leveraging Early Exit and Gradient Boosting","authors":"Jianglai Yu;Lei Zhang;Dongzhou Cheng;Wenbo Huang;Hao Wu;Aiguo Song","doi":"10.1109/TNSRE.2024.3457830","DOIUrl":"10.1109/TNSRE.2024.3457830","url":null,"abstract":"Early-exiting has recently provided an ideal solution for accelerating activity inference by attaching internal classifiers to deep neural networks. It allows easy activity samples to be predicted at shallower layers, without executing deeper layers, hence leading to notable adaptiveness in terms of accuracy-speed trade-off under varying resource demands. However, prior most works typically optimize all the classifiers equally on all types of activity data. As a result, deeper classifiers will only see hard samples during test phase, which renders the model suboptimal due to the training-test data distribution mismatch. Such issue has been rarely explored in the context of activity recognition. In this paper, to close the gap, we propose to organize all these classifiers as a dynamic-depth network and jointly optimize them in a similar gradient-boosting manner. Specifically, a gradient-rescaling is employed to bound the gradients of parameters at different depths, that makes such training procedure more stable. Particularly, we perform a prediction reweighting to emphasize current deep classifier while weakening the ensemble of its previous classifiers, so as to relieve the shortage of training data at deeper classifiers. Comprehensive experiments on multiple HAR benchmarks including UCI-HAR, PAMAP2, UniMiB-SHAR, and USC-HAD verify that it is state-of-the-art in accuracy and speed. A real implementation is measured on an ARM-based mobile device.","PeriodicalId":13419,"journal":{"name":"IEEE Transactions on Neural Systems and Rehabilitation Engineering","volume":"32 ","pages":"3452-3464"},"PeriodicalIF":4.8,"publicationDate":"2024-09-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10676997","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142204861","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Post-stroke gait control is a complex, often fail to account for the heterogeneity and continuity of gait in existing gait models. Precisely evaluating gait speed adjustability and gait instability in free-living environments is important to understand how individuals with post-stroke gait dysfunction approach diverse environments and contexts. This study aimed to explore individual causal interactions in the free-living gait control of persons with stroke. To this end, fifty persons with stroke wore an accelerometer on the fifth lumbar vertebra (L5) for 24 h in a free-living environment. Individually directed acyclic graphs (DAGs) were generated based on the spatiotemporal gait parameters at contemporaneous and temporal points calculated from the acceleration data. Spectral clustering and Bayesian model comparison were used to characterize the DAGs. Finally, the DAG patterns were interpreted via Bayesian logistic analysis. Spectral clustering identified three optimal clusters from the DAGs. Cluster 1 included persons with moderate stroke who showed high gait asymmetry and gait instability and primarily adjusted gait speed based on cadence. Cluster 2 included individuals with mild stroke who primarily adjusted their gait speed based on step length. Cluster 3 comprised individuals with mild stroke who primarily adjusted their gait speed based on both step length and cadence. These three clusters could be accurately classified based on four variables: Ashman’s D for step velocity, Fugl-Meyer Assessment, step time asymmetry, and step length. The diverse DAG patterns of gait control identified suggest the heterogeneity of gait patterns and the functional diversity of persons with stroke. Understanding the theoretical interactions between gait functions will provide a foundation for highly tailored rehabilitation.
中风后步态控制是一个复杂的问题，现有的步态模型往往无法解释步态的异质性和连续性。精确评估自由生活环境中的步速可调性和步态不稳定性对于了解卒中后步态功能障碍患者如何面对不同环境和情境非常重要。本研究旨在探索中风患者在自由生活步态控制中的个体因果相互作用。为此，50 名中风患者在自由生活环境中，在第五腰椎（L5）上佩戴加速度计 24 小时。根据加速度数据计算出的同时点和时间点的时空步态参数生成了独立的有向无环图（DAG）。光谱聚类和贝叶斯模型比较被用来描述 DAG 的特征。最后，通过贝叶斯逻辑分析对 DAG 模式进行解释。光谱聚类从 DAG 中识别出三个最佳聚类。聚类 1 包括中度中风患者，他们表现出高度步态不对称和步态不稳定，主要根据步幅调整步速。第 2 组包括轻度中风患者，他们主要根据步长调整步速。第 3 组包括主要根据步长和步幅调整步速的轻度卒中患者。这三个群组可根据四个变量进行准确分类：步速的 Ashman's D、Fugl-Meyer 评估、步速时间不对称和步长。步态控制的不同 DAG 模式表明了步态模式的异质性和中风患者功能的多样性。了解步态功能之间的理论相互作用将为高度定制化康复奠定基础。
{"title":"Modeling the Heterogeneity of Post-Stroke Gait Control in Free-Living Environments Using a Personalized Causal Network","authors":"Yuki Nishi;Koki Ikuno;Yusaku Takamura;Yuji Minamikawa;Shu Morioka","doi":"10.1109/TNSRE.2024.3457770","DOIUrl":"10.1109/TNSRE.2024.3457770","url":null,"abstract":"Post-stroke gait control is a complex, often fail to account for the heterogeneity and continuity of gait in existing gait models. Precisely evaluating gait speed adjustability and gait instability in free-living environments is important to understand how individuals with post-stroke gait dysfunction approach diverse environments and contexts. This study aimed to explore individual causal interactions in the free-living gait control of persons with stroke. To this end, fifty persons with stroke wore an accelerometer on the fifth lumbar vertebra (L5) for 24 h in a free-living environment. Individually directed acyclic graphs (DAGs) were generated based on the spatiotemporal gait parameters at contemporaneous and temporal points calculated from the acceleration data. Spectral clustering and Bayesian model comparison were used to characterize the DAGs. Finally, the DAG patterns were interpreted via Bayesian logistic analysis. Spectral clustering identified three optimal clusters from the DAGs. Cluster 1 included persons with moderate stroke who showed high gait asymmetry and gait instability and primarily adjusted gait speed based on cadence. Cluster 2 included individuals with mild stroke who primarily adjusted their gait speed based on step length. Cluster 3 comprised individuals with mild stroke who primarily adjusted their gait speed based on both step length and cadence. These three clusters could be accurately classified based on four variables: Ashman’s D for step velocity, Fugl-Meyer Assessment, step time asymmetry, and step length. The diverse DAG patterns of gait control identified suggest the heterogeneity of gait patterns and the functional diversity of persons with stroke. Understanding the theoretical interactions between gait functions will provide a foundation for highly tailored rehabilitation.","PeriodicalId":13419,"journal":{"name":"IEEE Transactions on Neural Systems and Rehabilitation Engineering","volume":"32 ","pages":"3522-3530"},"PeriodicalIF":4.8,"publicationDate":"2024-09-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10677489","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142204862","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-09-11DOI: 10.1109/TNSRE.2024.3457820
Wenjuan Lu;Huiting Ma;Daxing Zeng
When applying continuous motion estimation (CME) model based on sEMG to human-robot system, it is inevitable to encounter scenarios in which the motions performed by the user are different from the motions in the training stage of the model. It has been demonstrated that the prediction accuracy of the currently effective approaches on untrained motions will be significantly reduced. Therefore, we proposed a novel CME method by introducing muscle synergy as feature to achieve better prediction accuracy on untrained motion tasks. Specifically, deep non-smooth NMF (Deep-nsNMF) was firstly introduced on synergy extraction to improve the efficiency of synergy decomposition. After obtaining the activation primitives from various training motions, we proposed a redundancy classification algorithm (RC) to identify shared and task-specific synergies, optimizing the original redundancy segmentation algorithm (RS). NARX neural network was set as the regression model for training. Finally, the model was tested on prediction tasks of eight untrained motions. The prediction accuracy of the proposed method was found to perform better than using time-domain feature as input of the network. Through Deep-nsNMF with RS, the highest accuracy reached 99.7%. Deep-nsNMF with RC performed similarly well and its stability was relatively higher among different motions and subjects. Limitation of the approach is that the problem of positive correlation between the prediction error and the absolute value of real angle remains to be further addressed. Generally, this research demonstrates the potential for CME models to perform well in complex scenarios, providing the feasibility of dedicating CME to real-world applications.
{"title":"Performance of a Novel Muscle Synergy Approach for Continuous Motion Estimation on Untrained Motion","authors":"Wenjuan Lu;Huiting Ma;Daxing Zeng","doi":"10.1109/TNSRE.2024.3457820","DOIUrl":"10.1109/TNSRE.2024.3457820","url":null,"abstract":"When applying continuous motion estimation (CME) model based on sEMG to human-robot system, it is inevitable to encounter scenarios in which the motions performed by the user are different from the motions in the training stage of the model. It has been demonstrated that the prediction accuracy of the currently effective approaches on untrained motions will be significantly reduced. Therefore, we proposed a novel CME method by introducing muscle synergy as feature to achieve better prediction accuracy on untrained motion tasks. Specifically, deep non-smooth NMF (Deep-nsNMF) was firstly introduced on synergy extraction to improve the efficiency of synergy decomposition. After obtaining the activation primitives from various training motions, we proposed a redundancy classification algorithm (RC) to identify shared and task-specific synergies, optimizing the original redundancy segmentation algorithm (RS). NARX neural network was set as the regression model for training. Finally, the model was tested on prediction tasks of eight untrained motions. The prediction accuracy of the proposed method was found to perform better than using time-domain feature as input of the network. Through Deep-nsNMF with RS, the highest accuracy reached 99.7%. Deep-nsNMF with RC performed similarly well and its stability was relatively higher among different motions and subjects. Limitation of the approach is that the problem of positive correlation between the prediction error and the absolute value of real angle remains to be further addressed. Generally, this research demonstrates the potential for CME models to perform well in complex scenarios, providing the feasibility of dedicating CME to real-world applications.","PeriodicalId":13419,"journal":{"name":"IEEE Transactions on Neural Systems and Rehabilitation Engineering","volume":"32 ","pages":"3554-3564"},"PeriodicalIF":4.8,"publicationDate":"2024-09-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10677007","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142204863","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-09-10DOI: 10.1109/TNSRE.2024.3457502
Li Hu;Wei Gao;Zilin Lu;Chun Shan;Haiwei Ma;Wenyu Zhang;Yuanqing Li
The calibration procedure for a wearable P300 brain-computer interface (BCI) greatly impact the user experience of the system. Each user needs to spend additional time establishing a decoder adapted to their own brainwaves. Therefore, achieving subject independent is an urgent issue for wearable P300 BCI needs to be addressed. A dataset of electroencephalogram (EEG) signals was constructed from 100 individuals by conducting a P300 speller task with a wearable EEG amplifier. A framework is proposed that initially improves cross- subject consistency of EEG features through a common feature extractor. Subsequently, a simple and compact convolutional neural network (CNN) architecture is employed to learn an embedding sub-space, where the mapped EEG features are maximally separated, while pursuing the minimum distance within the same class and the maximum distance between different classes. Finally, the model’s generalization capability was further optimized through fine-tuning. Results: The proposed method significantly boosts the average accuracy of wearable P300 BCI to $73.23pm 7.62$ % without calibration and $78.75pm 6.37$ % with fine-tuning. The results demonstrate the feasibility and excellent performance of our dataset and framework. A calibration-free wearable P300 BCI system is feasible, suggesting significant potential for practical applications of the wearable P300 BCI system.
{"title":"Subject-Independent Wearable P300 Brain–Computer Interface Based on Convolutional Neural Network and Metric Learning","authors":"Li Hu;Wei Gao;Zilin Lu;Chun Shan;Haiwei Ma;Wenyu Zhang;Yuanqing Li","doi":"10.1109/TNSRE.2024.3457502","DOIUrl":"10.1109/TNSRE.2024.3457502","url":null,"abstract":"The calibration procedure for a wearable P300 brain-computer interface (BCI) greatly impact the user experience of the system. Each user needs to spend additional time establishing a decoder adapted to their own brainwaves. Therefore, achieving subject independent is an urgent issue for wearable P300 BCI needs to be addressed. A dataset of electroencephalogram (EEG) signals was constructed from 100 individuals by conducting a P300 speller task with a wearable EEG amplifier. A framework is proposed that initially improves cross- subject consistency of EEG features through a common feature extractor. Subsequently, a simple and compact convolutional neural network (CNN) architecture is employed to learn an embedding sub-space, where the mapped EEG features are maximally separated, while pursuing the minimum distance within the same class and the maximum distance between different classes. Finally, the model’s generalization capability was further optimized through fine-tuning. Results: The proposed method significantly boosts the average accuracy of wearable P300 BCI to \u0000<inline-formula> <tex-math>$73.23pm 7.62$ </tex-math></inline-formula>\u0000% without calibration and \u0000<inline-formula> <tex-math>$78.75pm 6.37$ </tex-math></inline-formula>\u0000% with fine-tuning. The results demonstrate the feasibility and excellent performance of our dataset and framework. A calibration-free wearable P300 BCI system is feasible, suggesting significant potential for practical applications of the wearable P300 BCI system.","PeriodicalId":13419,"journal":{"name":"IEEE Transactions on Neural Systems and Rehabilitation Engineering","volume":"32 ","pages":"3543-3553"},"PeriodicalIF":4.8,"publicationDate":"2024-09-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10672519","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142204865","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Training an accurate classifier for EEG-based brain-computer interface (BCI) requires EEG data from a large number of users, whereas protecting their data privacy is a critical consideration. Federated learning (FL) is a promising solution to this challenge. This paper proposes Federated classification with local Batch-specific batch normalization and Sharpness-aware minimization (FedBS) for privacy protection in EEG-based motor imagery (MI) classification. FedBS utilizes local batch-specific batch normalization to reduce data discrepancies among different clients, and sharpness-aware minimization optimizer in local training to improve model generalization. Experiments on three public MI datasets using three popular deep learning models demonstrated that FedBS outperformed six state-of-the-art FL approaches. Remarkably, it also outperformed centralized training, which does not consider privacy protection at all. In summary, FedBS protects user EEG data privacy, enabling multiple BCI users to participate in large-scale machine learning model training, which in turn improves the BCI decoding accuracy.
为基于脑电图的脑机接口（BCI）训练一个准确的分类器需要大量用户的脑电图数据，而保护用户数据隐私是一个重要的考虑因素。联邦学习（FL）是应对这一挑战的一个有前途的解决方案。本文提出了在基于脑电图的运动图像（MI）分类中保护隐私的联邦分类法（Federated classification with local Batch-specific batch normalization and Sharpness-aware minimization，FedBS）。FedBS 利用本地特定批次归一化来减少不同客户端之间的数据差异，并在本地训练中使用锐度感知最小化优化器来提高模型泛化能力。使用三种流行的深度学习模型在三个公共 MI 数据集上进行的实验表明，FedBS 的性能优于六种最先进的 FL 方法。值得注意的是，它的表现也优于完全不考虑隐私保护的集中式训练。总之，FedBS 保护了用户脑电图数据的隐私，使多个 BCI 用户能够参与大规模机器学习模型的训练，从而提高了 BCI 解码的准确性。
{"title":"Federated Motor Imagery Classification for Privacy-Preserving Brain-Computer Interfaces","authors":"Tianwang Jia;Lubin Meng;Siyang Li;Jiajing Liu;Dongrui Wu","doi":"10.1109/TNSRE.2024.3457504","DOIUrl":"10.1109/TNSRE.2024.3457504","url":null,"abstract":"Training an accurate classifier for EEG-based brain-computer interface (BCI) requires EEG data from a large number of users, whereas protecting their data privacy is a critical consideration. Federated learning (FL) is a promising solution to this challenge. This paper proposes Federated classification with local Batch-specific batch normalization and Sharpness-aware minimization (FedBS) for privacy protection in EEG-based motor imagery (MI) classification. FedBS utilizes local batch-specific batch normalization to reduce data discrepancies among different clients, and sharpness-aware minimization optimizer in local training to improve model generalization. Experiments on three public MI datasets using three popular deep learning models demonstrated that FedBS outperformed six state-of-the-art FL approaches. Remarkably, it also outperformed centralized training, which does not consider privacy protection at all. In summary, FedBS protects user EEG data privacy, enabling multiple BCI users to participate in large-scale machine learning model training, which in turn improves the BCI decoding accuracy.","PeriodicalId":13419,"journal":{"name":"IEEE Transactions on Neural Systems and Rehabilitation Engineering","volume":"32 ","pages":"3442-3451"},"PeriodicalIF":4.8,"publicationDate":"2024-09-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10672548","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142204867","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Objective: Speech brain-computer interfaces (speech BCIs), which convert brain signals into spoken words or sentences, have demonstrated great potential for high-performance BCI communication. Phonemes are the basic pronunciation units. For monosyllabic languages such as Chinese Mandarin, where a word usually contains less than three phonemes, accurate decoding of phonemes plays a vital role. We found that in the neural representation space, phonemes with similar pronunciations are often inseparable, leading to confusion in phoneme classification. Methods: We mapped the neural signals of phoneme pronunciation into a hyperbolic space for a more distinct phoneme representation. Critically, we proposed a hyperbolic hierarchical clustering approach to specifically learn a phoneme-level structure to guide the representation. Results: We found such representation facilitated greater distance between similar phonemes, effectively reducing confusion. In the phoneme decoding task, our approach demonstrated an average accuracy of 75.21% for 21 phonemes and outperformed existing methods across different experimental days. Conclusion: Our approach showed high accuracy in phoneme classification. By learning the phoneme-level neural structure, the representations of neural signals were more discriminative and interpretable. Significance: Our approach can potentially facilitate high-performance speech BCIs for Chinese and other monosyllabic languages.
{"title":"Effective Phoneme Decoding With Hyperbolic Neural Networks for High-Performance Speech BCIs","authors":"Xianhan Tan;Qi Lian;Junming Zhu;Jianmin Zhang;Yueming Wang;Yu Qi","doi":"10.1109/TNSRE.2024.3457313","DOIUrl":"10.1109/TNSRE.2024.3457313","url":null,"abstract":"Objective: Speech brain-computer interfaces (speech BCIs), which convert brain signals into spoken words or sentences, have demonstrated great potential for high-performance BCI communication. Phonemes are the basic pronunciation units. For monosyllabic languages such as Chinese Mandarin, where a word usually contains less than three phonemes, accurate decoding of phonemes plays a vital role. We found that in the neural representation space, phonemes with similar pronunciations are often inseparable, leading to confusion in phoneme classification. Methods: We mapped the neural signals of phoneme pronunciation into a hyperbolic space for a more distinct phoneme representation. Critically, we proposed a hyperbolic hierarchical clustering approach to specifically learn a phoneme-level structure to guide the representation. Results: We found such representation facilitated greater distance between similar phonemes, effectively reducing confusion. In the phoneme decoding task, our approach demonstrated an average accuracy of 75.21% for 21 phonemes and outperformed existing methods across different experimental days. Conclusion: Our approach showed high accuracy in phoneme classification. By learning the phoneme-level neural structure, the representations of neural signals were more discriminative and interpretable. Significance: Our approach can potentially facilitate high-performance speech BCIs for Chinese and other monosyllabic languages.","PeriodicalId":13419,"journal":{"name":"IEEE Transactions on Neural Systems and Rehabilitation Engineering","volume":"32 ","pages":"3432-3441"},"PeriodicalIF":4.8,"publicationDate":"2024-09-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10672534","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142204868","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Affective data is the basis of emotion recognition, which is mainly acquired through extrinsic elicitation. To investigate the enhancing effects of multi-sensory stimuli on emotion elicitation and emotion recognition, we designed an experimental paradigm involving visual, auditory, and olfactory senses. A multimodal emotional dataset (OVPD-II) that employed the video-only or video-odor patterns as the stimuli materials, and recorded the electroencephalogram (EEG) and electrooculogram (EOG) signals, was created. The feedback results reported by subjects after each trial demonstrated that the video-odor pattern outperformed the video-only pattern in evoking individuals’ emotions. To further validate the efficiency of the video-odor pattern, the transformer was employed to perform the emotion recognition task, where the highest accuracy reached 86.65% (66.12%) for EEG (EOG) modality with the video-odor pattern, which improved by 1.42% (3.43%) compared with the video-only pattern. What’s more, the hybrid fusion (HF) method combined with the transformer and joint training was developed to improve the performance of the emotion recognition task, which achieved classify accuracies of 89.50% and 88.47% for the video-odor and video-only patterns, respectively.
{"title":"Multimodal Emotion Recognition Based on EEG and EOG Signals Evoked by the Video-Odor Stimuli","authors":"Minchao Wu;Wei Teng;Cunhang Fan;Shengbing Pei;Ping Li;Guanxiong Pei;Taihao Li;Wen Liang;Zhao Lv","doi":"10.1109/TNSRE.2024.3457580","DOIUrl":"10.1109/TNSRE.2024.3457580","url":null,"abstract":"Affective data is the basis of emotion recognition, which is mainly acquired through extrinsic elicitation. To investigate the enhancing effects of multi-sensory stimuli on emotion elicitation and emotion recognition, we designed an experimental paradigm involving visual, auditory, and olfactory senses. A multimodal emotional dataset (OVPD-II) that employed the video-only or video-odor patterns as the stimuli materials, and recorded the electroencephalogram (EEG) and electrooculogram (EOG) signals, was created. The feedback results reported by subjects after each trial demonstrated that the video-odor pattern outperformed the video-only pattern in evoking individuals’ emotions. To further validate the efficiency of the video-odor pattern, the transformer was employed to perform the emotion recognition task, where the highest accuracy reached 86.65% (66.12%) for EEG (EOG) modality with the video-odor pattern, which improved by 1.42% (3.43%) compared with the video-only pattern. What’s more, the hybrid fusion (HF) method combined with the transformer and joint training was developed to improve the performance of the emotion recognition task, which achieved classify accuracies of 89.50% and 88.47% for the video-odor and video-only patterns, respectively.","PeriodicalId":13419,"journal":{"name":"IEEE Transactions on Neural Systems and Rehabilitation Engineering","volume":"32 ","pages":"3496-3505"},"PeriodicalIF":4.8,"publicationDate":"2024-09-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10672559","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142204869","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-09-09DOI: 10.1109/TNSRE.2024.3456591
Mushfika Sultana;Serafeim Perdikis
Early brain-computer interface (BCI) systems were mainly based on prior neurophysiological knowledge coupled with feedback training, while state-of-the-art interfaces rely on data-driven, machine learning (ML)-oriented methods. Despite the advances in BCI that ML can be credited with, the performance of BCI solutions is still not up to the mark, posing a major barrier to the widespread use of this technology. This paper proposes a novel, automatic feature selection method for BCI able to leverage both data-dependent and expert knowledge to suppress noisy features and highlight the most relevant ones thanks to a fuzzy logic (FL) system. Our approach exploits the capability of FL to increase the reliability of decision-making by fusing heterogeneous information channels while maintaining transparency and simplicity. We show that our method leads to significant improvement in classification accuracy, feature stability and class bias when applied to large motor imagery or attempt datasets including end-users with motor disabilities. We postulate that combining data-driven methods with knowledge derived from neuroscience literature through FL can enhance the performance, explainability, and learnability of BCIs.
{"title":"Automatic Feature Selection for Sensorimotor Rhythms Brain-Computer Interface Fusing Expert and Data-Driven Knowledge","authors":"Mushfika Sultana;Serafeim Perdikis","doi":"10.1109/TNSRE.2024.3456591","DOIUrl":"10.1109/TNSRE.2024.3456591","url":null,"abstract":"Early brain-computer interface (BCI) systems were mainly based on prior neurophysiological knowledge coupled with feedback training, while state-of-the-art interfaces rely on data-driven, machine learning (ML)-oriented methods. Despite the advances in BCI that ML can be credited with, the performance of BCI solutions is still not up to the mark, posing a major barrier to the widespread use of this technology. This paper proposes a novel, automatic feature selection method for BCI able to leverage both data-dependent and expert knowledge to suppress noisy features and highlight the most relevant ones thanks to a fuzzy logic (FL) system. Our approach exploits the capability of FL to increase the reliability of decision-making by fusing heterogeneous information channels while maintaining transparency and simplicity. We show that our method leads to significant improvement in classification accuracy, feature stability and class bias when applied to large motor imagery or attempt datasets including end-users with motor disabilities. We postulate that combining data-driven methods with knowledge derived from neuroscience literature through FL can enhance the performance, explainability, and learnability of BCIs.","PeriodicalId":13419,"journal":{"name":"IEEE Transactions on Neural Systems and Rehabilitation Engineering","volume":"32 ","pages":"3422-3431"},"PeriodicalIF":4.8,"publicationDate":"2024-09-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10669390","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142204713","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-09-06DOI: 10.1109/TNSRE.2024.3455262
Bradley M. Cornish;Laura E. Diamond;David J. Saxby;Zhengliang Xia;Claudio Pizzolato
Neuromusculoskeletal (NMS) models enable non-invasive estimation of clinically important internal biomechanics. A critical part of NMS modelling is the estimation of musculotendon kinematics, which comprise musculotendon unit lengths, moment arms, and lines of action. Musculotendon kinematics, which are partially dependent on joint angles, define the non-linear mapping of muscle forces to joint moments and contact forces. Currently, real-time computation of musculotendon kinematics requires creation of a per-individual surrogate model. The computational speed and accuracy of these surrogates degrade with increasing number of coordinates. We developed a feed-forward neural network that completely encodes musculotendon kinematics of a target model across a wide anthropometric range, enabling accurate real-time estimates of musculotendon kinematics without need for a priori creation of a per-individual surrogate model. Compared to reference, the neural network had median normalized errors ~0.1% for musculotendon lengths, <0.4%> $1.23pm 0.15$ %) compared to using reference musculotendon kinematics. Finally, execution time was <0.04 ms per frame and constant for increasing number of model coordinates. Our approach to musculoskeletal kinematics may facilitate deployment of complex real-time NMS modelling in computer vision or wearable sensors applications to realize biomechanics monitoring, rehabilitation, and disease management outside the research laboratory.
{"title":"Real-Time Calibration-Free Musculotendon Kinematics for Neuromusculoskeletal Models","authors":"Bradley M. Cornish;Laura E. Diamond;David J. Saxby;Zhengliang Xia;Claudio Pizzolato","doi":"10.1109/TNSRE.2024.3455262","DOIUrl":"10.1109/TNSRE.2024.3455262","url":null,"abstract":"Neuromusculoskeletal (NMS) models enable non-invasive estimation of clinically important internal biomechanics. A critical part of NMS modelling is the estimation of musculotendon kinematics, which comprise musculotendon unit lengths, moment arms, and lines of action. Musculotendon kinematics, which are partially dependent on joint angles, define the non-linear mapping of muscle forces to joint moments and contact forces. Currently, real-time computation of musculotendon kinematics requires creation of a per-individual surrogate model. The computational speed and accuracy of these surrogates degrade with increasing number of coordinates. We developed a feed-forward neural network that completely encodes musculotendon kinematics of a target model across a wide anthropometric range, enabling accurate real-time estimates of musculotendon kinematics without need for a priori creation of a per-individual surrogate model. Compared to reference, the neural network had median normalized errors ~0.1% for musculotendon lengths, <0.4%> <tex-math>$1.23pm 0.15$ </tex-math></inline-formula>\u0000%) compared to using reference musculotendon kinematics. Finally, execution time was <0.04 ms per frame and constant for increasing number of model coordinates. Our approach to musculoskeletal kinematics may facilitate deployment of complex real-time NMS modelling in computer vision or wearable sensors applications to realize biomechanics monitoring, rehabilitation, and disease management outside the research laboratory.","PeriodicalId":13419,"journal":{"name":"IEEE Transactions on Neural Systems and Rehabilitation Engineering","volume":"32 ","pages":"3486-3495"},"PeriodicalIF":4.8,"publicationDate":"2024-09-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10669099","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142142978","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-09-05DOI: 10.1109/TNSRE.2024.3454088
Fenqi Rong;Banghua Yang;Cuntai Guan
The EEG is a widely utilized neural signal source, particularly in motor imagery-based brain-computer interface (MI-BCI), offering distinct advantages in applications like stroke rehabilitation. Current research predominantly concentrates on the bilateral limbs paradigm and decoding, but the use scenarios for stroke rehabilitation are typically for unilateral upper limbs. There is a significant challenge to decoding unilateral MI of multitasks due to the overlapped spatial neural activities of the tasks. This study aims to formulate a novel MI-BCI experimental paradigm for unilateral limbs with multitasks. The paradigm encompasses four imagined movement directions: top-bottom, left-right, top right-bottom left, and top left-bottom right. Forty-six healthy subjects participated in this experiment. Commonly used machine learning techniques, such as FBCSP, EEGNet, deepConvNet, and FBCNet, were employed for evaluation. To improve decoding accuracy, we propose an MVCA method that introduces temporal convolution and attention mechanism to effectively capture temporal features from multiple perspectives. With the MVCA model, we have achieved 40.6% and 64.89% classification accuracies for the four-class and two-class scenarios (top right-bottom left and top left-bottom right), respectively. Conclusion: This is the first study demonstrating that motor imagery of multiple directions in unilateral limbs can be decoded. In particular, decoding two directions, right top to left bottom and left top to right bottom, provides the best accuracy, which sheds light on future studies. This study advances the development of the MI-BCI paradigm, offering preliminary evidence for the feasibility of decoding multiple directional information from EEG. This, in turn, enhances the dimensions of MI control commands.
{"title":"Decoding Multi-Class Motor Imagery From Unilateral Limbs Using EEG Signals","authors":"Fenqi Rong;Banghua Yang;Cuntai Guan","doi":"10.1109/TNSRE.2024.3454088","DOIUrl":"10.1109/TNSRE.2024.3454088","url":null,"abstract":"The EEG is a widely utilized neural signal source, particularly in motor imagery-based brain-computer interface (MI-BCI), offering distinct advantages in applications like stroke rehabilitation. Current research predominantly concentrates on the bilateral limbs paradigm and decoding, but the use scenarios for stroke rehabilitation are typically for unilateral upper limbs. There is a significant challenge to decoding unilateral MI of multitasks due to the overlapped spatial neural activities of the tasks. This study aims to formulate a novel MI-BCI experimental paradigm for unilateral limbs with multitasks. The paradigm encompasses four imagined movement directions: top-bottom, left-right, top right-bottom left, and top left-bottom right. Forty-six healthy subjects participated in this experiment. Commonly used machine learning techniques, such as FBCSP, EEGNet, deepConvNet, and FBCNet, were employed for evaluation. To improve decoding accuracy, we propose an MVCA method that introduces temporal convolution and attention mechanism to effectively capture temporal features from multiple perspectives. With the MVCA model, we have achieved 40.6% and 64.89% classification accuracies for the four-class and two-class scenarios (top right-bottom left and top left-bottom right), respectively. Conclusion: This is the first study demonstrating that motor imagery of multiple directions in unilateral limbs can be decoded. In particular, decoding two directions, right top to left bottom and left top to right bottom, provides the best accuracy, which sheds light on future studies. This study advances the development of the MI-BCI paradigm, offering preliminary evidence for the feasibility of decoding multiple directional information from EEG. This, in turn, enhances the dimensions of MI control commands.","PeriodicalId":13419,"journal":{"name":"IEEE Transactions on Neural Systems and Rehabilitation Engineering","volume":"32 ","pages":"3399-3409"},"PeriodicalIF":4.8,"publicationDate":"2024-09-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10666752","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142139973","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：