首页 > 最新文献

Journal of neural engineering最新文献

英文 中文
Estimating cognitive workload using a commercial in-ear EEG headset. 使用商用耳内脑电图耳机估算认知工作量。
Pub Date : 2024-11-05 DOI: 10.1088/1741-2552/ad8ef8
Christoph Tremmel, Dean J Krusienski, M C Schraefel

Objective: This study investigated the potential of estimating various mental workload levels during two different tasks using a commercial in-ear electroencephalography (EEG) system, the IDUN "Guardian". Approach: Participants performed versions of two classical workload tasks: an n-back task and a mental arithmetic task. Both in-ear and conventional EEG data were simultaneously collected during these tasks. In an effort to facilitate a more comprehensive comparison, the complexity of the tasks was intentionally increased beyond typical levels. Special emphasis was also placed on understanding the significance of gamma band activity in workload estimations. Therefore, each signal was analyzed across low frequency (1-35 Hz) and high frequency (1-100 Hz) ranges. Additionally, surrogate in-ear EEG measures, derived from the conventional EEG recordings, were extracted and examined. Main results: Workload estimation using in-ear EEG yielded statistically significant performance levels, surpassing chance levels with 44.1% for four classes and 68.4% for two classes in the n-back task and was better than a naive predictor for the mental arithmetic task. Conventional EEG exhibited significantly higher performance compared to in-ear EEG, achieving 80.3% and 92.9% accuracy for the respective tasks, along with lower error rates than the naive predictor. The developed surrogate measures achieved improved results, reaching accuracies of 57.5% and 85.5%, thus providing insights for enhancing future in-ear systems. Notably, most high frequency range signals outperformed their low frequency counterparts in terms of accuracy validating that high frequency gamma band features can improve workload estimation. Significance: The application of EEG-based Brain-Computer Interfaces (BCIs) beyond laboratory settings is often hindered by practical limitations. In-ear EEG systems offer a promising solution to this problem, potentially enabling everyday use. This study evaluates the performance of a commercial in-ear headset and provides guidelines for increased effectiveness.

研究目的本研究使用商用耳内脑电图(EEG)系统 IDUN "Guardian",对在两项不同任务中估计各种脑力劳动负荷水平的潜力进行了调查:受试者完成了两项经典的脑力劳动任务:n-back 任务和心算任务。在完成这些任务的过程中,同时收集耳内和传统脑电图数据。为了便于进行更全面的比较,我们有意提高了任务的复杂性,使其超出了一般水平。我们还特别强调要了解伽玛波段活动在工作量估算中的重要性。因此,对每个信号都进行了低频(1-35 赫兹)和高频(1-100 赫兹)范围的分析。此外,还从常规脑电图记录中提取并检查了替代耳内脑电图测量值:使用耳内脑电图估算工作量的结果具有显著的统计学意义,在 n-back 任务中,四个等级的工作量估算率为 44.1%,两个等级的工作量估算率为 68.4%,超过了偶然水平。与耳内脑电图相比,传统脑电图的性能明显更高,在相应任务中分别达到了 80.3% 和 92.9% 的准确率,错误率也低于天真预测器。所开发的替代测量方法取得了更好的结果,准确率分别达到 57.5% 和 85.5%,从而为增强未来的耳内式系统提供了启示。值得注意的是,大多数高频范围的信号在准确性方面优于低频范围的信号,这验证了高频伽玛频段特征可以改善工作量估算:基于脑电图的脑机接口(BCI)在实验室以外的应用往往受到实际限制的阻碍。入耳式脑电图系统为这一问题提供了一个很有前景的解决方案,有可能实现日常使用。本研究评估了商用入耳式耳机的性能,并提供了提高效率的指导原则。
{"title":"Estimating cognitive workload using a commercial in-ear EEG headset.","authors":"Christoph Tremmel, Dean J Krusienski, M C Schraefel","doi":"10.1088/1741-2552/ad8ef8","DOIUrl":"https://doi.org/10.1088/1741-2552/ad8ef8","url":null,"abstract":"<p><strong>Objective: </strong>This study investigated the potential of estimating various mental workload levels during two different tasks using a commercial in-ear electroencephalography (EEG) system, the IDUN \"Guardian\".&#xD;Approach: Participants performed versions of two classical workload tasks: an n-back task and a mental arithmetic task. Both in-ear and conventional EEG data were simultaneously collected during these tasks. In an effort to facilitate a more comprehensive comparison, the complexity of the tasks was intentionally increased beyond typical levels. Special emphasis was also placed on understanding the significance of gamma band activity in workload estimations. Therefore, each signal was analyzed across low frequency (1-35 Hz) and high frequency (1-100 Hz) ranges. Additionally, surrogate in-ear EEG measures, derived from the conventional EEG recordings, were extracted and examined.&#xD;Main results: Workload estimation using in-ear EEG yielded statistically significant performance levels, surpassing chance levels with 44.1% for four classes and 68.4% for two classes in the n-back task and was better than a naive predictor for the mental arithmetic task. Conventional EEG exhibited significantly higher performance compared to in-ear EEG, achieving 80.3% and 92.9% accuracy for the respective tasks, along with lower error rates than the naive predictor. The developed surrogate measures achieved improved results, reaching accuracies of 57.5% and 85.5%, thus providing insights for enhancing future in-ear systems. Notably, most high frequency range signals outperformed their low frequency counterparts in terms of accuracy validating that high frequency gamma band features can improve workload estimation.&#xD; Significance: The application of EEG-based Brain-Computer Interfaces (BCIs) beyond laboratory settings is often hindered by practical limitations. In-ear EEG systems offer a promising solution to this problem, potentially enabling everyday use. This study evaluates the performance of a commercial in-ear headset and provides guidelines for increased effectiveness.</p>","PeriodicalId":94096,"journal":{"name":"Journal of neural engineering","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2024-11-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142583725","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Functional connectivity of stimulus-evoked brain responses to natural speech in post-stroke aphasia. 中风后失语症患者大脑对自然语言的刺激诱发反应的功能连接性。
Pub Date : 2024-11-05 DOI: 10.1088/1741-2552/ad8ef9
Ramtin Mehraram, Pieter De Clercq, Jill Kries, Maaike Vandermosten, Tom Francart

Objective One out of three stroke-patients develop language processing impairment known as aphasia. The need for ecological validity of the existing diagnostic tools motivates research on biomarkers, such as stimulus-evoked brain responses. With the aim of enhancing the physiological interpretation of the latter, we used EEG to investigate how functional brain network patterns associated with the neural response to natural speech are affected in persons with post-stroke chronic aphasia. Approach EEG was recorded from 24 healthy controls and 40 persons with aphasia while they listened to a story. Stimulus-evoked brain responses at all scalp regions were measured as neural envelope tracking in the delta (0.5-4 Hz), theta (4-8 Hz) and low-gamma bands (30-49 Hz) using mutual information. Functional connectivity between neural-tracking signals was measured, and the Network-Based Statistics toolbox was used to: 1) assess the added value of the neural tracking vs EEG time series, 2) test between-group differences and 3) investigate any association with language performance in aphasia. Graph theory was also used to investigate topological alterations in aphasia. Main results Functional connectivity was higher when assessed from neural tracking compared to EEG time series. Persons with aphasia showed weaker low-gamma-band left-hemispheric connectivity, and graph theory-based results showed a greater network segregation and higher region-specific node strength. Aphasia also exhibited a correlation between delta-band connectivity within the left pre-frontal region and language performance. Significance We demonstrated the added value of combining brain connectomics with neural-tracking measurement when investigating natural speech processing in post-stroke aphasia. The higher sensitivity to language-related brain circuits of this approach favours its use as informative biomarker for the assessment of aphasia. .

目标 每三名中风患者中就有一人出现语言处理障碍,即失语症。对现有诊断工具生态有效性的需求推动了对生物标志物(如刺激诱发的大脑反应)的研究。为了加强对后者的生理学解释,我们使用脑电图研究中风后慢性失语症患者对自然语音的神经反应相关的脑功能网络模式是如何受到影响的。所有头皮区域的刺激诱发脑部反应均以神经包络跟踪的方式进行测量,包络跟踪的频段包括δ(0.5-4 Hz)、θ(4-8 Hz)和低γ频段(30-49 Hz)。对神经跟踪信号之间的功能连接性进行了测量,并使用基于网络的统计工具箱进行了以下分析1)评估神经跟踪与脑电图时间序列的附加值;2)测试组间差异;3)研究与失语症患者语言表达的关联。图论也被用来研究失语症的拓扑变化。主要结果 神经追踪评估的功能连接性高于脑电图时间序列。失语症患者的低伽马带左半球连通性较弱,基于图论的结果显示网络分离程度更高,特定区域节点强度更高。失语症患者左侧前额叶区域的δ波段连通性与语言能力之间也存在相关性。这种方法对语言相关脑回路的灵敏度更高,有利于将其用作评估失语症的信息生物标志物。
{"title":"Functional connectivity of stimulus-evoked brain responses to natural speech in post-stroke aphasia.","authors":"Ramtin Mehraram, Pieter De Clercq, Jill Kries, Maaike Vandermosten, Tom Francart","doi":"10.1088/1741-2552/ad8ef9","DOIUrl":"https://doi.org/10.1088/1741-2552/ad8ef9","url":null,"abstract":"<p><p>Objective&#xD;One out of three stroke-patients develop language processing impairment known as aphasia. The need for ecological validity of the existing diagnostic tools motivates research on biomarkers, such as stimulus-evoked brain responses. With the aim of enhancing the physiological interpretation of the latter, we used EEG to investigate how functional brain network patterns associated with the neural response to natural speech are affected in persons with post-stroke chronic aphasia.&#xD;&#xD;Approach&#xD;EEG was recorded from 24 healthy controls and 40 persons with aphasia while they listened to a story. Stimulus-evoked brain responses at all scalp regions were measured as neural envelope tracking in the delta (0.5-4 Hz), theta (4-8 Hz) and low-gamma bands (30-49 Hz) using mutual information. Functional connectivity between neural-tracking signals was measured, and the Network-Based Statistics toolbox was used to: 1) assess the added value of the neural tracking vs EEG time series, 2) test between-group differences and 3) investigate any association with language performance in aphasia. Graph theory was also used to investigate topological alterations in aphasia.&#xD;&#xD;Main results&#xD;Functional connectivity was higher when assessed from neural tracking compared to EEG time series. Persons with aphasia showed weaker low-gamma-band left-hemispheric connectivity, and graph theory-based results showed a greater network segregation and higher region-specific node strength. Aphasia also exhibited a correlation between delta-band connectivity within the left pre-frontal region and language performance.&#xD;&#xD;Significance&#xD;We demonstrated the added value of combining brain connectomics with neural-tracking measurement when investigating natural speech processing in post-stroke aphasia. The higher sensitivity to language-related brain circuits of this approach favours its use as informative biomarker for the assessment of aphasia.&#xD.</p>","PeriodicalId":94096,"journal":{"name":"Journal of neural engineering","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2024-11-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142584028","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Temporal attention fusion network with custom loss function for EEG--fNIRS classification. 采用自定义损失函数的时态注意力融合网络,用于脑电图-近红外成像分类。
Pub Date : 2024-11-04 DOI: 10.1088/1741-2552/ad8e86
Chayut Bunterngchit, Jiaxing Wang, Jianqiang Su, Yihan Wang, Shiqi Liu, Zeng-Guang Hou

Objective.Methods that can detect brain activities accurately are crucial owing to the increasing prevalence of neurological disorders. In this context, a combination of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) offers a powerful approach to understanding normal and pathological brain functions, thereby overcoming the limitations of each modality, such as susceptibility to artefacts of EEG and limited temporal resolution of fNIRS. However, challenges such as class imbalance and inter-class variability within multisubject data hinder their full potential.Approach.To address this issue, we propose a novel temporal attention fusion network (TAFN) with a custom loss function. The TAFN model incorporates attention mechanisms to its long short-term memory and temporal convolutional layers to accurately capture spatial and temporal dependencies in the EEG--fNIRS data. The custom loss function combines class weights and asymmetric loss terms to ensure the precise classification of cognitive and motor intentions, along with addressing class imbalance issues.Main resultsRigorous testing demonstrated the exceptional cross-subject accuracy of the TAFN, exceeding 99% for cognitive tasks and 97% for motor imagery (MI) tasks. Additionally, the ability of the model to detect subtle differences in epilepsy was analyzed using scalp topography in MI tasks.Significance.This study presents a technique that outperforms traditional methods for detecting high-precision brain activity with subtle differences in the associated patterns. This makes it a promising tool for applications such as epilepsy and seizure detection, in which discerning subtle pattern differences is of paramount importance.

目的:由于神经系统疾病的发病率越来越高,能够准确检测大脑活动的方法至关重要。在这种情况下,脑电图(EEG)和功能性近红外光谱(fNIRS)的结合为了解正常和病理大脑功能提供了一种强有力的方法,从而克服了每种模式的局限性,如脑电图易受伪影影响和 fNIRS 的时间分辨率有限。为解决这一问题,我们提出了一种具有自定义损失函数的新型时空注意力融合网络(TAFN)。TAFN 模型将注意力机制纳入其长短期记忆和时间卷积层,以准确捕捉 EEG-fNIRS 数据中的空间和时间依赖性。自定义损失函数结合了类权重和非对称损失项,以确保认知意图和运动意图的精确分类,同时解决类不平衡问题。主要结果严格的测试表明,TAFN 的跨受试者准确率非常高,认知任务超过 99%,运动想象(MI)任务超过 97%。这项研究提出的技术在检测相关模式中存在细微差别的高精度大脑活动方面优于传统方法。这使得该技术成为癫痫和癫痫发作检测等应用领域的一种前景广阔的工具,在这些应用领域中,辨别细微的模式差异至关重要。
{"title":"Temporal attention fusion network with custom loss function for EEG--fNIRS classification.","authors":"Chayut Bunterngchit, Jiaxing Wang, Jianqiang Su, Yihan Wang, Shiqi Liu, Zeng-Guang Hou","doi":"10.1088/1741-2552/ad8e86","DOIUrl":"https://doi.org/10.1088/1741-2552/ad8e86","url":null,"abstract":"<p><p><i>Objective.</i>Methods that can detect brain activities accurately are crucial owing to the increasing prevalence of neurological disorders. In this context, a combination of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) offers a powerful approach to understanding normal and pathological brain functions, thereby overcoming the limitations of each modality, such as susceptibility to artefacts of EEG and limited temporal resolution of fNIRS. However, challenges such as class imbalance and inter-class variability within multisubject data hinder their full potential.<i>Approach.</i>To address this issue, we propose a novel temporal attention fusion network (TAFN) with a custom loss function. The TAFN model incorporates attention mechanisms to its long short-term memory and temporal convolutional layers to accurately capture spatial and temporal dependencies in the EEG--fNIRS data. The custom loss function combines class weights and asymmetric loss terms to ensure the precise classification of cognitive and motor intentions, along with addressing class imbalance issues.<i>Main results</i>Rigorous testing demonstrated the exceptional cross-subject accuracy of the TAFN, exceeding 99% for cognitive tasks and 97% for motor imagery (MI) tasks. Additionally, the ability of the model to detect subtle differences in epilepsy was analyzed using scalp topography in MI tasks.<i>Significance.</i>This study presents a technique that outperforms traditional methods for detecting high-precision brain activity with subtle differences in the associated patterns. This makes it a promising tool for applications such as epilepsy and seizure detection, in which discerning subtle pattern differences is of paramount importance.</p>","PeriodicalId":94096,"journal":{"name":"Journal of neural engineering","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2024-11-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142577177","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Real-time TMS-EEG for brain state-controlled research and precision treatment: a narrative review and guide. 用于脑状态控制研究和精准治疗的实时 TMS-EEG 综述和指南。
Pub Date : 2024-11-01 DOI: 10.1088/1741-2552/ad8a8e
Miles Wischnewski, Sina Shirinpour, Ivan Alekseichuk, Maria I Lapid, Ziad Nahas, Kelvin O Lim, Paul E Croarkin, Alexander Opitz

Transcranial magnetic stimulation (TMS) modulates neuronal activity, but the efficacy of an open-loop approach is limited due to the brain state's dynamic nature. Real-time integration with electroencephalography (EEG) increases experimental reliability and offers personalized neuromodulation therapy by using immediate brain states as biomarkers. Here, we review brain state-controlled TMS-EEG studies since the first publication several years ago. A summary of experiments on the sensorimotor mu rhythm (8-13 Hz) shows increased cortical excitability due to TMS pulse at the trough and decreased excitability at the peak of the oscillation. Pre-TMS pulse mu power also affects excitability. Further, there is emerging evidence that the oscillation phase in theta and beta frequency bands modulates neural excitability. Here, we provide a guide for real-time TMS-EEG application and discuss experimental and technical considerations. We consider the effects of hardware choice, signal quality, spatial and temporal filtering, and neural characteristics of the targeted brain oscillation. Finally, we speculate on how closed-loop TMS-EEG potentially could improve the treatment of neurological and mental disorders such as depression, Alzheimer's, Parkinson's, schizophrenia, and stroke.

经颅磁刺激(TMS)可调节神经元活动,但由于大脑状态的动态性,开环方法的功效有限。与脑电图(EEG)的实时整合提高了实验的可靠性,并通过使用即时脑状态作为生物标记提供了个性化的神经调节疗法。在此,我们回顾了自几年前首次发表以来的脑状态控制 TMS-EEG 研究。对感觉运动μ节律(8-13 Hz)的实验总结显示,在振荡的低谷,TMS 脉冲会增加大脑皮层的兴奋性,而在振荡的峰值,兴奋性会降低。TMS脉冲前的μ功率也会影响兴奋性。此外,越来越多的证据表明,θ 和 β 频段的振荡相位会调节神经兴奋性。在此,我们将为 TMS-EEG 的实时应用提供指导,并讨论实验和技术方面的注意事项。我们考虑了硬件选择、信号质量、空间和时间滤波以及目标大脑振荡的神经特征的影响。最后,我们推测闭环 TMS-EEG 有可能改善抑郁症、阿尔茨海默氏症、帕金森氏症、精神分裂症和中风等神经和精神疾病的治疗。
{"title":"Real-time TMS-EEG for brain state-controlled research and precision treatment: a narrative review and guide.","authors":"Miles Wischnewski, Sina Shirinpour, Ivan Alekseichuk, Maria I Lapid, Ziad Nahas, Kelvin O Lim, Paul E Croarkin, Alexander Opitz","doi":"10.1088/1741-2552/ad8a8e","DOIUrl":"10.1088/1741-2552/ad8a8e","url":null,"abstract":"<p><p>Transcranial magnetic stimulation (TMS) modulates neuronal activity, but the efficacy of an open-loop approach is limited due to the brain state's dynamic nature. Real-time integration with electroencephalography (EEG) increases experimental reliability and offers personalized neuromodulation therapy by using immediate brain states as biomarkers. Here, we review brain state-controlled TMS-EEG studies since the first publication several years ago. A summary of experiments on the sensorimotor mu rhythm (8-13 Hz) shows increased cortical excitability due to TMS pulse at the trough and decreased excitability at the peak of the oscillation. Pre-TMS pulse mu power also affects excitability. Further, there is emerging evidence that the oscillation phase in theta and beta frequency bands modulates neural excitability. Here, we provide a guide for real-time TMS-EEG application and discuss experimental and technical considerations. We consider the effects of hardware choice, signal quality, spatial and temporal filtering, and neural characteristics of the targeted brain oscillation. Finally, we speculate on how closed-loop TMS-EEG potentially could improve the treatment of neurological and mental disorders such as depression, Alzheimer's, Parkinson's, schizophrenia, and stroke.</p>","PeriodicalId":94096,"journal":{"name":"Journal of neural engineering","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2024-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11528152/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142515455","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Review of deep representation learning techniques for brain-computer interfaces. 脑机接口深度表征学习技术综述。
Pub Date : 2024-11-01 DOI: 10.1088/1741-2552/ad8962
Pierre Guetschel, Sara Ahmadi, Michael Tangermann

In the field of brain-computer interfaces (BCIs), the potential for leveraging deep learning techniques for representing electroencephalogram (EEG) signals has gained substantial interest.Objective: This review synthesizes empirical findings from a collection of articles using deep representation learning techniques for BCI decoding, to provide a comprehensive analysis of the current state-of-the-art.Approach: Each article was scrutinized based on three criteria: (1) the deep representation learning technique employed, (2) the underlying motivation for its utilization, and (3) the approaches adopted for characterizing the learned representations.Main results: Among the 81 articles finally reviewed in depth, our analysis reveals a predominance of 31 articles using autoencoders. We identified 13 studies employing self-supervised learning (SSL) techniques, among which ten were published in 2022 or later, attesting to the relative youth of the field. However, at the time being, none of these have led to standard foundation models that are picked up by the BCI community. Likewise, only a few studies have introspected their learned representations. We observed that the motivation in most studies for using representation learning techniques is for solving transfer learning tasks, but we also found more specific motivations such as to learn robustness or invariances, as an algorithmic bridge, or finally to uncover the structure of the data.Significance: Given the potential of foundation models to effectively tackle these challenges, we advocate for a continued dedication to the advancement of foundation models specifically designed for EEG signal decoding by using SSL techniques. We also underline the imperative of establishing specialized benchmarks and datasets to facilitate the development and continuous improvement of such foundation models.

在脑机接口(BCI)领域,利用深度学习技术表示脑电图(EEG)信号的潜力已引起了广泛关注。本综述综合了使用深度表征学习技术进行 BCI 解码的一系列文章中的实证研究结果,对当前最先进的技术进行了全面分析。每篇文章都根据三个标准进行了仔细研究:(1) 采用的深度表征学习技术;(2) 使用该技术的根本动机;(3) 采用的表征所学表征的方法。在最终深入研究的 81 篇文章中,我们的分析显示有 31 篇文章主要采用了自动编码器。我们发现有 13 篇研究采用了自我监督学习(SSL)技术,其中有 10 篇发表于 2022 年或之后,证明了该领域的相对年轻。不过,目前这些研究都还没有形成被生物识别(BCI)领域采用的标准基础模型。同样,只有少数研究对学习到的表征进行了反省。我们注意到,大多数研究使用表征学习技术的动机都是为了解决迁移学习任务,但我们也发现了一些更具体的动机,如学习鲁棒性或不变性,作为算法桥梁,或最终揭示数据结构。鉴于基础模型在有效解决这些挑战方面的潜力,我们主张继续利用 SSL 技术,致力于推进专为脑电信号解码设计的基础模型。我们还强调必须建立专门的基准和数据集,以促进此类基础模型的开发和持续改进。
{"title":"Review of deep representation learning techniques for brain-computer interfaces.","authors":"Pierre Guetschel, Sara Ahmadi, Michael Tangermann","doi":"10.1088/1741-2552/ad8962","DOIUrl":"10.1088/1741-2552/ad8962","url":null,"abstract":"<p><p>In the field of brain-computer interfaces (BCIs), the potential for leveraging deep learning techniques for representing electroencephalogram (EEG) signals has gained substantial interest.<i>Objective</i>: This review synthesizes empirical findings from a collection of articles using deep representation learning techniques for BCI decoding, to provide a comprehensive analysis of the current state-of-the-art.<i>Approach</i>: Each article was scrutinized based on three criteria: (1) the deep representation learning technique employed, (2) the underlying motivation for its utilization, and (3) the approaches adopted for characterizing the learned representations.<i>Main results</i>: Among the 81 articles finally reviewed in depth, our analysis reveals a predominance of 31 articles using autoencoders. We identified 13 studies employing self-supervised learning (SSL) techniques, among which ten were published in 2022 or later, attesting to the relative youth of the field. However, at the time being, none of these have led to standard foundation models that are picked up by the BCI community. Likewise, only a few studies have introspected their learned representations. We observed that the motivation in most studies for using representation learning techniques is for solving transfer learning tasks, but we also found more specific motivations such as to learn robustness or invariances, as an algorithmic bridge, or finally to uncover the structure of the data.<i>Significance</i>: Given the potential of foundation models to effectively tackle these challenges, we advocate for a continued dedication to the advancement of foundation models specifically designed for EEG signal decoding by using SSL techniques. We also underline the imperative of establishing specialized benchmarks and datasets to facilitate the development and continuous improvement of such foundation models.</p>","PeriodicalId":94096,"journal":{"name":"Journal of neural engineering","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2024-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142484379","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Reducing power requirements for high-accuracy decoding in iBCIs. 降低 iBCI 中高精度解码的功耗要求。
Pub Date : 2024-11-01 DOI: 10.1088/1741-2552/ad88a4
Brianna M Karpowicz, Bareesh Bhaduri, Samuel R Nason-Tomaszewski, Brandon G Jacques, Yahia H Ali, Robert D Flint, Payton H Bechefsky, Leigh R Hochberg, Nicholas AuYong, Marc W Slutzky, Chethan Pandarinath

Objective.Current intracortical brain-computer interfaces (iBCIs) rely predominantly on threshold crossings ('spikes') for decoding neural activity into a control signal for an external device. Spiking data can yield high accuracy online control during complex behaviors; however, its dependence on high-sampling-rate data collection can pose challenges. An alternative signal for iBCI decoding is the local field potential (LFP), a continuous-valued signal that can be acquired simultaneously with spiking activity. However, LFPs are seldom used alone for online iBCI control as their decoding performance has yet to achieve parity with spikes.Approach.Here, we present a strategy to improve the performance of LFP-based decoders by first training a neural dynamics model to use LFPs to reconstruct the firing rates underlying spiking data, and then decoding from the estimated rates. We test these models on previously-collected macaque data during center-out and random-target reaching tasks as well as data collected from a human iBCI participant during attempted speech.Main results.In all cases, training models from LFPs enables firing rate reconstruction with accuracy comparable to spiking-based dynamics models. In addition, LFP-based dynamics models enable decoding performance exceeding that of LFPs alone and approaching that of spiking-based models. In all applications except speech, LFP-based dynamics models also facilitate decoding accuracy exceeding that of direct decoding from spikes.Significance.Because LFP-based dynamics models operate on lower bandwidth and with lower sampling rate than spiking models, our findings indicate that iBCI devices can be designed to operate with lower power requirements than devices dependent on recorded spiking activity, without sacrificing high-accuracy decoding.

目的:目前的皮层内脑机接口(iBCI)主要依靠阈值交叉("尖峰")将神经活动解码为外部设备的控制信号。尖峰数据可以在复杂行为中产生高精度的在线控制;然而,它对高采样率数据收集的依赖会带来挑战。用于 iBCI 解码的另一种信号是局部场电位(LFP),这是一种连续值信号,可与尖峰活动同时采集。然而,LFP 很少单独用于在线 iBCI 控制,因为其解码性能尚未达到与尖峰信号相当的水平:在此,我们提出了一种提高基于 LFP 的解码器性能的策略,首先训练神经动力学模型,利用 LFP 重建尖峰数据的发射率,然后根据估计的发射率进行解码。我们在以前收集的猕猴在中心向外和随机目标到达任务中的数据以及人类 iBCI 参与者在尝试说话时收集的数据上测试了这些模型:在所有情况下,通过 LFPs 训练模型可以重建发射率,其准确性可与基于尖峰脉冲的动力学模型相媲美。此外,基于 LFP 的动力学模型的解码性能超过了单独使用 LFP 的解码性能,接近基于尖峰模型的解码性能。在除语音外的所有应用中,基于 LFP 的动力学模型也有助于提高解码精度,超过直接从尖峰解码的精度:意义:与尖峰模型相比,基于 LFP 的动态模型的带宽更低,采样率也更低,因此我们的研究结果表明,与依赖尖峰活动记录的设备相比,iBCI 设备的运行功耗要求更低,而不会影响高精度解码。
{"title":"Reducing power requirements for high-accuracy decoding in iBCIs.","authors":"Brianna M Karpowicz, Bareesh Bhaduri, Samuel R Nason-Tomaszewski, Brandon G Jacques, Yahia H Ali, Robert D Flint, Payton H Bechefsky, Leigh R Hochberg, Nicholas AuYong, Marc W Slutzky, Chethan Pandarinath","doi":"10.1088/1741-2552/ad88a4","DOIUrl":"10.1088/1741-2552/ad88a4","url":null,"abstract":"<p><p><i>Objective.</i>Current intracortical brain-computer interfaces (iBCIs) rely predominantly on threshold crossings ('spikes') for decoding neural activity into a control signal for an external device. Spiking data can yield high accuracy online control during complex behaviors; however, its dependence on high-sampling-rate data collection can pose challenges. An alternative signal for iBCI decoding is the local field potential (LFP), a continuous-valued signal that can be acquired simultaneously with spiking activity. However, LFPs are seldom used alone for online iBCI control as their decoding performance has yet to achieve parity with spikes.<i>Approach.</i>Here, we present a strategy to improve the performance of LFP-based decoders by first training a neural dynamics model to use LFPs to reconstruct the firing rates underlying spiking data, and then decoding from the estimated rates. We test these models on previously-collected macaque data during center-out and random-target reaching tasks as well as data collected from a human iBCI participant during attempted speech.<i>Main results.</i>In all cases, training models from LFPs enables firing rate reconstruction with accuracy comparable to spiking-based dynamics models. In addition, LFP-based dynamics models enable decoding performance exceeding that of LFPs alone and approaching that of spiking-based models. In all applications except speech, LFP-based dynamics models also facilitate decoding accuracy exceeding that of direct decoding from spikes.<i>Significance.</i>Because LFP-based dynamics models operate on lower bandwidth and with lower sampling rate than spiking models, our findings indicate that iBCI devices can be designed to operate with lower power requirements than devices dependent on recorded spiking activity, without sacrificing high-accuracy decoding.</p>","PeriodicalId":94096,"journal":{"name":"Journal of neural engineering","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2024-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11528220/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142484315","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Adversarial artifact detection in EEG-based brain-computer interfaces. 基于脑电图的脑机接口中的对抗性伪影检测。
Pub Date : 2024-10-30 DOI: 10.1088/1741-2552/ad8964
Xiaoqing Chen, Lubin Meng, Yifan Xu, Dongrui Wu

Objective. machine learning has achieved significant success in electroencephalogram (EEG) based brain-computer interfaces (BCIs), with most existing research focusing on improving the decoding accuracy. However, recent studies have shown that EEG-based BCIs are vulnerable to adversarial attacks, where small perturbations added to the input can cause misclassification. Detecting adversarial examples is crucial for both understanding this phenomenon and developing effective defense strategies.Approach. this paper, for the first time, explores adversarial detection in EEG-based BCIs. We extend several popular adversarial detection approaches from computer vision to BCIs. Two new Mahalanobis distance based adversarial detection approaches, and three cosine distance based adversarial detection approaches, are also proposed, which showed promising performance in detecting three kinds of white-box attacks.Main results. we evaluated the performance of eight adversarial detection approaches on three EEG datasets, three neural networks, and four types of adversarial attacks. Our approach achieved an area under the curve score of up to 99.99% in detecting white-box attacks. Additionally, we assessed the transferability of different adversarial detectors to unknown attacks.Significance. through extensive experiments, we found that white-box attacks may be easily detected, and differences exist in the distributions of different types of adversarial examples. Our work should facilitate understanding the vulnerability of existing BCI models and developing more secure BCIs in the future.

目的:机器学习在基于脑电图(EEG)的脑机接口(BCI)方面取得了巨大成功,现有研究大多侧重于提高解码准确性。然而,最近的研究表明,基于脑电图的 BCI 很容易受到对抗性攻击的影响,在输入中添加的微小扰动会导致错误分类。检测对抗范例对于理解这一现象和制定有效的防御策略至关重要:本文首次探讨了基于脑电图的 BCI 中的对抗检测。我们将计算机视觉中几种流行的对抗检测方法扩展到了 BCI。我们还提出了两种新的基于马哈拉诺比斯距离的对抗检测方法和三种基于余弦距离的对抗检测方法,这些方法在检测三种白盒攻击方面表现出了良好的性能:我们在三个脑电图数据集、三个神经网络和四种对抗攻击中评估了八种对抗检测方法的性能。我们的方法在检测白盒攻击方面的曲线下面积(AUC)得分高达 99.99%。此外,我们还评估了不同对抗检测器对未知攻击的可转移性:通过大量实验,我们发现白盒攻击很容易被检测到,而且不同类型的对抗性实例的分布存在差异。我们的工作将有助于了解现有BCI模型的脆弱性,并在未来开发出更安全的BCI。
{"title":"Adversarial artifact detection in EEG-based brain-computer interfaces.","authors":"Xiaoqing Chen, Lubin Meng, Yifan Xu, Dongrui Wu","doi":"10.1088/1741-2552/ad8964","DOIUrl":"10.1088/1741-2552/ad8964","url":null,"abstract":"<p><p><i>Objective</i>. machine learning has achieved significant success in electroencephalogram (EEG) based brain-computer interfaces (BCIs), with most existing research focusing on improving the decoding accuracy. However, recent studies have shown that EEG-based BCIs are vulnerable to adversarial attacks, where small perturbations added to the input can cause misclassification. Detecting adversarial examples is crucial for both understanding this phenomenon and developing effective defense strategies.<i>Approach</i>. this paper, for the first time, explores adversarial detection in EEG-based BCIs. We extend several popular adversarial detection approaches from computer vision to BCIs. Two new Mahalanobis distance based adversarial detection approaches, and three cosine distance based adversarial detection approaches, are also proposed, which showed promising performance in detecting three kinds of white-box attacks.<i>Main results</i>. we evaluated the performance of eight adversarial detection approaches on three EEG datasets, three neural networks, and four types of adversarial attacks. Our approach achieved an area under the curve score of up to 99.99% in detecting white-box attacks. Additionally, we assessed the transferability of different adversarial detectors to unknown attacks.<i>Significance</i>. through extensive experiments, we found that white-box attacks may be easily detected, and differences exist in the distributions of different types of adversarial examples. Our work should facilitate understanding the vulnerability of existing BCI models and developing more secure BCIs in the future.</p>","PeriodicalId":94096,"journal":{"name":"Journal of neural engineering","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2024-10-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142484396","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Continuous and discrete decoding of overt speech with scalp electroencephalography (EEG). 利用头皮脑电图(EEG)对公开语音进行连续和离散解码。
Pub Date : 2024-10-30 DOI: 10.1088/1741-2552/ad8d0a
Alexander Craik, Heather R Dial, Jose L Contreras-Vidal

Neurological disorders affecting speech production adversely impact quality of life for over 7 million individuals in the US. Traditional speech interfaces like eyetracking devices and P300 spellers are slow and unnatural for these patients. An alternative solution, speech Brain-Computer Interfaces (BCIs), directly decodes speech characteristics, offering a more natural communication mechanism. This research explores the feasibility of decoding speech features using non-invasive EEG. Nine neurologically intact participants were equipped with a 63-channel EEG system with additional sensors to eliminate eye artifacts. Participants read aloud sentences displayed on a screen selected for phonetic similarity to the English language. Deep learning models, including Convolutional Neural Networks and Recurrent Neural Networks with and without attention modules, were optimized with a focus on minimizing trainable parameters and utilizing small input window sizes for real-time application. These models were employed for discrete and continuous speech decoding tasks, achieving statistically significant participant-independent decoding performance for discrete classes and continuous characteristics of the produced audio signal. A frequency sub-band analysis highlighted the significance of certain frequency bands (delta, theta, and gamma) for decoding performance, and a perturbation analysis was used to identify crucial channels. Assessed channel selection methods did not significantly improve performance, suggesting a distributed representation of speech information encoded in the EEG signals. Leave-One-Out training demonstrated the feasibility of utilizing common speech neural correlates, reducing data collection requirements from individual participants.

美国有 700 多万人因神经系统疾病而影响了语言能力,对生活质量造成了不利影响。传统的语音界面,如眼球追踪 设备和 P300 拼写器,对这些患者来说既缓慢又不自然。另一种解决方案--语音脑机接口(BCI)可直接解码语音特征,提供更自然的交流机制。这项研究探索了利用无创脑电图解码语音特征的可行性。九名神经系统完好的参与者配备了 63 通道脑电图系统 ,并增加了传感器以消除眼部伪影。参与者朗读屏幕上显示的与英语语音相似的句子。深度学习模型包括卷积神经网络(Convolutional Neural Networks)和递归神经网络(Recurrent Neural Networks),有注意力模块和无注意力模块。这些模型被用于离散和连续语音解码任务,在离散类和连续特征音频信号的解码性能上取得了显著的与参与者无关的统计效果 。频率子带分析强调了某些频段(delta、theta和gamma)对解码性能的重要性,扰动分析用于识别关键信道。经过评估的信道选择方法并没有明显提高性能,这表明脑电信号中编码的语音信息是分布式的。留空训练证明了利用普通语音神经相关性的可行性,从而减少了对个别参与者的数据收集要求 。
{"title":"Continuous and discrete decoding of overt speech with scalp electroencephalography (EEG).","authors":"Alexander Craik, Heather R Dial, Jose L Contreras-Vidal","doi":"10.1088/1741-2552/ad8d0a","DOIUrl":"https://doi.org/10.1088/1741-2552/ad8d0a","url":null,"abstract":"<p><p>Neurological disorders affecting speech production adversely impact quality of&#xD;life for over 7 million individuals in the US. Traditional speech interfaces like eyetracking&#xD;devices and P300 spellers are slow and unnatural for these patients. An&#xD;alternative solution, speech Brain-Computer Interfaces (BCIs), directly decodes speech&#xD;characteristics, offering a more natural communication mechanism. This research&#xD;explores the feasibility of decoding speech features using non-invasive EEG. Nine&#xD;neurologically intact participants were equipped with a 63-channel EEG system&#xD;with additional sensors to eliminate eye artifacts. Participants read aloud sentences&#xD;displayed on a screen selected for phonetic similarity to the English language. Deep&#xD;learning models, including Convolutional Neural Networks and Recurrent Neural&#xD;Networks with and without attention modules, were optimized with a focus on&#xD;minimizing trainable parameters and utilizing small input window sizes for real-time&#xD;application. These models were employed for discrete and continuous speech decoding&#xD;tasks, achieving statistically significant participant-independent decoding performance&#xD;for discrete classes and continuous characteristics of the produced audio signal. A&#xD;frequency sub-band analysis highlighted the significance of certain frequency bands&#xD;(delta, theta, and gamma) for decoding performance, and a perturbation analysis&#xD;was used to identify crucial channels. Assessed channel selection methods did not&#xD;significantly improve performance, suggesting a distributed representation of speech&#xD;information encoded in the EEG signals. Leave-One-Out training demonstrated&#xD;the feasibility of utilizing common speech neural correlates, reducing data collection&#xD;requirements from individual participants.</p>","PeriodicalId":94096,"journal":{"name":"Journal of neural engineering","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2024-10-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142549786","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Identification of perceived sentences using deep neural networks in EEG. 利用脑电图中的深度神经网络识别感知句子。
Pub Date : 2024-10-30 DOI: 10.1088/1741-2552/ad88a3
Carlos Valle, Carolina Mendez-Orellana, Christian Herff, Maria Rodriguez-Fernandez

Objetive. Decoding speech from brain activity can enable communication for individuals with speech disorders. Deep neural networks (DNNs) have shown great potential for speech decoding applications. However, the limited availability of large datasets containing neural recordings from speech-impaired subjects poses a challenge. Leveraging data from healthy participants can mitigate this limitation and expedite the development of speech neuroprostheses while minimizing the need for patient-specific training data.Approach. In this study, we collected a substantial dataset consisting of recordings from 56 healthy participants using 64 EEG channels. Multiple neural networks were trained to classify perceived sentences in the Spanish language using subject-independent, mixed-subjects, and fine-tuning approaches. The dataset has been made publicly available to foster further research in this area.Main results. Our results demonstrate a remarkable level of accuracy in distinguishing sentence identity across 30 classes, showcasing the feasibility of training DNNs to decode sentence identity from perceived speech using EEG. Notably, the subject-independent approach rendered accuracy comparable to the mixed-subjects approach, although with higher variability among subjects. Additionally, our fine-tuning approach yielded even higher accuracy, indicating an improved capability to adapt to individual subject characteristics, which enhances performance. This suggests that DNNs have effectively learned to decode universal features of brain activity across individuals while also being adaptable to specific participant data. Furthermore, our analyses indicate that EEGNet and DeepConvNet exhibit comparable performance, outperforming ShallowConvNet for sentence identity decoding. Finally, our Grad-CAM visualization analysis identifies key areas influencing the network's predictions, offering valuable insights into the neural processes underlying language perception and comprehension.Significance. These findings advance our understanding of EEG-based speech perception decoding and hold promise for the development of speech neuroprostheses, particularly in scenarios where subjects cannot provide their own training data.

目标从大脑活动中解码语音可以帮助有语言障碍的人进行交流。深度神经网络在语音解码应用方面展现出巨大潜力。然而,包含语言障碍受试者神经记录的大型数据集的可用性有限,这构成了一项挑战。利用健康参与者的数据可以缓解这一限制,加快语音神经义肢的开发,同时最大限度地减少对特定患者训练数据的需求。在这项研究中,我们收集了大量数据集,包括 56 名健康参与者使用 64 个脑电图通道的记录。我们使用独立于主体、混合主体和微调方法对多个神经网络进行了训练,以对西班牙语中的感知句子进行分类。该数据集已公开发布,以促进该领域的进一步研究。我们的结果表明,在区分 30 个类别的句子身份方面,我们的准确性达到了很高的水平,这展示了利用脑电图训练深度神经网络(DNN)从感知语音中解码句子身份的可行性。值得注意的是,与受试者无关的方法与混合受试者方法的准确性相当,但受试者之间的差异更大。此外,我们的微调方法还获得了更高的准确率,这表明我们适应个别受试者特征的能力得到了提高,从而提高了性能。这表明,DNN 已经有效地学会了解码不同个体大脑活动的普遍特征,同时也能适应特定的参与者数据。此外,我们的分析表明,EEGNet 和 DeepConvNet 的性能相当,在句子身份解码方面优于 ShallowConvNet。最后,我们的 Grad-CAM 可视化分析确定了影响网络预测的关键区域,为语言感知和理解的神经过程提供了宝贵的见解。这些发现加深了我们对基于脑电图的语音感知解码的理解,为语音神经义肢的开发带来了希望,尤其是在受试者无法提供自己的训练数据的情况下。
{"title":"Identification of perceived sentences using deep neural networks in EEG.","authors":"Carlos Valle, Carolina Mendez-Orellana, Christian Herff, Maria Rodriguez-Fernandez","doi":"10.1088/1741-2552/ad88a3","DOIUrl":"10.1088/1741-2552/ad88a3","url":null,"abstract":"<p><p><i>Objetive</i>. Decoding speech from brain activity can enable communication for individuals with speech disorders. Deep neural networks (DNNs) have shown great potential for speech decoding applications. However, the limited availability of large datasets containing neural recordings from speech-impaired subjects poses a challenge. Leveraging data from healthy participants can mitigate this limitation and expedite the development of speech neuroprostheses while minimizing the need for patient-specific training data.<i>Approach</i>. In this study, we collected a substantial dataset consisting of recordings from 56 healthy participants using 64 EEG channels. Multiple neural networks were trained to classify perceived sentences in the Spanish language using subject-independent, mixed-subjects, and fine-tuning approaches. The dataset has been made publicly available to foster further research in this area.<i>Main results</i>. Our results demonstrate a remarkable level of accuracy in distinguishing sentence identity across 30 classes, showcasing the feasibility of training DNNs to decode sentence identity from perceived speech using EEG. Notably, the subject-independent approach rendered accuracy comparable to the mixed-subjects approach, although with higher variability among subjects. Additionally, our fine-tuning approach yielded even higher accuracy, indicating an improved capability to adapt to individual subject characteristics, which enhances performance. This suggests that DNNs have effectively learned to decode universal features of brain activity across individuals while also being adaptable to specific participant data. Furthermore, our analyses indicate that EEGNet and DeepConvNet exhibit comparable performance, outperforming ShallowConvNet for sentence identity decoding. Finally, our Grad-CAM visualization analysis identifies key areas influencing the network's predictions, offering valuable insights into the neural processes underlying language perception and comprehension.<i>Significance</i>. These findings advance our understanding of EEG-based speech perception decoding and hold promise for the development of speech neuroprostheses, particularly in scenarios where subjects cannot provide their own training data.</p>","PeriodicalId":94096,"journal":{"name":"Journal of neural engineering","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2024-10-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142484312","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Feeling senseless sensations: a crossmodal EEG study of mismatched tactile and visual experiences in virtual reality. 无感的感觉:虚拟现实中不匹配的触觉和视觉体验的跨模态脑电图研究。
Pub Date : 2024-10-29 DOI: 10.1088/1741-2552/ad83f5
Caroline Lehser, Steven A Hillyard, Daniel J Strauss

Objective.To create highly immersive experiences in virtual reality (VR) it is important to not only include the visual sense but also to involve multimodal sensory input. To achieve optimal results, the temporal and spatial synchronization of these multimodal inputs is critical. It is therefore necessary to find methods to objectively evaluate the synchronization of VR experiences with a continuous tracking of the user.Approach.In this study a passive touch experience was incorporated in a visual-tactile VR setup using VR glasses and tactile sensations in mid-air. Inconsistencies of multimodal perception were intentionally integrated into a discrimination task. The participants' electroencephalogram (EEG) was recorded to obtain neural correlates of visual-tactile mismatch situations.Main results.The results showed significant differences in the event-related potentials (ERP) between match and mismatch situations. A biphasic ERP configuration consisting of a positivity at 120 ms and a later negativity at 370 ms was observed following a visual-tactile mismatch.Significance.This late negativity could be related to the N400 that is associated with semantic incongruency. These results provide a promising approach towards the objective evaluation of visual-tactile synchronization in virtual experiences.

要在虚拟现实(VR)中创造高度沉浸式体验,重要的是不仅要包括视觉感官,还要涉及多模态感官 输入。要达到最佳效果,这些多模态输入的时空同步至关重要。因此,有必要找到客观评估 VR 体验与用户连续跟踪同步性的方法。本研究利用 VR 眼镜和半空中的触觉,在视觉-触觉 VR 设置中加入了被动触摸体验。多模态感知的不一致性被有意整合到了一项辨别任务中。研究人员记录了参与者的脑电图(EEG),以获得视觉-触觉不匹配情况下的神经相关性。结果显示,在匹配和不匹配情况下,事件相关电位(ERP)存在明显差异。在视觉-触觉错配后,观察到一种双相的ERP配置,包括120毫秒时的阳性和370毫秒时的阴性。这种晚期负性可能与语义不一致相关的N400有关。这些结果为客观评估虚拟体验中的视觉-触觉同步性提供了一种很有前景的方法 。
{"title":"Feeling senseless sensations: a crossmodal EEG study of mismatched tactile and visual experiences in virtual reality.","authors":"Caroline Lehser, Steven A Hillyard, Daniel J Strauss","doi":"10.1088/1741-2552/ad83f5","DOIUrl":"10.1088/1741-2552/ad83f5","url":null,"abstract":"<p><p><i>Objective.</i>To create highly immersive experiences in virtual reality (VR) it is important to not only include the visual sense but also to involve multimodal sensory input. To achieve optimal results, the temporal and spatial synchronization of these multimodal inputs is critical. It is therefore necessary to find methods to objectively evaluate the synchronization of VR experiences with a continuous tracking of the user.<i>Approach.</i>In this study a passive touch experience was incorporated in a visual-tactile VR setup using VR glasses and tactile sensations in mid-air. Inconsistencies of multimodal perception were intentionally integrated into a discrimination task. The participants' electroencephalogram (EEG) was recorded to obtain neural correlates of visual-tactile mismatch situations.<i>Main results.</i>The results showed significant differences in the event-related potentials (ERP) between match and mismatch situations. A biphasic ERP configuration consisting of a positivity at 120 ms and a later negativity at 370 ms was observed following a visual-tactile mismatch.<i>Significance.</i>This late negativity could be related to the N400 that is associated with semantic incongruency. These results provide a promising approach towards the objective evaluation of visual-tactile synchronization in virtual experiences.</p>","PeriodicalId":94096,"journal":{"name":"Journal of neural engineering","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2024-10-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142396392","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
期刊
Journal of neural engineering
全部 Acc. Chem. Res. ACS Applied Bio Materials ACS Appl. Electron. Mater. ACS Appl. Energy Mater. ACS Appl. Mater. Interfaces ACS Appl. Nano Mater. ACS Appl. Polym. Mater. ACS BIOMATER-SCI ENG ACS Catal. ACS Cent. Sci. ACS Chem. Biol. ACS Chemical Health & Safety ACS Chem. Neurosci. ACS Comb. Sci. ACS Earth Space Chem. ACS Energy Lett. ACS Infect. Dis. ACS Macro Lett. ACS Mater. Lett. ACS Med. Chem. Lett. ACS Nano ACS Omega ACS Photonics ACS Sens. ACS Sustainable Chem. Eng. ACS Synth. Biol. Anal. Chem. BIOCHEMISTRY-US Bioconjugate Chem. BIOMACROMOLECULES Chem. Res. Toxicol. Chem. Rev. Chem. Mater. CRYST GROWTH DES ENERG FUEL Environ. Sci. Technol. Environ. Sci. Technol. Lett. Eur. J. Inorg. Chem. IND ENG CHEM RES Inorg. Chem. J. Agric. Food. Chem. J. Chem. Eng. Data J. Chem. Educ. J. Chem. Inf. Model. J. Chem. Theory Comput. J. Med. Chem. J. Nat. Prod. J PROTEOME RES J. Am. Chem. Soc. LANGMUIR MACROMOLECULES Mol. Pharmaceutics Nano Lett. Org. Lett. ORG PROCESS RES DEV ORGANOMETALLICS J. Org. Chem. J. Phys. Chem. J. Phys. Chem. A J. Phys. Chem. B J. Phys. Chem. C J. Phys. Chem. Lett. Analyst Anal. Methods Biomater. Sci. Catal. Sci. Technol. Chem. Commun. Chem. Soc. Rev. CHEM EDUC RES PRACT CRYSTENGCOMM Dalton Trans. Energy Environ. Sci. ENVIRON SCI-NANO ENVIRON SCI-PROC IMP ENVIRON SCI-WAT RES Faraday Discuss. Food Funct. Green Chem. Inorg. Chem. Front. Integr. Biol. J. Anal. At. Spectrom. J. Mater. Chem. A J. Mater. Chem. B J. Mater. Chem. C Lab Chip Mater. Chem. Front. Mater. Horiz. MEDCHEMCOMM Metallomics Mol. Biosyst. Mol. Syst. Des. Eng. Nanoscale Nanoscale Horiz. Nat. Prod. Rep. New J. Chem. Org. Biomol. Chem. Org. Chem. Front. PHOTOCH PHOTOBIO SCI PCCP Polym. Chem.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1