The role of vowel and consonant onsets in neural tracking of natural speech

IF 3.7 3区 医学 Q2 ENGINEERING, BIOMEDICAL Journal of neural engineering Pub Date : 2023-07-31 DOI:10.1088/1741-2552/ad1784
Mohammad Jalilpour Monesi, Jonas Vanthornhout, T. Francart, Hugo Van hamme
{"title":"The role of vowel and consonant onsets in neural tracking of natural speech","authors":"Mohammad Jalilpour Monesi, Jonas Vanthornhout, T. Francart, Hugo Van hamme","doi":"10.1088/1741-2552/ad1784","DOIUrl":null,"url":null,"abstract":"Objective. To investigate how the auditory system processes natural speech, models have been created to relate the electroencephalography (EEG) signal of a person listening to speech to various representations of the speech. Mainly the speech envelope has been used, but also phonetic representations. We investigated to which degree of granularity phonetic representations can be related to the EEG signal. Approach. We used recorded EEG signals from 105 subjects while they listened to fairy tale stories. We utilized speech representations, including onset of any phone, vowel-consonant onsets, broad phonetic class (BPC) onsets, and narrow phonetic class (NPC) onsets, and related them to EEG using forward modeling and match-mismatch tasks. In forward modeling, we used a linear model to predict EEG from speech representations. In the match-mismatch task, we trained a long short term memory (LSTM) based model to determine which of two candidate speech segments matches with a given EEG segment. Main results. Our results show that vowel-consonant onsets outperform onsets of any phone in both tasks, which suggests that neural tracking of the vowel vs. consonant exists in the EEG to some degree. We also observed that vowel (syllable nucleus) onsets exhibit a more consistent representation in EEG compared to syllable onsets. Significance. Finally, our findings suggest that neural tracking previously thought to be associated with broad phonetic classes might actually originate from vowel-consonant onsets rather than the differentiation between different phonetic classes.","PeriodicalId":16753,"journal":{"name":"Journal of neural engineering","volume":null,"pages":null},"PeriodicalIF":3.7000,"publicationDate":"2023-07-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of neural engineering","FirstCategoryId":"5","ListUrlMain":"https://doi.org/10.1088/1741-2552/ad1784","RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"ENGINEERING, BIOMEDICAL","Score":null,"Total":0}
引用次数: 0

Abstract

Objective. To investigate how the auditory system processes natural speech, models have been created to relate the electroencephalography (EEG) signal of a person listening to speech to various representations of the speech. Mainly the speech envelope has been used, but also phonetic representations. We investigated to which degree of granularity phonetic representations can be related to the EEG signal. Approach. We used recorded EEG signals from 105 subjects while they listened to fairy tale stories. We utilized speech representations, including onset of any phone, vowel-consonant onsets, broad phonetic class (BPC) onsets, and narrow phonetic class (NPC) onsets, and related them to EEG using forward modeling and match-mismatch tasks. In forward modeling, we used a linear model to predict EEG from speech representations. In the match-mismatch task, we trained a long short term memory (LSTM) based model to determine which of two candidate speech segments matches with a given EEG segment. Main results. Our results show that vowel-consonant onsets outperform onsets of any phone in both tasks, which suggests that neural tracking of the vowel vs. consonant exists in the EEG to some degree. We also observed that vowel (syllable nucleus) onsets exhibit a more consistent representation in EEG compared to syllable onsets. Significance. Finally, our findings suggest that neural tracking previously thought to be associated with broad phonetic classes might actually originate from vowel-consonant onsets rather than the differentiation between different phonetic classes.
查看原文
分享 分享
微信好友 朋友圈 QQ好友 复制链接
本刊更多论文
元音和辅音起始点在自然语音神经跟踪中的作用
目的。为了研究听觉系统如何处理自然语音,我们建立了一些模型,将听语音的人的脑电图(EEG)信号与语音的各种表征联系起来。主要使用的是语音包络线,但也使用了语音表征。我们研究了语音表征与脑电信号的关联粒度。研究方法我们使用了 105 名受试者聆听童话故事时录制的脑电信号。我们利用语音表征,包括任何电话的起始音、元音-辅音起始音、广义语音类(BPC)起始音和狭义语音类(NPC)起始音,并通过前向建模和匹配-错配任务将它们与脑电图联系起来。在前向建模中,我们使用线性模型根据语音表征预测脑电图。在匹配-错配任务中,我们训练了一个基于长短期记忆(LSTM)的模型,以确定两个候选语音片段中哪一个与给定的脑电图片段相匹配。主要结果我们的结果表明,在这两项任务中,元音与辅音的起音效果优于任何音调的起音效果,这表明脑电图在一定程度上存在对元音与辅音的神经跟踪。我们还观察到,元音(音节核)母音在脑电图中的表现比音节母音更一致。意义最后,我们的研究结果表明,以前认为与宽泛的语音类别相关的神经跟踪实际上可能源自元音-辅音的起始,而不是不同语音类别之间的区分。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 去求助
来源期刊
Journal of neural engineering
Journal of neural engineering 工程技术-工程:生物医学
CiteScore
7.80
自引率
12.50%
发文量
319
审稿时长
4.2 months
期刊介绍: The goal of Journal of Neural Engineering (JNE) is to act as a forum for the interdisciplinary field of neural engineering where neuroscientists, neurobiologists and engineers can publish their work in one periodical that bridges the gap between neuroscience and engineering. The journal publishes articles in the field of neural engineering at the molecular, cellular and systems levels. The scope of the journal encompasses experimental, computational, theoretical, clinical and applied aspects of: Innovative neurotechnology; Brain-machine (computer) interface; Neural interfacing; Bioelectronic medicines; Neuromodulation; Neural prostheses; Neural control; Neuro-rehabilitation; Neurorobotics; Optical neural engineering; Neural circuits: artificial & biological; Neuromorphic engineering; Neural tissue regeneration; Neural signal processing; Theoretical and computational neuroscience; Systems neuroscience; Translational neuroscience; Neuroimaging.
期刊最新文献
Building consensus on clinical outcome assessments for BCI devices. A summary of the 10th BCI society meeting 2023 workshop. o-CLEAN: a novel multi-stage algorithm for the ocular artifacts' correction from EEG data in out-of-the-lab applications. PDMS/CNT electrodes with bioamplifier for practical in-the-ear and conventional biosignal recordings. DOCTer: a novel EEG-based diagnosis framework for disorders of consciousness. I see artifacts: ICA-based EEG artifact removal does not improve deep network decoding across three BCI tasks.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
已复制链接
已复制链接
快去分享给好友吧!
我知道了
×
扫码分享
扫码分享
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1