Pub Date : 2024-06-06DOI: 10.1109/TCDS.2024.3410371
Hu Zhang;Yanchen Li;Luziwei Leng;Kaiwei Che;Qian Liu;Qinghai Guo;Jianxing Liao;Ran Cheng
Event-based sensors, distinguished by their high temporal resolution of $1 {boldsymbol{mu}}text{s}$ and a dynamic range of $120 mathrm{dB}$ , stand out as ideal tools for deployment in fast-paced settings such as vehicles and drones. Traditional object detection techniques that utilize artificial neural networks (ANNs) face challenges due to the sparse and asynchronous nature of the events these sensors capture. In contrast, spiking neural networks (SNNs) offer a promising alternative, providing a temporal representation that is inherently aligned with event-based data. This article explores the unique membrane potential dynamics of SNNs and their ability to modulate sparse events. We introduce an innovative spike-triggered adaptive threshold mechanism designed for stable training. Building on these insights, we present a specialized spiking feature pyramid network (SpikeFPN) optimized for automotive event-based object detection. Comprehensive evaluations demonstrate that SpikeFPN surpasses both traditional SNNs and advanced ANNs enhanced with attention mechanisms. Evidently, SpikeFPN achieves a mean average precision (mAP) of 0.477 on the GEN1 automotive detection (GAD) benchmark dataset, marking significant increases over the selected SNN baselines. Moreover, the efficient design of SpikeFPN ensures robust performance while optimizing computational resources, attributed to its innate sparse computation capabilities.
基于事件的传感器以其$1 {boldsymbol{mu}}text{s}$的高时间分辨率和$120 math {dB}$的动态范围而闻名，是在车辆和无人机等快节奏环境中部署的理想工具。利用人工神经网络（ann）的传统目标检测技术由于这些传感器捕获的事件的稀疏性和异步性而面临挑战。相比之下，峰值神经网络（snn）提供了一个很有前途的替代方案，它提供了一个与基于事件的数据内在一致的时间表示。本文探讨了snn独特的膜电位动力学及其调节稀疏事件的能力。我们引入了一种创新的峰值触发自适应阈值机制，用于稳定训练。基于这些见解，我们提出了一种专门针对汽车基于事件的目标检测进行优化的峰值特征金字塔网络（SpikeFPN）。综合评价表明，SpikeFPN优于传统的snn和增强了注意力机制的高级ann。显然，SpikeFPN在GEN1汽车检测（GAD）基准数据集上实现了0.477的平均精度（mAP），这标志着所选SNN基线的显著提高。此外，由于其固有的稀疏计算能力，SpikeFPN的高效设计在优化计算资源的同时确保了鲁棒性。
{"title":"Automotive Object Detection via Learning Sparse Events by Spiking Neurons","authors":"Hu Zhang;Yanchen Li;Luziwei Leng;Kaiwei Che;Qian Liu;Qinghai Guo;Jianxing Liao;Ran Cheng","doi":"10.1109/TCDS.2024.3410371","DOIUrl":"10.1109/TCDS.2024.3410371","url":null,"abstract":"Event-based sensors, distinguished by their high temporal resolution of \u0000<inline-formula><tex-math>$1 {boldsymbol{mu}}text{s}$</tex-math></inline-formula>\u0000 and a dynamic range of \u0000<inline-formula><tex-math>$120 mathrm{dB}$</tex-math></inline-formula>\u0000, stand out as ideal tools for deployment in fast-paced settings such as vehicles and drones. Traditional object detection techniques that utilize artificial neural networks (ANNs) face challenges due to the sparse and asynchronous nature of the events these sensors capture. In contrast, spiking neural networks (SNNs) offer a promising alternative, providing a temporal representation that is inherently aligned with event-based data. This article explores the unique membrane potential dynamics of SNNs and their ability to modulate sparse events. We introduce an innovative spike-triggered adaptive threshold mechanism designed for stable training. Building on these insights, we present a specialized spiking feature pyramid network (SpikeFPN) optimized for automotive event-based object detection. Comprehensive evaluations demonstrate that SpikeFPN surpasses both traditional SNNs and advanced ANNs enhanced with attention mechanisms. Evidently, SpikeFPN achieves a mean average precision (mAP) of 0.477 on the GEN1 automotive detection (GAD) benchmark dataset, marking significant increases over the selected SNN baselines. Moreover, the efficient design of SpikeFPN ensures robust performance while optimizing computational resources, attributed to its innate sparse computation capabilities.","PeriodicalId":54300,"journal":{"name":"IEEE Transactions on Cognitive and Developmental Systems","volume":"16 6","pages":"2110-2124"},"PeriodicalIF":5.0,"publicationDate":"2024-06-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141940256","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-06-05DOI: 10.1109/TCDS.2024.3410110
Elisa Luque-Buzo;Mehdi Bejani;Julián D. Arias-Londoñ;Jorge A. Gómez-García;Francisco Grandas-Pérez;Juan I. Godino-Llorente
In binocular vision, the visual system combines images in the retina to generate a single perception, which triggers a sensorimotor process that forces the eyes to point to the same target. Thus, following a moving target, both eyes are expected to move synchronously following identical motor triggers but, in practise, significant differences between eyes are found due to the presence of certain artifacts and effects. Thus, a better indirect characterization of the underlying neurological behavior during eye motion would require new automatic preprocessing methods applied to the eye-tracking sequences for rendering the common and most significant movements of both eyes. To address this need, the present study proposes an automatic method for extracting the common components of the left- and right-eye motions from a set of Smooth Pursuit tests by applying an independent component analysis. To do so, both sequences are decomposed into two independent latent components: the first presumably correlates with the common motor triggering at the brain, while the second collects artifacts introduced during the recording process and small effects due to convergence deficits and eye dominance biases. The evaluations were carried out using data corresponding to 12 different smooth pursuit eye movements tests, which were collected using an infrared high-speed video-based eye-tracking device from 41 parkinsonian patients and 47 controls. The results show that the automatic method can separate the aforementioned components in 99.50% of cases, extracting a latent component correlated with the common motor triggering at the brain, which we hypothesize is characterizing the movements of the cyclopean eye. The estimated component could be used to simplify any other potential automatic analysis.
{"title":"Estimation of the Cyclopean Eye From Binocular Smooth Pursuit Tests","authors":"Elisa Luque-Buzo;Mehdi Bejani;Julián D. Arias-Londoñ;Jorge A. Gómez-García;Francisco Grandas-Pérez;Juan I. Godino-Llorente","doi":"10.1109/TCDS.2024.3410110","DOIUrl":"10.1109/TCDS.2024.3410110","url":null,"abstract":"In binocular vision, the visual system combines images in the retina to generate a single perception, which triggers a sensorimotor process that forces the eyes to point to the same target. Thus, following a moving target, both eyes are expected to move synchronously following identical motor triggers but, in practise, significant differences between eyes are found due to the presence of certain artifacts and effects. Thus, a better indirect characterization of the underlying neurological behavior during eye motion would require new automatic preprocessing methods applied to the eye-tracking sequences for rendering the common and most significant movements of both eyes. To address this need, the present study proposes an automatic method for extracting the common components of the left- and right-eye motions from a set of Smooth Pursuit tests by applying an independent component analysis. To do so, both sequences are decomposed into two independent latent components: the first presumably correlates with the common motor triggering at the brain, while the second collects artifacts introduced during the recording process and small effects due to convergence deficits and eye dominance biases. The evaluations were carried out using data corresponding to 12 different smooth pursuit eye movements tests, which were collected using an infrared high-speed video-based eye-tracking device from 41 parkinsonian patients and 47 controls. The results show that the automatic method can separate the aforementioned components in 99.50% of cases, extracting a latent component correlated with the common motor triggering at the brain, which we hypothesize is characterizing the movements of the cyclopean eye. The estimated component could be used to simplify any other potential automatic analysis.","PeriodicalId":54300,"journal":{"name":"IEEE Transactions on Cognitive and Developmental Systems","volume":"16 6","pages":"2125-2137"},"PeriodicalIF":5.0,"publicationDate":"2024-06-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10549994","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141940257","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-06-04DOI: 10.1109/TCDS.2024.3409555
Huaqin Sun;Yu Qi;Xiaodi Wu;Junming Zhu;Jianmin Zhang;Yueming Wang
Fine movements of hands play an important role in everyday life. While existing studies have successfully decoded hand gestures or finger movements from brain signals, direct decoding of single-joint kinematics remains challenging. This study aims to investigate the decoding of fine hand movements at the single-joint level. Neural activities were recorded from the motor cortex (MC) of a human participant while imagining eleven different hand movements. We comprehensively evaluated the decoding efficiency of various brain signal features, neural decoding algorithms, and single-joint kinematic variables for decoding. Results showed that using the spiking band power (SBP) signals, we could faithfully decode the single-joint angles with an average correlation coefficient of 0.77, outperforming other brain signal features. Nonlinear approaches that incorporate temporal context information, particularly recurrent neural networks, significantly outperformed traditional methods. Decoding joint angles yielded superior results compared to joint angular velocities. Our approach facilitates the construction of high-performance brain–computer interfaces for dexterous hand control.
{"title":"Decoding Joint-Level Hand Movements With Intracortical Neural Signals in a Human Brain–Computer Interface","authors":"Huaqin Sun;Yu Qi;Xiaodi Wu;Junming Zhu;Jianmin Zhang;Yueming Wang","doi":"10.1109/TCDS.2024.3409555","DOIUrl":"10.1109/TCDS.2024.3409555","url":null,"abstract":"Fine movements of hands play an important role in everyday life. While existing studies have successfully decoded hand gestures or finger movements from brain signals, direct decoding of single-joint kinematics remains challenging. This study aims to investigate the decoding of fine hand movements at the single-joint level. Neural activities were recorded from the motor cortex (MC) of a human participant while imagining eleven different hand movements. We comprehensively evaluated the decoding efficiency of various brain signal features, neural decoding algorithms, and single-joint kinematic variables for decoding. Results showed that using the spiking band power (SBP) signals, we could faithfully decode the single-joint angles with an average correlation coefficient of 0.77, outperforming other brain signal features. Nonlinear approaches that incorporate temporal context information, particularly recurrent neural networks, significantly outperformed traditional methods. Decoding joint angles yielded superior results compared to joint angular velocities. Our approach facilitates the construction of high-performance brain–computer interfaces for dexterous hand control.","PeriodicalId":54300,"journal":{"name":"IEEE Transactions on Cognitive and Developmental Systems","volume":"16 6","pages":"2100-2109"},"PeriodicalIF":5.0,"publicationDate":"2024-06-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141940258","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-06-03DOI: 10.1109/TCDS.2024.3406889
Peng Gao;Xiuxian Li;Jinwen Hu
This article studies multiplayer reach-avoid games with a plane being the goal in 3-D space. Due to the difficulty that directly analyzing multipursuer multievader scenarios brings the curse of dimensionality, the whole problem is decomposed to distinct subgames. In the subgames, a single pursuer or multiple pursuers, which have different speeds, form a team to capture one evader cooperatively while the evader struggles to reach the plane. With the players’ dominance region based on the definition of isochronous surfaces, the target points and value functions are obtained for the game of degree by using Apollonius spheres. Additionally, the corresponding closed-loop saddle-point strategies are shown to be Nash equilibrium. The degeneration between scenarios of different scales is also discussed. To minimize the sum of subgames’ costs, the tasks of intercepting multiple evaders are assigned to individuals or teams in the form of bipartite graph matching. A hierarchical matching algorithm and a state-feedback rematching method are proposed which can be updated in real-time to improve the solution. Finally, diverse empirical experiments and comparisons with state-of-the-art methods are illustrated to demonstrate the optimality of proposed strategies and algorithms in this article.
{"title":"Optimal Strategies and Cooperative Teaming for 3-D Multiplayer Reach-Avoid Games","authors":"Peng Gao;Xiuxian Li;Jinwen Hu","doi":"10.1109/TCDS.2024.3406889","DOIUrl":"10.1109/TCDS.2024.3406889","url":null,"abstract":"This article studies multiplayer reach-avoid games with a plane being the goal in 3-D space. Due to the difficulty that directly analyzing multipursuer multievader scenarios brings the curse of dimensionality, the whole problem is decomposed to distinct subgames. In the subgames, a single pursuer or multiple pursuers, which have different speeds, form a team to capture one evader cooperatively while the evader struggles to reach the plane. With the players’ dominance region based on the definition of isochronous surfaces, the target points and value functions are obtained for the game of degree by using Apollonius spheres. Additionally, the corresponding closed-loop saddle-point strategies are shown to be Nash equilibrium. The degeneration between scenarios of different scales is also discussed. To minimize the sum of subgames’ costs, the tasks of intercepting multiple evaders are assigned to individuals or teams in the form of bipartite graph matching. A hierarchical matching algorithm and a state-feedback rematching method are proposed which can be updated in real-time to improve the solution. Finally, diverse empirical experiments and comparisons with state-of-the-art methods are illustrated to demonstrate the optimality of proposed strategies and algorithms in this article.","PeriodicalId":54300,"journal":{"name":"IEEE Transactions on Cognitive and Developmental Systems","volume":"16 6","pages":"2085-2099"},"PeriodicalIF":5.0,"publicationDate":"2024-06-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141940306","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-04-30DOI: 10.1109/TCDS.2024.3395443
Yulin Li;Liangwei Fan;Hui Shen;Dewen Hu
Spiking neural network (SNN) excels in processing temporal information and conserving energy, particularly when deployed on neuromorphic hardware. These strengths position SNN as an ideal choice for developing wearable brain–computer interface (BCI) devices. However, the application of SNN in complex BCI tasks, like four-class motor imagery classification, is limited. In light of this, this study introduces a powerful SNN architecture hybrid response SNN (HR-SNN). We employ parameterwise gradient descent methods to optimize spike encoding efficiency. The SNN's frequency perception is improved by integrating a hybrid response spiking module. In addition, a diff-potential spiking decoder is designed to optimize SNN output potential utilization. Validation experiments are performed on PhysioNet and BCI competition IV 2a datasets. On PhysioNet, our model achieves accuracies of 67.24% and 74.95% using global training and subject-specific transfer learning, respectively. On BCI competition IV 2a, our approach attains an average accuracy of 77.58%, surpassing all the compared SNN models and demonstrating competitiveness against state-of-the-art (SOTA) convolution neural network (CNN) approaches. We validate the robustness of HR-SNN under noise and channel loss scenarios. Additionally, energy analysis reveals HR-SNN's superior energy efficiency compared to existing CNN models. Notably, HR-SNN exhibits a 2–16 times energy consumption advantage over existing SNN methods.
尖峰神经网络（SNN）在处理时间信息和节约能量方面表现突出，特别是在神经形态硬件上部署时。这些优势使SNN成为开发可穿戴脑机接口（BCI）设备的理想选择。然而，SNN在复杂脑机接口任务中的应用，如四类运动图像分类，是有限的。鉴于此，本研究引入了一种功能强大的SNN架构混合响应SNN （HR-SNN）。我们采用参数梯度下降方法来优化尖峰编码效率。通过集成混合响应尖峰模块，SNN的频率感知得到了改善。此外，设计了一种差分电位尖峰解码器，以优化SNN输出电位利用率。验证实验在PhysioNet和BCI competition IV 2a数据集上进行。在PhysioNet上，我们的模型使用全局训练和特定学科迁移学习分别达到67.24%和74.95%的准确率。在BCI竞赛IV 2a上，我们的方法达到了77.58%的平均准确率，超过了所有比较的SNN模型，并展示了与最先进的（SOTA）卷积神经网络（CNN）方法的竞争力。我们验证了HR-SNN在噪声和信道损失情况下的鲁棒性。此外，能量分析表明，与现有的CNN模型相比，HR-SNN具有优越的能源效率。值得注意的是，HR-SNN的能耗优势是现有SNN方法的2-16倍。
{"title":"HR-SNN: An End-to-End Spiking Neural Network for Four-Class Classification Motor Imagery Brain–Computer Interface","authors":"Yulin Li;Liangwei Fan;Hui Shen;Dewen Hu","doi":"10.1109/TCDS.2024.3395443","DOIUrl":"10.1109/TCDS.2024.3395443","url":null,"abstract":"Spiking neural network (SNN) excels in processing temporal information and conserving energy, particularly when deployed on neuromorphic hardware. These strengths position SNN as an ideal choice for developing wearable brain–computer interface (BCI) devices. However, the application of SNN in complex BCI tasks, like four-class motor imagery classification, is limited. In light of this, this study introduces a powerful SNN architecture hybrid response SNN (HR-SNN). We employ parameterwise gradient descent methods to optimize spike encoding efficiency. The SNN's frequency perception is improved by integrating a hybrid response spiking module. In addition, a diff-potential spiking decoder is designed to optimize SNN output potential utilization. Validation experiments are performed on PhysioNet and BCI competition IV 2a datasets. On PhysioNet, our model achieves accuracies of 67.24% and 74.95% using global training and subject-specific transfer learning, respectively. On BCI competition IV 2a, our approach attains an average accuracy of 77.58%, surpassing all the compared SNN models and demonstrating competitiveness against state-of-the-art (SOTA) convolution neural network (CNN) approaches. We validate the robustness of HR-SNN under noise and channel loss scenarios. Additionally, energy analysis reveals HR-SNN's superior energy efficiency compared to existing CNN models. Notably, HR-SNN exhibits a 2–16 times energy consumption advantage over existing SNN methods.","PeriodicalId":54300,"journal":{"name":"IEEE Transactions on Cognitive and Developmental Systems","volume":"16 6","pages":"1955-1968"},"PeriodicalIF":5.0,"publicationDate":"2024-04-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140829368","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Training programs, based on principles of brain-plasticity and skill learning, are useful in counteracting functional decline in pathological conditions. Training effects of such procedures are well described but their adaptive features are usually not reported. A software framework designed for a long-term home training program is presented. It gradually trains users, provides a multidimensional range of stimulus differentiation, encompasses a strategy to increase the task demand and includes motivational reinforcement components. The structured framework was tested in a feasibility study involving two perceptual discrimination tasks (visual and auditory) in four persons in middle-to-older adulthood who were trained for 30 days. Practicability of the training was shown in a home setting by high adherence to the procedure, adaptive increase in task demand over time and positive learning effects on an individual level. Participants learned to distinguish progressively smaller target objects in the visual task (with diminished contrast) and sweeps progressively varying less in frequency in the auditory task (with overlapping noise). This adaptive procedure can provide the basis for the design of extended training programs engaging sensory function in individuals with impaired sensorimotor and cognitive functions. Further investigations are necessary to assess the generalization of learning effects and clinical validity.
{"title":"Adaptive Framework for Long-Term Sensory Home Training: A Feasibility Study","authors":"Stefano Silvoni;Simon Desch;Florian Beier;Robin Bekrater-Bodmann;Annette Löffler;Dieter Kleinböhl;Stefano Tamascelli;Herta Flor","doi":"10.1109/TCDS.2024.3393635","DOIUrl":"10.1109/TCDS.2024.3393635","url":null,"abstract":"Training programs, based on principles of brain-plasticity and skill learning, are useful in counteracting functional decline in pathological conditions. Training effects of such procedures are well described but their adaptive features are usually not reported. A software framework designed for a long-term home training program is presented. It gradually trains users, provides a multidimensional range of stimulus differentiation, encompasses a strategy to increase the task demand and includes motivational reinforcement components. The structured framework was tested in a feasibility study involving two perceptual discrimination tasks (visual and auditory) in four persons in middle-to-older adulthood who were trained for 30 days. Practicability of the training was shown in a home setting by high adherence to the procedure, adaptive increase in task demand over time and positive learning effects on an individual level. Participants learned to distinguish progressively smaller target objects in the visual task (with diminished contrast) and sweeps progressively varying less in frequency in the auditory task (with overlapping noise). This adaptive procedure can provide the basis for the design of extended training programs engaging sensory function in individuals with impaired sensorimotor and cognitive functions. Further investigations are necessary to assess the generalization of learning effects and clinical validity.","PeriodicalId":54300,"journal":{"name":"IEEE Transactions on Cognitive and Developmental Systems","volume":"16 6","pages":"1929-1942"},"PeriodicalIF":5.0,"publicationDate":"2024-04-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10508624","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140797954","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-04-23DOI: 10.1109/TCDS.2024.3392745
Jing Jin;Xinjie He;Brendan Z. Allison;Ke Qin;Xingyu Wang;Andrzej Cichocki
Online adaptive canonical correction analysis (OACCA) has been applied successfully in the recently popular steady-state visual evoked potential (SSVEP) target recognition methods. However, due to the significant amount of spatiotemporal relevant background noise in the online historical sample label data of OACCA, there is redundant noise component in the learned common spatial filter that can reduce online classification accuracy. Aiming at solving this defect in OACCA, we designed an online spatial–temporal equalization filter (STE) to suppress the background noise component in the electroencephalography (EEG). Meanwhile, an adaptive decoding method for SSVEP based on online spatial–temporal estimation (STE-OACCA) is proposed by combining the online STE filter and the OACCA algorithm. A pseudoonline test on the Tsinghua University FBCCA-DW dataset shows that the proposed STE-OACCA method significantly outperforms the CCA, MSI, OACCA approaches as well as STE-CCA. More importantly, proposed method can be directly used in online SSVEP recognition without calibration. The proposed algorithm is robust, which is promising for the development of practical brain computer interface (BCI).
{"title":"Leveraging Spatiotemporal Estimation for Online Adaptive Steady-State Visual Evoked Potential Recognition","authors":"Jing Jin;Xinjie He;Brendan Z. Allison;Ke Qin;Xingyu Wang;Andrzej Cichocki","doi":"10.1109/TCDS.2024.3392745","DOIUrl":"10.1109/TCDS.2024.3392745","url":null,"abstract":"Online adaptive canonical correction analysis (OACCA) has been applied successfully in the recently popular steady-state visual evoked potential (SSVEP) target recognition methods. However, due to the significant amount of spatiotemporal relevant background noise in the online historical sample label data of OACCA, there is redundant noise component in the learned common spatial filter that can reduce online classification accuracy. Aiming at solving this defect in OACCA, we designed an online spatial–temporal equalization filter (STE) to suppress the background noise component in the electroencephalography (EEG). Meanwhile, an adaptive decoding method for SSVEP based on online spatial–temporal estimation (STE-OACCA) is proposed by combining the online STE filter and the OACCA algorithm. A pseudoonline test on the Tsinghua University FBCCA-DW dataset shows that the proposed STE-OACCA method significantly outperforms the CCA, MSI, OACCA approaches as well as STE-CCA. More importantly, proposed method can be directly used in online SSVEP recognition without calibration. The proposed algorithm is robust, which is promising for the development of practical brain computer interface (BCI).","PeriodicalId":54300,"journal":{"name":"IEEE Transactions on Cognitive and Developmental Systems","volume":"16 6","pages":"1943-1954"},"PeriodicalIF":5.0,"publicationDate":"2024-04-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140797952","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-04-18DOI: 10.1109/TCDS.2024.3391131
Haojin Deng;Shiqi Wang;Yimin Yang;W. G. Will Zhao;Hui Zhang;Ruizhong Wei;Q. M. Jonathan Wu;Bao-Liang Lu
Emotion is one of the main psychological factors that affects human behavior. Using a neural network model trained with electroencephalography (EEG)-based frequency features has been widely used to accurately recognize human emotions. However, utilizing EEG-based spatial information with popular 2-D kernels of convolutional neural networks (CNNs) has rarely been explored in the extant literature. This article addresses these challenges by proposing an EEG-based spatial-frequency-based framework for recognizing human emotion, resulting in fewer human interference parameters with better generalization performance. Specifically, we propose a two-stream hierarchical network framework that learns features from two networks, one trained from the frequency domain while another trained from the spatial domain. Our approach is extensively validated on the SEED, SEED-V, and DREAMER datasets. Our proposed method achieved an accuracy of 94.84% on the SEED dataset and 68.61% on the SEED-V dataset with EEG data only. The average accuracy of the Dreamer dataset is 93.01%, 92.04%, and 91.74% in valence, arousal, and dominance dimensions, respectively. The experiments directly support that our motivation of utilizing the two-stream domain features significantly improves the final recognition performance. The experimental results show that the proposed framework obtains improvements over state-of-the-art methods over these three varied scaled datasets. Furthermore, it also indicates the potential of the proposed framework in conjunction with current ImageNet pretrained models for improving performance on 1-D psychological signals.
{"title":"Minimizing EEG Human Interference: A Study of an Adaptive EEG Spatial Feature Extraction With Deep Convolutional Neural Networks","authors":"Haojin Deng;Shiqi Wang;Yimin Yang;W. G. Will Zhao;Hui Zhang;Ruizhong Wei;Q. M. Jonathan Wu;Bao-Liang Lu","doi":"10.1109/TCDS.2024.3391131","DOIUrl":"10.1109/TCDS.2024.3391131","url":null,"abstract":"Emotion is one of the main psychological factors that affects human behavior. Using a neural network model trained with electroencephalography (EEG)-based frequency features has been widely used to accurately recognize human emotions. However, utilizing EEG-based spatial information with popular 2-D kernels of convolutional neural networks (CNNs) has rarely been explored in the extant literature. This article addresses these challenges by proposing an EEG-based spatial-frequency-based framework for recognizing human emotion, resulting in fewer human interference parameters with better generalization performance. Specifically, we propose a two-stream hierarchical network framework that learns features from two networks, one trained from the frequency domain while another trained from the spatial domain. Our approach is extensively validated on the SEED, SEED-V, and DREAMER datasets. Our proposed method achieved an accuracy of 94.84% on the SEED dataset and 68.61% on the SEED-V dataset with EEG data only. The average accuracy of the Dreamer dataset is 93.01%, 92.04%, and 91.74% in valence, arousal, and dominance dimensions, respectively. The experiments directly support that our motivation of utilizing the two-stream domain features significantly improves the final recognition performance. The experimental results show that the proposed framework obtains improvements over state-of-the-art methods over these three varied scaled datasets. Furthermore, it also indicates the potential of the proposed framework in conjunction with current ImageNet pretrained models for improving performance on 1-D psychological signals.","PeriodicalId":54300,"journal":{"name":"IEEE Transactions on Cognitive and Developmental Systems","volume":"16 6","pages":"1915-1928"},"PeriodicalIF":5.0,"publicationDate":"2024-04-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140627221","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Significant advancements in semantic parsing for text-to-SQL (T2S) tasks have been achieved through the employment of neural network models, such as LSTM, BERT, and T5. The exceptional performance of large language models, such as ChatGPT, has been demonstrated in recent research, even in zero-shot scenarios. However, the inherent transparency of T2S models presents them as black boxes, concealing their inner workings from both developers and users, which complicates the diagnosis of potential error patterns. Despite the fact that numerous visual analysis studies have been conducted in natural language processing communities, scant attention has been paid to addressing the challenges of semantic parsing, specifically in T2S tasks. This limitation hinders the development of effective tools for model optimization and evaluation. This article presents an interactive visual analysis tool, MAVIDSQL, to assist model developers and users in understanding and diagnosing T2S tasks. The system comprises three modules: the model manager, the feature extractor, and the visualization interface, which adopt a model-agnostic approach to diagnose potential errors and infer model decisions by analyzing input–output data, facilitating interactive visual analysis to identify error patterns and assess model performance. Two case studies and interviews with domain experts demonstrate the effectiveness of MAVIDSQL in facilitating the understanding of T2S tasks and identifying potential errors.
{"title":"MAVIDSQL: A Model-Agnostic Visualization for Interpretation and Diagnosis of Text-to-SQL Tasks","authors":"Jingwei Tang;Guodao Sun;Jiahui Chen;Gefei Zhang;Baofeng Chang;Haixia Wang;Ronghua Liang","doi":"10.1109/TCDS.2024.3391278","DOIUrl":"10.1109/TCDS.2024.3391278","url":null,"abstract":"Significant advancements in semantic parsing for text-to-SQL (T2S) tasks have been achieved through the employment of neural network models, such as LSTM, BERT, and T5. The exceptional performance of large language models, such as ChatGPT, has been demonstrated in recent research, even in zero-shot scenarios. However, the inherent transparency of T2S models presents them as black boxes, concealing their inner workings from both developers and users, which complicates the diagnosis of potential error patterns. Despite the fact that numerous visual analysis studies have been conducted in natural language processing communities, scant attention has been paid to addressing the challenges of semantic parsing, specifically in T2S tasks. This limitation hinders the development of effective tools for model optimization and evaluation. This article presents an interactive visual analysis tool, MAVIDSQL, to assist model developers and users in understanding and diagnosing T2S tasks. The system comprises three modules: the model manager, the feature extractor, and the visualization interface, which adopt a model-agnostic approach to diagnose potential errors and infer model decisions by analyzing input–output data, facilitating interactive visual analysis to identify error patterns and assess model performance. Two case studies and interviews with domain experts demonstrate the effectiveness of MAVIDSQL in facilitating the understanding of T2S tasks and identifying potential errors.","PeriodicalId":54300,"journal":{"name":"IEEE Transactions on Cognitive and Developmental Systems","volume":"16 5","pages":"1887-1903"},"PeriodicalIF":5.0,"publicationDate":"2024-04-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140627741","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-04-17DOI: 10.1109/TCDS.2024.3390597
Timur Ibrayev;Amitangshu Mukherjee;Sai Aparna Aketi;Kaushik Roy
Deep neural network (DNN) based machine perception frameworks process the entire input in a one-shot manner to provide answers to both “ what object is being observed” and “ where it is located.” In contrast, the “two-stream hypothesis” from neuroscience explains the neural processing in the human visual cortex as an active vision system that utilizes two separate regions of the brain to answer the what and the where questions. In this work, we propose a machine learning framework inspired by the “two-stream hypothesis” and explore the potential benefits that it offers. Specifically, the proposed framework models the following mechanisms: 1) ventral ( what ) stream focusing on the input regions perceived by the fovea part of an eye (foveation); 2) dorsal ( where ) stream providing visual guidance; and 3) iterative processing of the two streams to calibrate visual focus and process the sequence of focused image patches. The training of the proposed framework is accomplished by label-based DNN training for the ventral stream model and reinforcement learning (RL) for the dorsal stream model. We show that the two-stream foveation-based learning is applicable to the challenging task of weakly-supervised object localization (WSOL), where the training data is limited to the object class or its attributes. The framework is capable of both predicting the properties of an object and successfully localizing it by predicting its bounding box. We also show that, due to the independent nature of the two streams, the dorsal model can be applied on its own to unseen images to localize objects from different datasets.
{"title":"Toward Two-Stream Foveation-Based Active Vision Learning","authors":"Timur Ibrayev;Amitangshu Mukherjee;Sai Aparna Aketi;Kaushik Roy","doi":"10.1109/TCDS.2024.3390597","DOIUrl":"10.1109/TCDS.2024.3390597","url":null,"abstract":"Deep neural network (DNN) based machine perception frameworks process the entire input in a one-shot manner to provide answers to both “\u0000<italic>what</i>\u0000 object is being observed” and “\u0000<italic>where</i>\u0000 it is located.” In contrast, the \u0000<italic>“two-stream hypothesis”</i>\u0000 from neuroscience explains the neural processing in the human visual cortex as an active vision system that utilizes two separate regions of the brain to answer the \u0000<italic>what</i>\u0000 and the \u0000<italic>where</i>\u0000 questions. In this work, we propose a machine learning framework inspired by the \u0000<italic>“two-stream hypothesis”</i>\u0000 and explore the potential benefits that it offers. Specifically, the proposed framework models the following mechanisms: 1) ventral (\u0000<italic>what</i>\u0000) stream focusing on the input regions perceived by the fovea part of an eye (foveation); 2) dorsal (\u0000<italic>where</i>\u0000) stream providing visual guidance; and 3) iterative processing of the two streams to calibrate visual focus and process the sequence of focused image patches. The training of the proposed framework is accomplished by label-based DNN training for the ventral stream model and reinforcement learning (RL) for the dorsal stream model. We show that the two-stream foveation-based learning is applicable to the challenging task of weakly-supervised object localization (WSOL), where the training data is limited to the object class or its attributes. The framework is capable of both predicting the properties of an object \u0000<italic>and</i>\u0000 successfully localizing it by predicting its bounding box. We also show that, due to the independent nature of the two streams, the dorsal model can be applied on its own to unseen images to localize objects from different datasets.","PeriodicalId":54300,"journal":{"name":"IEEE Transactions on Cognitive and Developmental Systems","volume":"16 5","pages":"1843-1860"},"PeriodicalIF":5.0,"publicationDate":"2024-04-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140615331","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：