Guanchao Qiao;Ning Ning;Yue Zuo;Pujun Zhou;Mingliang Sun;Shaogang Hu;Qi Yu;Yang Liu
{"title":"Spatio-Temporal Fusion Spiking Neural Network for Frame-Based and Event-Based Camera Sensor Fusion","authors":"Guanchao Qiao;Ning Ning;Yue Zuo;Pujun Zhou;Mingliang Sun;Shaogang Hu;Qi Yu;Yang Liu","doi":"10.1109/TETCI.2024.3363071","DOIUrl":null,"url":null,"abstract":"Traditional frame-based cameras capture high-resolution images at specific sampling rates while suffering from motion blur and uneven exposure. Emerging event-based cameras can address these issues with event-driven sampling, but fail to capture texture details. Additional information can be obtained by complementing the characteristics of frame- and event-based sensors. A spatio-temporal fusion spiking neural network (STF-SNN) is proposed here for fusing frame- and event-based information. STF-SNN achieves competitive recognition performance on popular datasets. For example, it achieves 95.77% accuracy on the fusion of CIFAR10 and DVS-CIFAR10, which is 5.01% and 19.27% higher than the non-fused SNN based only on the frame- or event-based information, respectively. To the best of our knowledge, this work first uses SNN to mine spatio-temporal information in the frame-event data stream. The main contributions of this work are: (1) it is proposed to fuse information in the spatio-temporal domain at the feature and decision levels, which yields great accuracy improvement; (2) a weight quantization method for STF-SNN is proposed, which well solves the parameter doubling problem caused by information fusion; (3) it is proposed to prepare data by weak correspondence between frame- and event-based data, which lowers the data preparation barrier of STF-SNN.","PeriodicalId":13135,"journal":{"name":"IEEE Transactions on Emerging Topics in Computational Intelligence","volume":null,"pages":null},"PeriodicalIF":5.3000,"publicationDate":"2024-02-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"IEEE Transactions on Emerging Topics in Computational Intelligence","FirstCategoryId":"94","ListUrlMain":"https://ieeexplore.ieee.org/document/10444915/","RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE","Score":null,"Total":0}
引用次数: 0
Abstract
Traditional frame-based cameras capture high-resolution images at specific sampling rates while suffering from motion blur and uneven exposure. Emerging event-based cameras can address these issues with event-driven sampling, but fail to capture texture details. Additional information can be obtained by complementing the characteristics of frame- and event-based sensors. A spatio-temporal fusion spiking neural network (STF-SNN) is proposed here for fusing frame- and event-based information. STF-SNN achieves competitive recognition performance on popular datasets. For example, it achieves 95.77% accuracy on the fusion of CIFAR10 and DVS-CIFAR10, which is 5.01% and 19.27% higher than the non-fused SNN based only on the frame- or event-based information, respectively. To the best of our knowledge, this work first uses SNN to mine spatio-temporal information in the frame-event data stream. The main contributions of this work are: (1) it is proposed to fuse information in the spatio-temporal domain at the feature and decision levels, which yields great accuracy improvement; (2) a weight quantization method for STF-SNN is proposed, which well solves the parameter doubling problem caused by information fusion; (3) it is proposed to prepare data by weak correspondence between frame- and event-based data, which lowers the data preparation barrier of STF-SNN.
期刊介绍:
The IEEE Transactions on Emerging Topics in Computational Intelligence (TETCI) publishes original articles on emerging aspects of computational intelligence, including theory, applications, and surveys.
TETCI is an electronics only publication. TETCI publishes six issues per year.
Authors are encouraged to submit manuscripts in any emerging topic in computational intelligence, especially nature-inspired computing topics not covered by other IEEE Computational Intelligence Society journals. A few such illustrative examples are glial cell networks, computational neuroscience, Brain Computer Interface, ambient intelligence, non-fuzzy computing with words, artificial life, cultural learning, artificial endocrine networks, social reasoning, artificial hormone networks, computational intelligence for the IoT and Smart-X technologies.