{"title":"通过轻量级多流交叉和知识蒸馏实现高效 RGB-D 室内场景解析解决方案","authors":"Wujie Zhou;Yuming Zhang;Weiqing Yan;Lv Ye","doi":"10.1109/JSTSP.2024.3400030","DOIUrl":null,"url":null,"abstract":"The rapid progression of convolutional neural networks (CNNs) has significantly improved indoor scene parsing, transforming the fields of robotics, autonomous navigation, augmented reality, and surveillance. Currently, societal demand is propelling these technologies toward integration into mobile smart device applications. However, the processing capabilities of mobile devices cannot support the comprehensive system requirements of CNNs, which poses a challenge for several deep-learning applications. One promising solution to this predicament is the deployment of lightweight student networks. These streamlined networks learn from their robust, cloud-based counterparts—that is, teacher networks—through knowledge distillation (KD). This facilitates a reduction in parameter count and optimizes student classification. Furthermore, a lightweight multiflow intersection network (LMINet) is proposed and developed for red–green–blue–depth (RGB-D) indoor scene parsing. The proposed method relies on dual-frequency KD (FKD) and compression KD (CKD) methods. A multiflow intersection module is introduced to efficiently integrate feature information from disparate layers. To maximize the performance of lightweight LMINet student (LMINet-S) networks, the FKD module employs a discrete cosine transform to capture feature information from different frequencies, whereas the CKD module compresses the features of diverse layers and distills their corresponding dimensions. Experiments using the NYUDv2 and SUN-RGBD datasets demonstrate that our LMINet teacher (LMINet-T) model, LMINet-S (without KD), and LMINet-S* (LMINet-S with KD) outperform state-of-the-art scene-parsing tools without increasing the parameter count (26.2M). Consequently, the technology is now closer to integration into mobile devices.","PeriodicalId":13038,"journal":{"name":"IEEE Journal of Selected Topics in Signal Processing","volume":null,"pages":null},"PeriodicalIF":8.7000,"publicationDate":"2024-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"An Efficient RGB-D Indoor Scene-Parsing Solution via Lightweight Multiflow Intersection and Knowledge Distillation\",\"authors\":\"Wujie Zhou;Yuming Zhang;Weiqing Yan;Lv Ye\",\"doi\":\"10.1109/JSTSP.2024.3400030\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"The rapid progression of convolutional neural networks (CNNs) has significantly improved indoor scene parsing, transforming the fields of robotics, autonomous navigation, augmented reality, and surveillance. Currently, societal demand is propelling these technologies toward integration into mobile smart device applications. However, the processing capabilities of mobile devices cannot support the comprehensive system requirements of CNNs, which poses a challenge for several deep-learning applications. One promising solution to this predicament is the deployment of lightweight student networks. These streamlined networks learn from their robust, cloud-based counterparts—that is, teacher networks—through knowledge distillation (KD). This facilitates a reduction in parameter count and optimizes student classification. Furthermore, a lightweight multiflow intersection network (LMINet) is proposed and developed for red–green–blue–depth (RGB-D) indoor scene parsing. The proposed method relies on dual-frequency KD (FKD) and compression KD (CKD) methods. A multiflow intersection module is introduced to efficiently integrate feature information from disparate layers. To maximize the performance of lightweight LMINet student (LMINet-S) networks, the FKD module employs a discrete cosine transform to capture feature information from different frequencies, whereas the CKD module compresses the features of diverse layers and distills their corresponding dimensions. Experiments using the NYUDv2 and SUN-RGBD datasets demonstrate that our LMINet teacher (LMINet-T) model, LMINet-S (without KD), and LMINet-S* (LMINet-S with KD) outperform state-of-the-art scene-parsing tools without increasing the parameter count (26.2M). Consequently, the technology is now closer to integration into mobile devices.\",\"PeriodicalId\":13038,\"journal\":{\"name\":\"IEEE Journal of Selected Topics in Signal Processing\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":8.7000,\"publicationDate\":\"2024-03-13\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"IEEE Journal of Selected Topics in Signal Processing\",\"FirstCategoryId\":\"5\",\"ListUrlMain\":\"https://ieeexplore.ieee.org/document/10529499/\",\"RegionNum\":1,\"RegionCategory\":\"工程技术\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"ENGINEERING, ELECTRICAL & ELECTRONIC\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"IEEE Journal of Selected Topics in Signal Processing","FirstCategoryId":"5","ListUrlMain":"https://ieeexplore.ieee.org/document/10529499/","RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"ENGINEERING, ELECTRICAL & ELECTRONIC","Score":null,"Total":0}
An Efficient RGB-D Indoor Scene-Parsing Solution via Lightweight Multiflow Intersection and Knowledge Distillation
The rapid progression of convolutional neural networks (CNNs) has significantly improved indoor scene parsing, transforming the fields of robotics, autonomous navigation, augmented reality, and surveillance. Currently, societal demand is propelling these technologies toward integration into mobile smart device applications. However, the processing capabilities of mobile devices cannot support the comprehensive system requirements of CNNs, which poses a challenge for several deep-learning applications. One promising solution to this predicament is the deployment of lightweight student networks. These streamlined networks learn from their robust, cloud-based counterparts—that is, teacher networks—through knowledge distillation (KD). This facilitates a reduction in parameter count and optimizes student classification. Furthermore, a lightweight multiflow intersection network (LMINet) is proposed and developed for red–green–blue–depth (RGB-D) indoor scene parsing. The proposed method relies on dual-frequency KD (FKD) and compression KD (CKD) methods. A multiflow intersection module is introduced to efficiently integrate feature information from disparate layers. To maximize the performance of lightweight LMINet student (LMINet-S) networks, the FKD module employs a discrete cosine transform to capture feature information from different frequencies, whereas the CKD module compresses the features of diverse layers and distills their corresponding dimensions. Experiments using the NYUDv2 and SUN-RGBD datasets demonstrate that our LMINet teacher (LMINet-T) model, LMINet-S (without KD), and LMINet-S* (LMINet-S with KD) outperform state-of-the-art scene-parsing tools without increasing the parameter count (26.2M). Consequently, the technology is now closer to integration into mobile devices.
期刊介绍:
The IEEE Journal of Selected Topics in Signal Processing (JSTSP) focuses on the Field of Interest of the IEEE Signal Processing Society, which encompasses the theory and application of various signal processing techniques. These techniques include filtering, coding, transmitting, estimating, detecting, analyzing, recognizing, synthesizing, recording, and reproducing signals using digital or analog devices. The term "signal" covers a wide range of data types, including audio, video, speech, image, communication, geophysical, sonar, radar, medical, musical, and others.
The journal format allows for in-depth exploration of signal processing topics, enabling the Society to cover both established and emerging areas. This includes interdisciplinary fields such as biomedical engineering and language processing, as well as areas not traditionally associated with engineering.