Efficient knowledge distillation for hybrid models: A vision transformer-convolutional neural network to convolutional neural network approach for classifying remote sensing images

IF 1.5 Q3 AUTOMATION & CONTROL SYSTEMS IET Cybersystems and Robotics Pub Date : 2024-07-10 DOI:10.1049/csy2.12120
Huaxiang Song, Yuxuan Yuan, Zhiwei Ouyang, Yu Yang, Hui Xiang
{"title":"Efficient knowledge distillation for hybrid models: A vision transformer-convolutional neural network to convolutional neural network approach for classifying remote sensing images","authors":"Huaxiang Song,&nbsp;Yuxuan Yuan,&nbsp;Zhiwei Ouyang,&nbsp;Yu Yang,&nbsp;Hui Xiang","doi":"10.1049/csy2.12120","DOIUrl":null,"url":null,"abstract":"<p>In various fields, knowledge distillation (KD) techniques that combine vision transformers (ViTs) and convolutional neural networks (CNNs) as a hybrid teacher have shown remarkable results in classification. However, in the realm of remote sensing images (RSIs), existing KD research studies are not only scarce but also lack competitiveness. This issue significantly impedes the deployment of the notable advantages of ViTs and CNNs. To tackle this, the authors introduce a novel hybrid-model KD approach named HMKD-Net, which comprises a CNN-ViT ensemble teacher and a CNN student. Contrary to popular opinion, the authors posit that the sparsity in RSI data distribution limits the effectiveness and efficiency of hybrid-model knowledge transfer. As a solution, a simple yet innovative method to handle variances during the KD phase is suggested, leading to substantial enhancements in the effectiveness and efficiency of hybrid knowledge transfer. The authors assessed the performance of HMKD-Net on three RSI datasets. The findings indicate that HMKD-Net significantly outperforms other cutting-edge methods while maintaining a significantly smaller size. Specifically, HMKD-Net exceeds other KD-based methods with a maximum accuracy improvement of 22.8% across various datasets. As ablation experiments indicated, HMKD-Net has cut down on time expenses by about 80% in the KD process. This research study validates that the hybrid-model KD technique can be more effective and efficient if the data distribution sparsity in RSIs is well handled.</p>","PeriodicalId":34110,"journal":{"name":"IET Cybersystems and Robotics","volume":null,"pages":null},"PeriodicalIF":1.5000,"publicationDate":"2024-07-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1049/csy2.12120","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"IET Cybersystems and Robotics","FirstCategoryId":"1085","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1049/csy2.12120","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"AUTOMATION & CONTROL SYSTEMS","Score":null,"Total":0}
引用次数: 0

Abstract

In various fields, knowledge distillation (KD) techniques that combine vision transformers (ViTs) and convolutional neural networks (CNNs) as a hybrid teacher have shown remarkable results in classification. However, in the realm of remote sensing images (RSIs), existing KD research studies are not only scarce but also lack competitiveness. This issue significantly impedes the deployment of the notable advantages of ViTs and CNNs. To tackle this, the authors introduce a novel hybrid-model KD approach named HMKD-Net, which comprises a CNN-ViT ensemble teacher and a CNN student. Contrary to popular opinion, the authors posit that the sparsity in RSI data distribution limits the effectiveness and efficiency of hybrid-model knowledge transfer. As a solution, a simple yet innovative method to handle variances during the KD phase is suggested, leading to substantial enhancements in the effectiveness and efficiency of hybrid knowledge transfer. The authors assessed the performance of HMKD-Net on three RSI datasets. The findings indicate that HMKD-Net significantly outperforms other cutting-edge methods while maintaining a significantly smaller size. Specifically, HMKD-Net exceeds other KD-based methods with a maximum accuracy improvement of 22.8% across various datasets. As ablation experiments indicated, HMKD-Net has cut down on time expenses by about 80% in the KD process. This research study validates that the hybrid-model KD technique can be more effective and efficient if the data distribution sparsity in RSIs is well handled.

Abstract Image

查看原文
分享 分享
微信好友 朋友圈 QQ好友 复制链接
本刊更多论文
混合模型的高效知识提炼:用于遥感图像分类的视觉转换器-卷积神经网络-卷积神经网络方法
在各个领域,结合视觉转换器(ViT)和卷积神经网络(CNN)作为混合教师的知识提炼(KD)技术在分类方面取得了显著效果。然而,在遥感图像(RSI)领域,现有的知识提炼研究不仅数量稀少,而且缺乏竞争力。这一问题严重阻碍了 ViT 和 CNN 显著优势的发挥。为解决这一问题,作者引入了一种名为 HMKD-Net 的新型混合模型 KD 方法,该方法由 CNN-ViT 组合教师和 CNN 学生组成。与流行观点相反,作者认为 RSI 数据分布的稀疏性限制了混合模型知识转移的效果和效率。作为解决方案,作者提出了一种简单而创新的方法来处理 KD 阶段的差异,从而大大提高了混合知识转移的效果和效率。作者在三个 RSI 数据集上评估了 HMKD-Net 的性能。研究结果表明,HMKD-Net 的性能明显优于其他前沿方法,同时体积明显缩小。具体来说,HMKD-Net 超越了其他基于 KD 的方法,在各种数据集上的准确率最高提高了 22.8%。消融实验表明,HMKD-Net 在 KD 过程中减少了约 80% 的时间支出。这项研究验证了,如果能很好地处理 RSI 中的数据分布稀疏性,混合模型 KD 技术将更加有效和高效。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 去求助
来源期刊
IET Cybersystems and Robotics
IET Cybersystems and Robotics Computer Science-Information Systems
CiteScore
3.70
自引率
0.00%
发文量
31
审稿时长
34 weeks
期刊最新文献
Correction-enabled reversible data hiding with pixel repetition for high embedding rate and quality preservation Anti-sloshing control: Flatness-based trajectory planning and tracking control with an integrated extended state observer Internal and external disturbances aware motion planning and control for quadrotors Multi-feature fusion and memory-based mobile robot target tracking system Efficient knowledge distillation for hybrid models: A vision transformer-convolutional neural network to convolutional neural network approach for classifying remote sensing images
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
已复制链接
已复制链接
快去分享给好友吧!
我知道了
×
扫码分享
扫码分享
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1