{"title":"基于近红外光谱GADF和RGB图像的木材树种识别深度学习多模态融合框架","authors":"Xi Pan, Zhiming Yu, Zhong Yang","doi":"10.1515/hf-2023-0062","DOIUrl":null,"url":null,"abstract":"Abstract Accurate and rapid wood species identification is vital for wood utilization and trade. This goal is achievable with the fast development of deep learning (DL). Several studies have been published related to this topic; however, they were limited by their generalization performance in practical applications. Therefore, this study proposed a DL multimodal fusion framework to bridge this gap. The study utilized a state-of-the-art convolutional neural network (CNN) to simultaneously extract both short-wavelength near-infrared (NIR) spectra and RGB image feature, fully leveraging the advantages of both data types. Using portable devices for collecting spectra and image data enhances the feasibility of onsite rapid identification. In particular, a two-branch CNN framework was developed to extract spectra and image features. For NIR spectra feature extraction, 1 dimensional NIR (1D NIR) spectra were innovatively encoded as 2 dimensional (2D) images using the Gramian angular difference field (GADF) method. This representation enhances better data alignment with CNN operations, facilitating more robust discriminative feature extraction. Moreover, wood’s spectral and image features were fused at the full connection layer for species identification. In the experimental phase conducted on 16 difficult-to-distinguish wood samples from the Lauraceae family, all achieved identification metrics results exceed 99 %. The findings illustrate that the proposed multimodal fusion framework effectively extracts and fully integrates the wood’s features, thereby, improving wood species identification.","PeriodicalId":13083,"journal":{"name":"Holzforschung","volume":" 38","pages":"0"},"PeriodicalIF":2.2000,"publicationDate":"2023-11-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"A deep learning multimodal fusion framework for wood species identification using near-infrared spectroscopy GADF and RGB image\",\"authors\":\"Xi Pan, Zhiming Yu, Zhong Yang\",\"doi\":\"10.1515/hf-2023-0062\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"Abstract Accurate and rapid wood species identification is vital for wood utilization and trade. This goal is achievable with the fast development of deep learning (DL). Several studies have been published related to this topic; however, they were limited by their generalization performance in practical applications. Therefore, this study proposed a DL multimodal fusion framework to bridge this gap. The study utilized a state-of-the-art convolutional neural network (CNN) to simultaneously extract both short-wavelength near-infrared (NIR) spectra and RGB image feature, fully leveraging the advantages of both data types. Using portable devices for collecting spectra and image data enhances the feasibility of onsite rapid identification. In particular, a two-branch CNN framework was developed to extract spectra and image features. For NIR spectra feature extraction, 1 dimensional NIR (1D NIR) spectra were innovatively encoded as 2 dimensional (2D) images using the Gramian angular difference field (GADF) method. This representation enhances better data alignment with CNN operations, facilitating more robust discriminative feature extraction. Moreover, wood’s spectral and image features were fused at the full connection layer for species identification. In the experimental phase conducted on 16 difficult-to-distinguish wood samples from the Lauraceae family, all achieved identification metrics results exceed 99 %. The findings illustrate that the proposed multimodal fusion framework effectively extracts and fully integrates the wood’s features, thereby, improving wood species identification.\",\"PeriodicalId\":13083,\"journal\":{\"name\":\"Holzforschung\",\"volume\":\" 38\",\"pages\":\"0\"},\"PeriodicalIF\":2.2000,\"publicationDate\":\"2023-11-09\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Holzforschung\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.1515/hf-2023-0062\",\"RegionNum\":3,\"RegionCategory\":\"农林科学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q2\",\"JCRName\":\"FORESTRY\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Holzforschung","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1515/hf-2023-0062","RegionNum":3,"RegionCategory":"农林科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"FORESTRY","Score":null,"Total":0}
A deep learning multimodal fusion framework for wood species identification using near-infrared spectroscopy GADF and RGB image
Abstract Accurate and rapid wood species identification is vital for wood utilization and trade. This goal is achievable with the fast development of deep learning (DL). Several studies have been published related to this topic; however, they were limited by their generalization performance in practical applications. Therefore, this study proposed a DL multimodal fusion framework to bridge this gap. The study utilized a state-of-the-art convolutional neural network (CNN) to simultaneously extract both short-wavelength near-infrared (NIR) spectra and RGB image feature, fully leveraging the advantages of both data types. Using portable devices for collecting spectra and image data enhances the feasibility of onsite rapid identification. In particular, a two-branch CNN framework was developed to extract spectra and image features. For NIR spectra feature extraction, 1 dimensional NIR (1D NIR) spectra were innovatively encoded as 2 dimensional (2D) images using the Gramian angular difference field (GADF) method. This representation enhances better data alignment with CNN operations, facilitating more robust discriminative feature extraction. Moreover, wood’s spectral and image features were fused at the full connection layer for species identification. In the experimental phase conducted on 16 difficult-to-distinguish wood samples from the Lauraceae family, all achieved identification metrics results exceed 99 %. The findings illustrate that the proposed multimodal fusion framework effectively extracts and fully integrates the wood’s features, thereby, improving wood species identification.
期刊介绍:
Holzforschung is an international scholarly journal that publishes cutting-edge research on the biology, chemistry, physics and technology of wood and wood components. High quality papers about biotechnology and tree genetics are also welcome. Rated year after year as one of the top scientific journals in the category of Pulp and Paper (ISI Journal Citation Index), Holzforschung represents innovative, high quality basic and applied research. The German title reflects the journal''s origins in a long scientific tradition, but all articles are published in English to stimulate and promote cooperation between experts all over the world. Ahead-of-print publishing ensures fastest possible knowledge transfer.