{"title":"Brain tissue classification in hyperspectral images using multistage diffusion features and transformer.","authors":"Neetu Sigger, Tuan T Nguyen, Gianluca Tozzi","doi":"10.1111/jmi.13372","DOIUrl":null,"url":null,"abstract":"<p><p>Brain surgery is a widely practised and effective treatment for brain tumours, but accurately identifying and classifying tumour boundaries is crucial to maximise resection and avoid neurological complications. This precision in classification is essential for guiding surgical decisions and subsequent treatment planning. Hyperspectral (HS) imaging (HSI) is an emerging multidimensional optical imaging method that captures detailed spectral information across multiple wavelengths, allowing for the identification of nuanced differences in tissue composition, with the potential to enhance intraoperative tissue classification. However, current frameworks often require retraining models for each HSI to extract meaningful features, resulting in long processing times and high computational costs. Additionally, most methods utilise the deep semantic features at the end of the network for classification, ignoring the spatial details contained in the shallow features. To overcome these challenges, we propose a novel approach called MedDiffHSI, which combines diffusion and transformer techniques. Our method involves training an unsupervised learning framework based on the diffusion model to extract high-level and low-level spectral-spatial features from HSI. This approach eliminates the need for retraining of spectral-spatial feature learning model, thereby reducing time complexity. We then extract intermediate multistage features from different timestamps for classification using a pretrained denoising U-Net. To fully explore and exploit the rich contextual semantics and textual information hidden in the extracted diffusion feature, we utilise a spectral-spatial attention module. This module not only learns multistage information about features at different depths, but also extracts and enhances effective information from them. Finally, we employ a supervised transformer-based classifier with weighted majority voting (WMV) to perform the HSI classification. To validate our approach, we conduct comprehensive experiments on in vivo brain database data sets and also extend the analysis to include additional HSI data sets for breast cancer to evaluate the framework performance across different types of tissue. The results demonstrate that our framework outperforms existing approaches by using minimal training samples (5%) while achieving state-of-the-art performance.</p>","PeriodicalId":16484,"journal":{"name":"Journal of microscopy","volume":" ","pages":""},"PeriodicalIF":1.5000,"publicationDate":"2024-11-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of microscopy","FirstCategoryId":"5","ListUrlMain":"https://doi.org/10.1111/jmi.13372","RegionNum":4,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"MICROSCOPY","Score":null,"Total":0}
引用次数: 0
Abstract
Brain surgery is a widely practised and effective treatment for brain tumours, but accurately identifying and classifying tumour boundaries is crucial to maximise resection and avoid neurological complications. This precision in classification is essential for guiding surgical decisions and subsequent treatment planning. Hyperspectral (HS) imaging (HSI) is an emerging multidimensional optical imaging method that captures detailed spectral information across multiple wavelengths, allowing for the identification of nuanced differences in tissue composition, with the potential to enhance intraoperative tissue classification. However, current frameworks often require retraining models for each HSI to extract meaningful features, resulting in long processing times and high computational costs. Additionally, most methods utilise the deep semantic features at the end of the network for classification, ignoring the spatial details contained in the shallow features. To overcome these challenges, we propose a novel approach called MedDiffHSI, which combines diffusion and transformer techniques. Our method involves training an unsupervised learning framework based on the diffusion model to extract high-level and low-level spectral-spatial features from HSI. This approach eliminates the need for retraining of spectral-spatial feature learning model, thereby reducing time complexity. We then extract intermediate multistage features from different timestamps for classification using a pretrained denoising U-Net. To fully explore and exploit the rich contextual semantics and textual information hidden in the extracted diffusion feature, we utilise a spectral-spatial attention module. This module not only learns multistage information about features at different depths, but also extracts and enhances effective information from them. Finally, we employ a supervised transformer-based classifier with weighted majority voting (WMV) to perform the HSI classification. To validate our approach, we conduct comprehensive experiments on in vivo brain database data sets and also extend the analysis to include additional HSI data sets for breast cancer to evaluate the framework performance across different types of tissue. The results demonstrate that our framework outperforms existing approaches by using minimal training samples (5%) while achieving state-of-the-art performance.
期刊介绍:
The Journal of Microscopy is the oldest journal dedicated to the science of microscopy and the only peer-reviewed publication of the Royal Microscopical Society. It publishes papers that report on the very latest developments in microscopy such as advances in microscopy techniques or novel areas of application. The Journal does not seek to publish routine applications of microscopy or specimen preparation even though the submission may otherwise have a high scientific merit.
The scope covers research in the physical and biological sciences and covers imaging methods using light, electrons, X-rays and other radiations as well as atomic force and near field techniques. Interdisciplinary research is welcome. Papers pertaining to microscopy are also welcomed on optical theory, spectroscopy, novel specimen preparation and manipulation methods and image recording, processing and analysis including dynamic analysis of living specimens.
Publication types include full papers, hot topic fast tracked communications and review articles. Authors considering submitting a review article should contact the editorial office first.