Journal of Biomedical Informatics最新文献

{"title":"Unveiling pathology-related predictive uncertainty of glomerular lesion recognition using prototype learning","authors":"Qiming He ,&nbsp;Yingming Xu ,&nbsp;Qiang Huang ,&nbsp;Yanxia Wang ,&nbsp;Jing Ye ,&nbsp;Yonghong He ,&nbsp;Jing Li ,&nbsp;Lianghui Zhu ,&nbsp;Zhe Wang ,&nbsp;Tian Guan","doi":"10.1016/j.jbi.2024.104745","DOIUrl":"10.1016/j.jbi.2024.104745","url":null,"abstract":"<div><h3>Objective</h3><div>Recognizing glomerular lesions is essential in diagnosing chronic kidney disease. However, deep learning faces challenges due to the lesion heterogeneity, superposition, progression, and tissue incompleteness, leading to uncertainty in model predictions. Therefore, it is crucial to analyze pathology-related predictive uncertainty in glomerular lesion recognition and unveil its relationship with pathological properties and its impact on model performance.</div></div><div><h3>Methods</h3><div>This paper presents a novel framework for pathology-related predictive uncertainty analysis towards glomerular lesion recognition, including prototype learning based predictive uncertainty estimation, pathology-characterized correlation analysis and weight-redistributed prediction rectification. The prototype learning based predictive uncertainty estimation includes deep prototyping, affinity embedding, and multi-dimensional uncertainty fusion. The pathology-characterized correlation analysis is the first to use expert-based and learning- based approach to construct the pathology-related characterization of lesions and tissues. The weight-redistributed prediction rectification module performs reweighting- based lesion recognition.</div></div><div><h3>Results</h3><div>To validate the performance, extensive experiments were conducted. Based on the Spearman and Pearson correlation analysis, the proposed framework enables more efficient correlation analysis, and strong correlation with pathology-related characterization can be achieved (c index &gt; 0.6 and p &lt; 0.01). Furthermore, the prediction rectification module demonstrated improved lesion recognition performance across most metrics, with enhancements of up to 6.36 %.</div></div><div><h3>Conclusion</h3><div>The proposed predictive uncertainty analysis in glomerular lesion recognition offers a valuable approach for assessing computational pathology’s predictive uncertainty from a pathology-related perspective.</div></div><div><h3>Significance</h3><div>The paper provides a solution for pathology-related predictive uncertainty estimation in algorithm development and clinical practice.</div></div>","PeriodicalId":15263,"journal":{"name":"Journal of Biomedical Informatics","volume":"161 ","pages":"Article 104745"},"PeriodicalIF":4.0,"publicationDate":"2025-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142921240","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Early multi-cancer detection through deep learning: An anomaly detection approach using Variational Autoencoder","authors":"Innocent Tatchum Sado ,&nbsp;Louis Fippo Fitime ,&nbsp;Geraud Fokou Pelap ,&nbsp;Claude Tinku ,&nbsp;Gaelle Mireille Meudje ,&nbsp;Thomas Bouetou Bouetou","doi":"10.1016/j.jbi.2024.104751","DOIUrl":"10.1016/j.jbi.2024.104751","url":null,"abstract":"<div><div>Cancer is a disease that causes many deaths worldwide. The treatment of cancer is first and foremost a matter of detection, a treatment that is most effective when the disease is detected at an early stage. With the evolution of technology, several computer-aided diagnosis tools have been developed around cancer; several image-based cancer detection methods have been developed too. However, cancer detection faces many difficulties related to early detection which is crucial for patient survival rate. To detect cancer early, scientists have been using transcriptomic data. However, this presents some challenges such as unlabeled data, a large amount of data, and image-based techniques that only focus on one type of cancer. The purpose of this work is to develop a deep learning model that can effectively detect as soon as possible, specifically in the early stages, any type of cancer as an anomaly in transcriptomic data. This model must have the ability to act independently and not be restricted to any specific type of cancer. To achieve this goal, we modeled a deep neural network (a Variational Autoencoder) and then defined an algorithm for detecting anomalies in the output of the Variational Autoencoder. The Variational Autoencoder consists of an encoder and a decoder with a hidden layer. With the TCGA and GTEx data, we were able to train the model for six types of cancer using the Adam optimizer with decay learning for training, and a two-component loss function. As a result, we obtained the lowest value of accuracy 0.950, and the lowest value of recall 0.830. This research leads us to the design of a deep learning model for the detection of cancer as an anomaly in transcriptomic data.</div></div>","PeriodicalId":15263,"journal":{"name":"Journal of Biomedical Informatics","volume":"160 ","pages":"Article 104751"},"PeriodicalIF":4.0,"publicationDate":"2024-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142687219","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"How to identify patient perception of AI voice robots in the follow-up scenario? A multimodal identity perception method based on deep learning","authors":"Mingjie Liu ,&nbsp;Kuiyou Chen ,&nbsp;Qing Ye ,&nbsp;Hong Wu","doi":"10.1016/j.jbi.2024.104757","DOIUrl":"10.1016/j.jbi.2024.104757","url":null,"abstract":"<div><h3>Objectives</h3><div>Post-discharge follow-up stands as a critical component of post-diagnosis management, and the constraints of healthcare resources impede comprehensive manual follow-up. However, patients are less cooperative with AI follow-up calls or may even hang up once AI voice robots are perceived. To improve the effectiveness of follow-up, alternative measures should be taken when patients perceive AI voice robots. Therefore, identifying how patients perceive AI voice robots is crucial. This study aims to construct a multimodal identity perception model based on deep learning to identify how patients perceive AI voice robots.</div></div><div><h3>Methods</h3><div>Our dataset includes 2030 response audio recordings and corresponding texts from patients. We conduct comparative experiments and perform an ablation study. The proposed model employs a transfer learning approach, utilizing BERT and TextCNN for text feature extraction, AST and LSTM for audio feature extraction, and self-attention for feature fusion.</div></div><div><h3>Results</h3><div>Our model demonstrates superior performance against existing baselines, with a precision of 86.67%, an AUC of 84%, and an accuracy of 94.38%. Additionally, a generalization experiment was conducted using 144 patients’ response audio recordings and corresponding text data from other departments in the hospital, confirming the model’s robustness and effectiveness.</div></div><div><h3>Conclusion</h3><div>Our multimodal identity perception model can identify how patients perceive AI voice robots effectively. Identifying how patients perceive AI not only helps to optimize the follow-up process and improve patient cooperation, but also provides support for the evaluation and optimization of AI voice robots.</div></div>","PeriodicalId":15263,"journal":{"name":"Journal of Biomedical Informatics","volume":"160 ","pages":"Article 104757"},"PeriodicalIF":4.0,"publicationDate":"2024-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142780183","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Biomedical document-level relation extraction with thematic capture and localized entity pooling","authors":"Yuqing Li,&nbsp;Xinhui Shao","doi":"10.1016/j.jbi.2024.104756","DOIUrl":"10.1016/j.jbi.2024.104756","url":null,"abstract":"<div><div>In contrast to sentence-level relational extraction, document-level relation extraction poses greater challenges as a document typically contains multiple entities, and one entity may be associated with multiple other entities. Existing methods often rely on graph structures to capture path representations between entity pairs. However, this paper introduces a novel approach called local entity pooling that solely relies on the pre-training model to identify the bridge entity related to the current entity pair and generate the reasoning path representation. This technique effectively mitigates the multi-entity problem. Additionally, the model leverages the multi-entity and multi-label characteristics of the document to acquire the document’s thematic representation, thereby enhancing the document-level relation extraction task. Experimental evaluations conducted on two biomedical datasets, CDR and GDA. Our TCLEP (<strong>T</strong>hematic <strong>C</strong>apture and <strong>L</strong>ocalized <strong>E</strong>ntity <strong>P</strong>ooling) model achieved the Macro-F1 scores of 71.7% and 85.3%, respectively. Simultaneously, we incorporated local entity pooling and thematic capture modules into the state-of-the-art model, resulting in performance improvements of 1.5% and 0.2% on the respective datasets. These results highlight the advanced performance of our proposed approach.</div></div>","PeriodicalId":15263,"journal":{"name":"Journal of Biomedical Informatics","volume":"160 ","pages":"Article 104756"},"PeriodicalIF":4.0,"publicationDate":"2024-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142769374","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Taxonomy-based prompt engineering to generate synthetic drug-related patient portal messages","authors":"Natalie Wang ,&nbsp;Sukrit Treewaree ,&nbsp;Ayah Zirikly ,&nbsp;Yuzhi L. Lu ,&nbsp;Michelle H. Nguyen ,&nbsp;Bhavik Agarwal ,&nbsp;Jash Shah ,&nbsp;James Michael Stevenson ,&nbsp;Casey Overby Taylor","doi":"10.1016/j.jbi.2024.104752","DOIUrl":"10.1016/j.jbi.2024.104752","url":null,"abstract":"<div><h3>Objective:</h3><div>The objectives of this study were to: (1) create a corpus of synthetic drug-related patient portal messages to address the current lack of publicly available datasets for model development, (2) assess differences in language used and linguistics among the synthetic patient portal messages, and (3) assess the accuracy of patient-reported drug side effects for different racial groups.</div></div><div><h3>Methods:</h3><div>We leveraged a taxonomy for patient- and clinician-generated content to guide prompt engineering for synthetic drug-related patient portal messages. We generated two groups of messages: the first group (200 messages) used a subset of the taxonomy relevant to a broad range of drug-related messages and the second group (250 messages) used a subset of the taxonomy relevant to a narrow range of messages focused on side effects. Prompts also include one of five racial groups. Next, we assessed linguistic characteristics among message parts (subject, beginning, body, ending) across different prompt specifications (urgency, patient portal taxa, race). We also assessed the performance and frequency of patient-reported side effects across different racial groups and compared to data present in a real world data source (SIDER).</div></div><div><h3>Results:</h3><div>The study generated 450 synthetic patient portal messages, and we assessed linguistic patterns, accuracy of drug-side effect pairs, frequency of pairs compared to real world data. Linguistic analysis revealed variations in language usage and politeness and analysis of positive predictive values identified differences in symptoms reported based on urgency levels and racial groups in the prompt. We also found that low incident SIDER drug-side effect pairs were observed less frequently in our dataset.</div></div><div><h3>Conclusion:</h3><div>This study demonstrates the potential of synthetic patient portal messages as a valuable resource for healthcare research. After creating a corpus of synthetic drug-related patient portal messages, we identified significant language differences and provided evidence that drug-side effect pairs observed in messages are comparable to what is expected in real world settings.</div></div>","PeriodicalId":15263,"journal":{"name":"Journal of Biomedical Informatics","volume":"160 ","pages":"Article 104752"},"PeriodicalIF":4.0,"publicationDate":"2024-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142739561","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Sleep apnea test prediction based on Electronic Health Records","authors":"Lama Abu Tahoun ,&nbsp;Amit Shay Green ,&nbsp;Tal Patalon ,&nbsp;Yaron Dagan ,&nbsp;Robert Moskovitch","doi":"10.1016/j.jbi.2024.104737","DOIUrl":"10.1016/j.jbi.2024.104737","url":null,"abstract":"<div><div>The identification of Obstructive Sleep Apnea (OSA) is done by a Polysomnography test which is often done in later ages. Being able to notify potential insured members at earlier ages is desirable. For that, we develop predictive models that rely on Electronic Health Records (EHR) and predict whether a person will go through a sleep apnea test after the age of 50. A major challenge is the variability in EHR records in various insured members over the years, which this study investigates as well in the context of controls matching, and prediction. Since there are many temporal variables, the RankLi method was introduced for temporal variable selection. This approach employs the t-test to calculate a divergence score for each temporal variable between the target classes. We also investigate here the need to consider the number of EHR records, as part of control matching, and whether modeling separately for subgroups according to the number of EHR records is more effective. For each prediction task, we trained 4 different classifiers including 1-CNN, LSTM, Random Forest, and Logistic Regression, on data until the age of 40 or 50, and on several numbers of temporal variables. Using the number of EHR records for control matching was found crucial, and using learning models for subsets of the population according to the number of EHR records they have was found more effective. The deep learning models, particularly the 1-CNN, achieved the highest balanced accuracy and AUC scores in both male and female groups. In the male group, the highest results were also observed at age 50 with 100 temporal variables, resulting in a balanced accuracy of 90% and an AUC of 93%.</div></div>","PeriodicalId":15263,"journal":{"name":"Journal of Biomedical Informatics","volume":"160 ","pages":"Article 104737"},"PeriodicalIF":4.0,"publicationDate":"2024-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142568735","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Structural analysis and intelligent classification of clinical trial eligibility criteria based on deep learning and medical text mining","authors":"Yongzhong Han ,&nbsp;Qianmin Su ,&nbsp;Liang Liu ,&nbsp;Ying Li ,&nbsp;Jihan Huang","doi":"10.1016/j.jbi.2024.104753","DOIUrl":"10.1016/j.jbi.2024.104753","url":null,"abstract":"<div><h3>Objective:</h3><div>To enhance the efficiency, quality, and innovation capability of clinical trials, this paper introduces a novel model called CTEC-AC (Clinical Trial Eligibility Criteria Automatic Classification), aimed at structuring clinical trial eligibility criteria into computationally explainable classifications.</div></div><div><h3>Methods:</h3><div>We obtained detailed information on the latest 2,500 clinical trials from ClinicalTrials.gov, generating over 20,000 eligibility criteria data entries. To enhance the expressiveness of these criteria, we integrated two powerful methods: ClinicalBERT and MetaMap. The resulting enhanced features were used as input for a hierarchical clustering algorithm. Post-processing included expert validation of the algorithm’s output to ensure the accuracy of the constructed annotated eligibility text corpus. Ultimately, our model was employed to automate the classification of eligibility criteria.</div></div><div><h3>Results:</h3><div>We identified 31 distinct categories to summarize the eligibility criteria written by clinical researchers and uncovered common themes in how these criteria are expressed. Using our automated classification model on a labeled dataset, we achieved a macro-average F1 score of 0.94.</div></div><div><h3>Conclusion:</h3><div>This work can automatically extract structured representations from unstructured eligibility criteria text, significantly advancing the informatization of clinical trials. This, in turn, can significantly enhance the intelligence of automated participant recruitment for clinical researchers.</div></div>","PeriodicalId":15263,"journal":{"name":"Journal of Biomedical Informatics","volume":"160 ","pages":"Article 104753"},"PeriodicalIF":4.0,"publicationDate":"2024-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142739557","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Importance of variables from different time frames for predicting self-harm using health system data","authors":"Charles J. Wolock ,&nbsp;Brian D. Williamson ,&nbsp;Susan M. Shortreed ,&nbsp;Gregory E. Simon ,&nbsp;Karen J. Coleman ,&nbsp;Rodney Yeargans ,&nbsp;Brian K. Ahmedani ,&nbsp;Yihe Daida ,&nbsp;Frances L. Lynch ,&nbsp;Rebecca C. Rossom ,&nbsp;Rebecca A. Ziebell ,&nbsp;Maricela Cruz ,&nbsp;Robert D. Wellman ,&nbsp;R. Yates Coley","doi":"10.1016/j.jbi.2024.104750","DOIUrl":"10.1016/j.jbi.2024.104750","url":null,"abstract":"<div><h3>Objective:</h3><div>Self-harm risk prediction models developed using health system data (electronic health records and insurance claims information) often use patient information from up to several years prior to the index visit when the prediction is made. Measurements from some time periods may not be available for all patients. Using the framework of algorithm-agnostic variable importance, we study the predictive potential of variables corresponding to different time horizons prior to the index visit and demonstrate the application of variable importance techniques in the biomedical informatics setting.</div></div><div><h3>Materials and Methods:</h3><div>We use variable importance to quantify the potential of recent (up to three months before the index visit) and distant (more than one year before the index visit) patient mental health information for predicting self-harm risk using data from seven health systems. We quantify importance as the decrease in predictiveness when the variable set of interest is excluded from the prediction task. We define predictiveness using discriminative metrics: area under the receiver operating characteristic curve (AUC), sensitivity, and positive predictive value.</div></div><div><h3>Results:</h3><div>Mental health predictors corresponding to the three months prior to the index visit show strong signal of importance; in one setting, excluding these variables decreased AUC from 0.85 to 0.77. Predictors corresponding to more distant information were less important.</div></div><div><h3>Discussion:</h3><div>Predictors from the months immediately preceding the index visit are highly important. Implementation of self-harm prediction models may be challenging in settings where recent data are not completely available (e.g., due to lags in insurance claims processing) at the time a prediction is made.</div></div><div><h3>Conclusion:</h3><div>Clinically derived variables from different time frames exhibit varying levels of importance for predicting self-harm. Variable importance analyses can inform whether and how to implement risk prediction models into clinical practice given real-world data limitations. These analyses be applied more broadly in biomedical informatics research to provide insight into general clinical risk prediction tasks.</div></div>","PeriodicalId":15263,"journal":{"name":"Journal of Biomedical Informatics","volume":"160 ","pages":"Article 104750"},"PeriodicalIF":4.0,"publicationDate":"2024-11-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142668134","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Machine learning approaches for the discovery of clinical pathways from patient data: A systematic review","authors":"Lillian Muyama ,&nbsp;Antoine Neuraz ,&nbsp;Adrien Coulet","doi":"10.1016/j.jbi.2024.104746","DOIUrl":"10.1016/j.jbi.2024.104746","url":null,"abstract":"<div><h3>Background:</h3><div>Clinical pathways are sequences of events followed during the clinical care of a group of patients who meet pre-defined criteria. They have many applications ranging from healthcare evaluation and optimization to clinical decision support. These pathways can be discovered from existing healthcare data, in particular with machine learning which is a family of methods used to learn patterns from data. This review provides a comprehensive overview of the literature concerning the use of machine learning methods for clinical pathway discovery from patient data.</div></div><div><h3>Methods:</h3><div>Guided by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) method , we conducted a systematic review of the existing literature. We searched 6 databases, <em>i.e.</em>, ACM Digital Library, ScienceDirect, Web of Science, PubMed, IEEE Xplore, and Scopus spanning from January 2004 to December 2023 using search terms pertinent to clinical pathways and their development. Subsequently, the retrieved papers were analyzed to assess their relevance to the scope of this study.</div></div><div><h3>Results:</h3><div>In total, 131 papers that met the specified inclusion criteria were identified. These papers expressed diverse motivations behind data-driven clinical pathway discovery ranging from knowledge discovery to conformance checking with established clinical guidelines (derived from existing literature and clinical experts). Notably, the predominant methods employed (67.2%, <span><math><mi>n</mi></math></span>=88) involved unsupervised machine learning techniques, such as clustering and process mining.</div></div><div><h3>Conclusions:</h3><div>Relevant clinical pathways can be discovered from patient data using machine learning methods, with the desirable potential to aid clinical decision-making in healthcare. However, to reach this objective, the methods used to discover pathways should be reproducible, and rigorous performance evaluation by clinical experts needs to be conducted for validation.</div></div>","PeriodicalId":15263,"journal":{"name":"Journal of Biomedical Informatics","volume":"160 ","pages":"Article 104746"},"PeriodicalIF":4.0,"publicationDate":"2024-11-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142621220","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Cross-Modal self-supervised vision language pre-training with multiple objectives for medical visual question answering","authors":"Gang Liu ,&nbsp;Jinlong He ,&nbsp;Pengfei Li ,&nbsp;Zixu Zhao ,&nbsp;Shenjun Zhong","doi":"10.1016/j.jbi.2024.104748","DOIUrl":"10.1016/j.jbi.2024.104748","url":null,"abstract":"<div><div>Medical Visual Question Answering (VQA) is a task that aims to provide answers to questions about medical images, which utilizes both visual and textual information in the reasoning process. The absence of large-scale annotated medical VQA datasets presents a formidable obstacle to training a medical VQA model from scratch in an end-to-end manner. Existing works have been using image captioning dataset in the pre-training stage and fine-tuning to downstream VQA tasks. Following the same paradigm, we use a collection of public medical image captioning datasets to pre-train multimodality models in a self-supervised setup, and fine-tune to downstream medical VQA tasks. In the work, we propose a method that featured with Cross-Modal pre-training with Multiple Objectives (CMMO), which includes masked image modeling, masked language modeling, image-text matching, and image-text contrastive learning. The proposed method is designed to associate the visual features of medical images with corresponding medical concepts in captions, for learning aligned vision and language feature representations, and multi-modal interactions. The experimental results reveal that our proposed CMMO method outperforms state-of-the-art methods on three public medical VQA datasets, showing absolute improvements of 2.6%, 0.9%, and 4.0% on the VQA-RAD, PathVQA, and SLAKE dataset, respectively. We also conduct comprehensive ablation studies to validate our method, and visualize the attention maps which show a strong interpretability. The code and pre-trained weights will be released at <span><span>https://github.com/pengfeiliHEU/CMMO</span><svg><path></path></svg></span>.</div></div>","PeriodicalId":15263,"journal":{"name":"Journal of Biomedical Informatics","volume":"160 ","pages":"Article 104748"},"PeriodicalIF":4.0,"publicationDate":"2024-11-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142621216","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}