{"title":"Improving Image-Text Matching by Integrating Word Sense Disambiguation","authors":"Xiao Pu;Ping Yang;Lin Yuan;Xinbo Gao","doi":"10.1109/LSP.2024.3466992","DOIUrl":null,"url":null,"abstract":"This letter presents a novel approach to enhance image-text matching by incorporating word sense disambiguation (WSD) within the text encoder. Our method explicitly models the senses of potentially ambiguous words, refining the semantic understanding between images and text. We introduce a sense-aware mechanism for image-text alignment by integrating a lightweight WSD component into the matching framework, optimizing both tasks simultaneously. Our WSD module operates on extensive word contexts, leveraging the power of graph attention networks (GAT), and distills knowledge from a substantially larger pre-trained WSD model through multi-task learning. Our experiments demonstrate the effectiveness of augmenting original word embeddings with sense representations derived from our WSD approach. We systematically evaluate our method against several baselines and state-of-the-art approaches on two widely-used image-text matching benchmarks: MS-COCO and Flickr30K. The results illustrate significant improvements in matching accuracy, highlighting the efficacy of our proposed approach.","PeriodicalId":13154,"journal":{"name":"IEEE Signal Processing Letters","volume":null,"pages":null},"PeriodicalIF":3.2000,"publicationDate":"2024-09-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"IEEE Signal Processing Letters","FirstCategoryId":"5","ListUrlMain":"https://ieeexplore.ieee.org/document/10693344/","RegionNum":2,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"ENGINEERING, ELECTRICAL & ELECTRONIC","Score":null,"Total":0}
引用次数: 0
Abstract
This letter presents a novel approach to enhance image-text matching by incorporating word sense disambiguation (WSD) within the text encoder. Our method explicitly models the senses of potentially ambiguous words, refining the semantic understanding between images and text. We introduce a sense-aware mechanism for image-text alignment by integrating a lightweight WSD component into the matching framework, optimizing both tasks simultaneously. Our WSD module operates on extensive word contexts, leveraging the power of graph attention networks (GAT), and distills knowledge from a substantially larger pre-trained WSD model through multi-task learning. Our experiments demonstrate the effectiveness of augmenting original word embeddings with sense representations derived from our WSD approach. We systematically evaluate our method against several baselines and state-of-the-art approaches on two widely-used image-text matching benchmarks: MS-COCO and Flickr30K. The results illustrate significant improvements in matching accuracy, highlighting the efficacy of our proposed approach.
期刊介绍:
The IEEE Signal Processing Letters is a monthly, archival publication designed to provide rapid dissemination of original, cutting-edge ideas and timely, significant contributions in signal, image, speech, language and audio processing. Papers published in the Letters can be presented within one year of their appearance in signal processing conferences such as ICASSP, GlobalSIP and ICIP, and also in several workshop organized by the Signal Processing Society.