Pub Date : 2023-03-01DOI: 10.1109/ICNLP58431.2023.00019
Lan Rong, Shumin Wang, He Hu, Zhao Feng, Haiyan Yu, Zhang Lu
In order to address the issue that the center-free fuzzy c-means (CFFCM) clustering algorithm does not consider the texture features and spatial information of pixels, and the time complexity is too high, a center-free intuitionistic fuzzy c-means clustering algorithm based on similarity of hybrid spatial membership for image segmentation is proposed. In the proposed algorithm, the voting model is used to generate intuitionistic fuzzy sets (IFS), and the generated hesitation degree and membership degree are combined with spatial information to design a spatial intuitionistic membership degree similarity model. This model can deal with the similarity between pixels and classes in gray information, so the segmentation efficiency is improved. At the same time, the intuitionistic fuzzy local binary pattern (IFLBP) operator is used to extract the image texture information and introduce it into the objective function. Spatial membership similarity model is used to process texture information and improve the segmentation accuracy of the algorithm. The results of simulation experiment show that the proposed has advantages in both visual effect and evaluation index.
{"title":"Center-free intuitionistic fuzzy c-means clustering algorithm based on similarity of hybrid spatial membership for image segmentation","authors":"Lan Rong, Shumin Wang, He Hu, Zhao Feng, Haiyan Yu, Zhang Lu","doi":"10.1109/ICNLP58431.2023.00019","DOIUrl":"https://doi.org/10.1109/ICNLP58431.2023.00019","url":null,"abstract":"In order to address the issue that the center-free fuzzy c-means (CFFCM) clustering algorithm does not consider the texture features and spatial information of pixels, and the time complexity is too high, a center-free intuitionistic fuzzy c-means clustering algorithm based on similarity of hybrid spatial membership for image segmentation is proposed. In the proposed algorithm, the voting model is used to generate intuitionistic fuzzy sets (IFS), and the generated hesitation degree and membership degree are combined with spatial information to design a spatial intuitionistic membership degree similarity model. This model can deal with the similarity between pixels and classes in gray information, so the segmentation efficiency is improved. At the same time, the intuitionistic fuzzy local binary pattern (IFLBP) operator is used to extract the image texture information and introduce it into the objective function. Spatial membership similarity model is used to process texture information and improve the segmentation accuracy of the algorithm. The results of simulation experiment show that the proposed has advantages in both visual effect and evaluation index.","PeriodicalId":53637,"journal":{"name":"Icon","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"80167774","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-03-01DOI: 10.1109/ICNLP58431.2023.00056
Jin Cai, Guoyong Cai
Visual Question Answering is a multimedia understanding task that gives an image and natural language questions related to its content and allows the computer to answer them correctly. The early visual question answering models often ignore the emotional information in the image, resulting in insufficient performance in answering emotional-related questions; on the other hand, the existing visual question answering models that integrate emotional information do not make full use of the key areas of the image and text keywords, and do not understand fine-grained questions deeply enough, resulting in low accuracy. In order to fully integrate image emotional information into the visual question answering model and enhance the ability of the model to answer questions, a multimodal visual question answering model (IEMVQA) enhanced by image emotional information is proposed, and experiments are carried out on the visual question answering benchmark dataset. The final experiment shows that the IEMVQA model performs better than other comparison methods in comprehensive indicators, and verifies the effectiveness of using emotional information to assist visual question answering model.
{"title":"Multimodal Visual Question Answering Model Enhanced with Image Emotional Information","authors":"Jin Cai, Guoyong Cai","doi":"10.1109/ICNLP58431.2023.00056","DOIUrl":"https://doi.org/10.1109/ICNLP58431.2023.00056","url":null,"abstract":"Visual Question Answering is a multimedia understanding task that gives an image and natural language questions related to its content and allows the computer to answer them correctly. The early visual question answering models often ignore the emotional information in the image, resulting in insufficient performance in answering emotional-related questions; on the other hand, the existing visual question answering models that integrate emotional information do not make full use of the key areas of the image and text keywords, and do not understand fine-grained questions deeply enough, resulting in low accuracy. In order to fully integrate image emotional information into the visual question answering model and enhance the ability of the model to answer questions, a multimodal visual question answering model (IEMVQA) enhanced by image emotional information is proposed, and experiments are carried out on the visual question answering benchmark dataset. The final experiment shows that the IEMVQA model performs better than other comparison methods in comprehensive indicators, and verifies the effectiveness of using emotional information to assist visual question answering model.","PeriodicalId":53637,"journal":{"name":"Icon","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"73803245","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-03-01DOI: 10.1109/ICNLP58431.2023.00011
Md. Tanvir Rouf Shawon, G. M. Shahariar, F. Shah, Mohammad Shafiul Alam, M. S. Mahbub
This paper investigates the potential of semi-supervised Generative Adversarial Networks (GANs) to fine-tune pretrained language models in order to classify Bengali fake reviews from real reviews with a few annotated data. With the rise of social media and e-commerce, the ability to detect fake or deceptive reviews is becoming increasingly important in order to protect consumers from being misled by false information. Any machine learning model will have trouble identifying a fake review, especially for a low resource language like Bengali. We have demonstrated that the proposed semi-supervised GAN-LM architecture (generative adversarial network on top of a pretrained language model) is a viable solution in classifying Bengali fake reviews as the experimental results suggest that even with only 1024 annotated samples, BanglaBERT with semi-supervised GAN (SSGAN) achieved an accuracy of 83.59% and a f1-score of 84.89% outperforming other pretrained language models - BanglaBERT generator, Bangla BERT Base and BanglaElectra by almost 3%, 4% and 10% respectively in terms of accuracy. The experiments were conducted on a manually labeled food review dataset consisting of total 6014 real and fake reviews collected from various social media groups. Researchers that are experiencing difficulty recognizing not just fake reviews but other classification issues owing to a lack of labeled data may find a solution in our proposed methodology.
{"title":"Bengali Fake Review Detection using Semi-supervised Generative Adversarial Networks","authors":"Md. Tanvir Rouf Shawon, G. M. Shahariar, F. Shah, Mohammad Shafiul Alam, M. S. Mahbub","doi":"10.1109/ICNLP58431.2023.00011","DOIUrl":"https://doi.org/10.1109/ICNLP58431.2023.00011","url":null,"abstract":"This paper investigates the potential of semi-supervised Generative Adversarial Networks (GANs) to fine-tune pretrained language models in order to classify Bengali fake reviews from real reviews with a few annotated data. With the rise of social media and e-commerce, the ability to detect fake or deceptive reviews is becoming increasingly important in order to protect consumers from being misled by false information. Any machine learning model will have trouble identifying a fake review, especially for a low resource language like Bengali. We have demonstrated that the proposed semi-supervised GAN-LM architecture (generative adversarial network on top of a pretrained language model) is a viable solution in classifying Bengali fake reviews as the experimental results suggest that even with only 1024 annotated samples, BanglaBERT with semi-supervised GAN (SSGAN) achieved an accuracy of 83.59% and a f1-score of 84.89% outperforming other pretrained language models - BanglaBERT generator, Bangla BERT Base and BanglaElectra by almost 3%, 4% and 10% respectively in terms of accuracy. The experiments were conducted on a manually labeled food review dataset consisting of total 6014 real and fake reviews collected from various social media groups. Researchers that are experiencing difficulty recognizing not just fake reviews but other classification issues owing to a lack of labeled data may find a solution in our proposed methodology.","PeriodicalId":53637,"journal":{"name":"Icon","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"73889263","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-03-01DOI: 10.1109/icnlp58431.2023.00049
Tingzhen Liu, Qianqian Xiong, Shengxi Zhang
Large language model (LLM) is a representation of a major advancement in AI, and has been used in multiple natural language processing tasks. Nevertheless, in different business scenarios, LLM requires fine-tuning by engineers to achieve satisfactory performance, and the cost of achieving target performance and fine-turning may not match. Based on the Baidu STI dataset, we study the upper bound of the performance that classical information retrieval methods can achieve under a specific business, and compare it with the cost and performance of the participating team based on LLM. This paper gives an insight into the potential of classical computational linguistics algorithms, and which can help decision-makers make reasonable choices for LLM and low-cost methods in business R& D.
大型语言模型(Large language model, LLM)是人工智能领域取得重大进展的代表，已被用于多种自然语言处理任务。然而，在不同的业务场景中，LLM需要工程师进行微调才能达到满意的性能，而实现目标性能和微调的成本可能并不匹配。基于百度STI数据集，研究了特定业务下经典信息检索方法所能达到的性能上界，并与基于LLM的参与团队的成本和性能进行了比较。本文揭示了经典计算语言学算法的潜力，可以帮助决策者在商业研发中合理选择LLM和低成本方法。
{"title":"When to Use Large Language Model: Upper Bound Analysis of BM25 Algorithms in Reading Comprehension Task","authors":"Tingzhen Liu, Qianqian Xiong, Shengxi Zhang","doi":"10.1109/icnlp58431.2023.00049","DOIUrl":"https://doi.org/10.1109/icnlp58431.2023.00049","url":null,"abstract":"Large language model (LLM) is a representation of a major advancement in AI, and has been used in multiple natural language processing tasks. Nevertheless, in different business scenarios, LLM requires fine-tuning by engineers to achieve satisfactory performance, and the cost of achieving target performance and fine-turning may not match. Based on the Baidu STI dataset, we study the upper bound of the performance that classical information retrieval methods can achieve under a specific business, and compare it with the cost and performance of the participating team based on LLM. This paper gives an insight into the potential of classical computational linguistics algorithms, and which can help decision-makers make reasonable choices for LLM and low-cost methods in business R& D.","PeriodicalId":53637,"journal":{"name":"Icon","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"74508894","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-03-01DOI: 10.1109/ICNLP58431.2023.00058
Tu Ying
Metaphor has been observed to be a widespread method of communication in international news coverage. However, there haven’t been many studies on metaphors in English news reporting, let alone ones that used computer-assisted tools. The study uses computer-assisted tools to analyze and interpret the types of metaphors and their structural models present in 12 news reports about the Belt and Road Initiative (BRI) published in The New York Times. The findings of the research aim to present the Chinese national image depicted in the mainstream western media and reveal the underlying values and attitudes contained in the news reports on BRI.
{"title":"Computer-aided Analysis of Conceptual Metaphors in English News Report","authors":"Tu Ying","doi":"10.1109/ICNLP58431.2023.00058","DOIUrl":"https://doi.org/10.1109/ICNLP58431.2023.00058","url":null,"abstract":"Metaphor has been observed to be a widespread method of communication in international news coverage. However, there haven’t been many studies on metaphors in English news reporting, let alone ones that used computer-assisted tools. The study uses computer-assisted tools to analyze and interpret the types of metaphors and their structural models present in 12 news reports about the Belt and Road Initiative (BRI) published in The New York Times. The findings of the research aim to present the Chinese national image depicted in the mainstream western media and reveal the underlying values and attitudes contained in the news reports on BRI.","PeriodicalId":53637,"journal":{"name":"Icon","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"85777741","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-02-16DOI: 10.1109/ICNLP58431.2023.00066
Xiaobing Zhang, Zhenhao Tang, Zi Long, Xianghua Fu
Recently, a large number of studies have shown that the introduction of visual information can effectively improve the effect of neural machine translation (NMT). Its effectiveness largely depends on the availability of a large number of bilingual parallel sentence pairs and manual image annotation. The lack of images and the effectiveness of images have been difficult to solve. In this paper, a multimodal pre-training generalization algorithm for self-supervised training is proposed, which overcomes the lack of visual information and inaccuracy, and thus extends the applicability of images on NMT. Specifically, we will search for many pictures from the existing sentences through the search engine, and then through the relationship between visual information and text, do the self-supervised training task of graphics and text to obtain more effective visual information for text. We show that when the filtered information is used as multimodal machine translation for fine-tuning, the effect of translation in the global voice dataset is 0.5 BLEU higher than the baseline.
{"title":"Generalization Algorithm of Multimodal Pre-Training Model Based on Graph-Text Self-Supervised Training","authors":"Xiaobing Zhang, Zhenhao Tang, Zi Long, Xianghua Fu","doi":"10.1109/ICNLP58431.2023.00066","DOIUrl":"https://doi.org/10.1109/ICNLP58431.2023.00066","url":null,"abstract":"Recently, a large number of studies have shown that the introduction of visual information can effectively improve the effect of neural machine translation (NMT). Its effectiveness largely depends on the availability of a large number of bilingual parallel sentence pairs and manual image annotation. The lack of images and the effectiveness of images have been difficult to solve. In this paper, a multimodal pre-training generalization algorithm for self-supervised training is proposed, which overcomes the lack of visual information and inaccuracy, and thus extends the applicability of images on NMT. Specifically, we will search for many pictures from the existing sentences through the search engine, and then through the relationship between visual information and text, do the self-supervised training task of graphics and text to obtain more effective visual information for text. We show that when the filtered information is used as multimodal machine translation for fine-tuning, the effect of translation in the global voice dataset is 0.5 BLEU higher than the baseline.","PeriodicalId":53637,"journal":{"name":"Icon","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-02-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"86395941","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Massive knowledge graphs like Wikidata attempt to capture world knowledge about multiple entities. Recent approaches concentrate on automatically enriching these KGs from text. However a lot of information present in the form of natural text in low resource languages is often missed out. Cross Lingual Information Extraction aims at extracting factual information in the form of English triples from low resource Indian Language text. Despite its massive potential, progress made on this task is lagging when compared to Monolingual Information Extraction. In this paper, we propose the task of Cross Lingual Fact Extraction(CLFE) from text and devise an end-to-end generative approach for the same which achieves an overall F1 score of 77.46
{"title":"Massively Multilingual Language Models for Cross Lingual Fact Extraction from Low Resource Indian Languages","authors":"Bhavyajeet Singh, Pavan Kandru, Anubhav Sharma, Vasudeva Varma","doi":"10.48550/arXiv.2302.04790","DOIUrl":"https://doi.org/10.48550/arXiv.2302.04790","url":null,"abstract":"Massive knowledge graphs like Wikidata attempt to capture world knowledge about multiple entities. Recent approaches concentrate on automatically enriching these KGs from text. However a lot of information present in the form of natural text in low resource languages is often missed out. Cross Lingual Information Extraction aims at extracting factual information in the form of English triples from low resource Indian Language text. Despite its massive potential, progress made on this task is lagging when compared to Monolingual Information Extraction. In this paper, we propose the task of Cross Lingual Fact Extraction(CLFE) from text and devise an end-to-end generative approach for the same which achieves an overall F1 score of 77.46","PeriodicalId":53637,"journal":{"name":"Icon","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-02-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"48963928","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-01-10DOI: 10.48550/arXiv.2301.04013
Ankush Agarwal, Sakharam Gawade, Sachin Channabasavarajendra, P. Bhattacharyya
The integration of knowledge graphs with deep learning is thriving in improving the performance of various natural language processing (NLP) tasks. In this paper, we focus on knowledge-infused link prediction and question answering using language models, T5, and BLOOM across three domains:Aviation, Movie, and Web. In this context, we infuse knowledge in large and small language models and study their performance, and find the performance to be similar. For the link prediction task on the Aviation Knowledge Graph, we obtain a 0.2 hits@1 score using T5-small, T5-base, T5-large, and BLOOM. Using template-based scripts, we create a set of 1 million synthetic factoid QA pairs in the aviation domain from National Transportation Safety Board (NTSB) reports. On our curated QA pairs, the three models of T5 achieve a 0.7 hits@1 score. We validate our findings with the paired student t test and Cohen’s kappa scores. For link prediction on Aviation Knowledge Graph using T5-small and T5-large, we obtain a Cohen’s kappa score of 0.76, showing substantial agreement between the models. Thus, we infer that small language models perform similar to large language models with the infusion of knowledge.
{"title":"There is No Big Brother or Small Brother:Knowledge Infusion in Language Models for Link Prediction and Question Answering","authors":"Ankush Agarwal, Sakharam Gawade, Sachin Channabasavarajendra, P. Bhattacharyya","doi":"10.48550/arXiv.2301.04013","DOIUrl":"https://doi.org/10.48550/arXiv.2301.04013","url":null,"abstract":"The integration of knowledge graphs with deep learning is thriving in improving the performance of various natural language processing (NLP) tasks. In this paper, we focus on knowledge-infused link prediction and question answering using language models, T5, and BLOOM across three domains:Aviation, Movie, and Web. In this context, we infuse knowledge in large and small language models and study their performance, and find the performance to be similar. For the link prediction task on the Aviation Knowledge Graph, we obtain a 0.2 hits@1 score using T5-small, T5-base, T5-large, and BLOOM. Using template-based scripts, we create a set of 1 million synthetic factoid QA pairs in the aviation domain from National Transportation Safety Board (NTSB) reports. On our curated QA pairs, the three models of T5 achieve a 0.7 hits@1 score. We validate our findings with the paired student t test and Cohen’s kappa scores. For link prediction on Aviation Knowledge Graph using T5-small and T5-large, we obtain a Cohen’s kappa score of 0.76, showing substantial agreement between the models. Thus, we infer that small language models perform similar to large language models with the infusion of knowledge.","PeriodicalId":53637,"journal":{"name":"Icon","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-01-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46274359","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Abbreviations and contractions are commonly found in text across different domains. For example, doctors’ notes contain many contractions that can be personalized based on their choices. Existing spelling correction models are not suitable to handle expansions because of many reductions of characters in words. In this work, we propose ABB-BERT, a BERT-based model, which deals with an ambiguous language containing abbreviations and contractions. ABB-BERT can rank them from thousands of options and is designed for scale. It is trained on Wikipedia text, and the algorithm allows it to be fine-tuned with little compute to get better performance for a domain or person. We are publicly releasing the training dataset for abbreviations and contractions derived from Wikipedia.
{"title":"ABB-BERT: A BERT model for disambiguating abbreviations and contractions","authors":"Prateek Kacker, Andi Cupallari, Aswin Giridhar Subramanian, Nimit Jain","doi":"10.48550/arXiv.2207.04008","DOIUrl":"https://doi.org/10.48550/arXiv.2207.04008","url":null,"abstract":"Abbreviations and contractions are commonly found in text across different domains. For example, doctors’ notes contain many contractions that can be personalized based on their choices. Existing spelling correction models are not suitable to handle expansions because of many reductions of characters in words. In this work, we propose ABB-BERT, a BERT-based model, which deals with an ambiguous language containing abbreviations and contractions. ABB-BERT can rank them from thousands of options and is designed for scale. It is trained on Wikipedia text, and the algorithm allows it to be fine-tuned with little compute to get better performance for a domain or person. We are publicly releasing the training dataset for abbreviations and contractions derived from Wikipedia.","PeriodicalId":53637,"journal":{"name":"Icon","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2022-07-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"43169427","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2022-06-28DOI: 10.1609/aaai.v36i11.21705
Wes Robbins
Vision-language models can assess visual context in an image and generate descriptive text. While the generated text may be accurate and syntactically correct, it is often overly general. To address this, recent work has used optical character recognition to supplement visual information with text extracted from an image. In this work, we contend that vision-language models can benefit from information that can be extracted from an image, but are not used by current models. We modify previous multimodal frameworks to accept relevant information from any number of auxiliary classifiers. In particular, we focus on person names as an additional set of tokens and create a novel image-caption dataset to facilitate captioning with person names. The dataset, Politicians and Athletes in Captions (PAC), consists of captioned images of well-known people in context. By fine-tuning pretrained models with this dataset, we demonstrate a model that can naturally integrate facial recognition tokens into generated text by training on limited data. For the PAC dataset, we provide a discussion on collection and baseline benchmark scores.
{"title":"Towards Multimodal Vision-Language Models Generating Non-Generic Text","authors":"Wes Robbins","doi":"10.1609/aaai.v36i11.21705","DOIUrl":"https://doi.org/10.1609/aaai.v36i11.21705","url":null,"abstract":"Vision-language models can assess visual context in an image and generate descriptive text. While the generated text may be accurate and syntactically correct, it is often overly general. To address this, recent work has used optical character recognition to supplement visual information with text extracted from an image. In this work, we contend that vision-language models can benefit from information that can be extracted from an image, but are not used by current models. We modify previous multimodal frameworks to accept relevant information from any number of auxiliary classifiers. In particular, we focus on person names as an additional set of tokens and create a novel image-caption dataset to facilitate captioning with person names. The dataset, Politicians and Athletes in Captions (PAC), consists of captioned images of well-known people in context. By fine-tuning pretrained models with this dataset, we demonstrate a model that can naturally integrate facial recognition tokens into generated text by training on limited data. For the PAC dataset, we provide a discussion on collection and baseline benchmark scores.","PeriodicalId":53637,"journal":{"name":"Icon","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2022-06-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42691546","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：