{"title":"MF-Saudi:弥合音频和文本数据鸿沟的多模态框架,用于沙特方言检测","authors":"Raed Alharbi","doi":"10.1016/j.jksuci.2024.102084","DOIUrl":null,"url":null,"abstract":"<div><p>Detecting variations in dialects within a language can be challenging, particularly in regions with rich linguistic diversity like Saudi Arabia. To our knowledge, no prior attempts have been made to develop a multimodal, audio–textual framework for Saudi dialect detection. The current approaches often concentrate on detecting dialects only based on audio or textual data, which fails to capture the complex relationship between both modalities. In this paper, we propose a novel Multimodal Framework, called MF-Saudi, for Saudi dialect detection. The framework consists of three main components: (1) a pretrained BERT encoder for extracting and encoding textual information; (2) an acoustic model for representing audio signals and fusing them with textual information via the fusion layer; and (3) an alignment learning module to develop meaningful representations that capture the complexities of audio–text relationships, resulting in improved dialect detection. We conduct empirical evaluations on a real-world dataset, demonstrating that our solution outperforms some of the state-of-the-art baseline methods. The experiment’s code can be found here: <span>https://github.com/raed19/MF-Saudi</span><svg><path></path></svg>.</p></div>","PeriodicalId":48547,"journal":{"name":"Journal of King Saud University-Computer and Information Sciences","volume":null,"pages":null},"PeriodicalIF":5.2000,"publicationDate":"2024-06-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S1319157824001733/pdfft?md5=99b69313cadb5fce44b832f5ddaa2066&pid=1-s2.0-S1319157824001733-main.pdf","citationCount":"0","resultStr":"{\"title\":\"MF-Saudi: A multimodal framework for bridging the gap between audio and textual data for Saudi dialect detection\",\"authors\":\"Raed Alharbi\",\"doi\":\"10.1016/j.jksuci.2024.102084\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div><p>Detecting variations in dialects within a language can be challenging, particularly in regions with rich linguistic diversity like Saudi Arabia. To our knowledge, no prior attempts have been made to develop a multimodal, audio–textual framework for Saudi dialect detection. The current approaches often concentrate on detecting dialects only based on audio or textual data, which fails to capture the complex relationship between both modalities. In this paper, we propose a novel Multimodal Framework, called MF-Saudi, for Saudi dialect detection. The framework consists of three main components: (1) a pretrained BERT encoder for extracting and encoding textual information; (2) an acoustic model for representing audio signals and fusing them with textual information via the fusion layer; and (3) an alignment learning module to develop meaningful representations that capture the complexities of audio–text relationships, resulting in improved dialect detection. We conduct empirical evaluations on a real-world dataset, demonstrating that our solution outperforms some of the state-of-the-art baseline methods. The experiment’s code can be found here: <span>https://github.com/raed19/MF-Saudi</span><svg><path></path></svg>.</p></div>\",\"PeriodicalId\":48547,\"journal\":{\"name\":\"Journal of King Saud University-Computer and Information Sciences\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":5.2000,\"publicationDate\":\"2024-06-14\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"https://www.sciencedirect.com/science/article/pii/S1319157824001733/pdfft?md5=99b69313cadb5fce44b832f5ddaa2066&pid=1-s2.0-S1319157824001733-main.pdf\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Journal of King Saud University-Computer and Information Sciences\",\"FirstCategoryId\":\"94\",\"ListUrlMain\":\"https://www.sciencedirect.com/science/article/pii/S1319157824001733\",\"RegionNum\":2,\"RegionCategory\":\"计算机科学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"COMPUTER SCIENCE, INFORMATION SYSTEMS\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of King Saud University-Computer and Information Sciences","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1319157824001733","RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, INFORMATION SYSTEMS","Score":null,"Total":0}
MF-Saudi: A multimodal framework for bridging the gap between audio and textual data for Saudi dialect detection
Detecting variations in dialects within a language can be challenging, particularly in regions with rich linguistic diversity like Saudi Arabia. To our knowledge, no prior attempts have been made to develop a multimodal, audio–textual framework for Saudi dialect detection. The current approaches often concentrate on detecting dialects only based on audio or textual data, which fails to capture the complex relationship between both modalities. In this paper, we propose a novel Multimodal Framework, called MF-Saudi, for Saudi dialect detection. The framework consists of three main components: (1) a pretrained BERT encoder for extracting and encoding textual information; (2) an acoustic model for representing audio signals and fusing them with textual information via the fusion layer; and (3) an alignment learning module to develop meaningful representations that capture the complexities of audio–text relationships, resulting in improved dialect detection. We conduct empirical evaluations on a real-world dataset, demonstrating that our solution outperforms some of the state-of-the-art baseline methods. The experiment’s code can be found here: https://github.com/raed19/MF-Saudi.
期刊介绍:
In 2022 the Journal of King Saud University - Computer and Information Sciences will become an author paid open access journal. Authors who submit their manuscript after October 31st 2021 will be asked to pay an Article Processing Charge (APC) after acceptance of their paper to make their work immediately, permanently, and freely accessible to all. The Journal of King Saud University Computer and Information Sciences is a refereed, international journal that covers all aspects of both foundations of computer and its practical applications.