Pub Date : 2024-01-23DOI: 10.18653/v1/2023.arabicnlp-1.81
Mohammed Elkomy, Amany Sarhan
In this paper, we present our approach to tackle Qur’an QA 2023 shared tasks A and B. To address the challenge of low-resourced training data, we rely on transfer learning together with a voting ensemble to improve prediction stability across multiple runs. Additionally, we employ different architectures and learning mechanisms for a range of Arabic pre-trained transformer-based models for both tasks. To identify unanswerable questions, we propose using a thresholding mechanism. Our top-performing systems greatly surpass the baseline performance on the hidden split, achieving a MAP score of 25.05% for task A and a partial Average Precision (pAP) of 57.11% for task B.
在本文中，我们介绍了处理《古兰经》QA 2023 共同任务 A 和 B 的方法。为了应对训练数据资源不足的挑战，我们依靠迁移学习和投票组合来提高多次运行中的预测稳定性。此外，我们还针对这两个任务的一系列基于转换器的阿拉伯语预训练模型采用了不同的架构和学习机制。为了识别无法回答的问题，我们建议使用阈值机制。我们性能最佳的系统在隐藏分词上大大超过了基线性能，在任务 A 中的 MAP 得分为 25.05%，在任务 B 中的部分平均精度 (pAP) 为 57.11%。
{"title":"TCE at Qur’an QA 2023 Shared Task: Low Resource Enhanced Transformer-based Ensemble Approach for Qur’anic QA","authors":"Mohammed Elkomy, Amany Sarhan","doi":"10.18653/v1/2023.arabicnlp-1.81","DOIUrl":"https://doi.org/10.18653/v1/2023.arabicnlp-1.81","url":null,"abstract":"In this paper, we present our approach to tackle Qur’an QA 2023 shared tasks A and B. To address the challenge of low-resourced training data, we rely on transfer learning together with a voting ensemble to improve prediction stability across multiple runs. Additionally, we employ different architectures and learning mechanisms for a range of Arabic pre-trained transformer-based models for both tasks. To identify unanswerable questions, we propose using a thresholding mechanism. Our top-performing systems greatly surpass the baseline performance on the hidden split, achieving a MAP score of 25.05% for task A and a partial Average Precision (pAP) of 57.11% for task B.","PeriodicalId":503921,"journal":{"name":"ARABICNLP","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2024-01-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139603885","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-12-16DOI: 10.18653/v1/2023.arabicnlp-1.69
Mohamed Lichouri, Khaled Lounnas, Aicha Zitouni, H. Latrache, R. Djeradi
In this paper, we conduct an in-depth analysis of several key factors influencing the performance of Arabic Dialect Identification NADI’2023, with a specific focus on the first subtask involving country-level dialect identification. Our investigation encompasses the effects of surface preprocessing, morphological preprocessing, FastText vector model, and the weighted concatenation of TF-IDF features. For classification purposes, we employ the Linear Support Vector Classification (LSVC) model. During the evaluation phase, our system demonstrates noteworthy results, achieving an F_1 score of 62.51%. This achievement closely aligns with the average F_1 scores attained by other systems submitted for the first subtask, which stands at 72.91%.
{"title":"USTHB at NADI 2023 shared task: Exploring Preprocessing and Feature Engineering Strategies for Arabic Dialect Identification","authors":"Mohamed Lichouri, Khaled Lounnas, Aicha Zitouni, H. Latrache, R. Djeradi","doi":"10.18653/v1/2023.arabicnlp-1.69","DOIUrl":"https://doi.org/10.18653/v1/2023.arabicnlp-1.69","url":null,"abstract":"In this paper, we conduct an in-depth analysis of several key factors influencing the performance of Arabic Dialect Identification NADI’2023, with a specific focus on the first subtask involving country-level dialect identification. Our investigation encompasses the effects of surface preprocessing, morphological preprocessing, FastText vector model, and the weighted concatenation of TF-IDF features. For classification purposes, we employ the Linear Support Vector Classification (LSVC) model. During the evaluation phase, our system demonstrates noteworthy results, achieving an F_1 score of 62.51%. This achievement closely aligns with the average F_1 scores attained by other systems submitted for the first subtask, which stands at 72.91%.","PeriodicalId":503921,"journal":{"name":"ARABICNLP","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-12-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139176897","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-12-13DOI: 10.18653/v1/2023.arabicnlp-1.9
S. Kwon, Gagan Bhatia, El Moatez Billah Nagoudi, M. Abdul-Mageed
Large language models (LLMs) finetuned to follow human instruction have recently exhibited significant capabilities in various English NLP tasks. However, their performance in grammatical error correction (GEC), especially on languages other than English, remains significantly unexplored. In this work, we evaluate the abilities of instruction finetuned LLMs in Arabic GEC, a complex task due to Arabic’s rich morphology. Our findings suggest that various prompting methods, coupled with (in-context) few-shot learning, demonstrate considerable effectiveness, with GPT-4 achieving up to 65.49 F1 score under expert prompting (approximately 5 points higher than our established baseline). Despite these positive results, we find that instruction finetuned models, regardless of their size, are still outperformed by fully finetuned ones, even if they are significantly smaller in size. This disparity highlights substantial room for improvements for LLMs. Inspired by methods used in low-resource machine translation, we also develop a method exploiting synthetic data that significantly outperforms previous models on two standard Arabic benchmarks. Our best model achieves a new SOTA on Arabic GEC, with 73.29 and 73.26 F1 on the 2014 and 2015 QALB datasets, respectively, compared to peer-reviewed published baselines.
最近，根据人类指令微调的大型语言模型（LLMs）在各种英语 NLP 任务中表现出了显著的能力。然而，它们在语法纠错（GEC）中的表现，尤其是在英语以外的语言中的表现，仍有待进一步探索。在这项工作中，我们评估了经过指令微调的 LLMs 在阿拉伯语语法纠错中的能力，由于阿拉伯语丰富的语态，这是一项复杂的任务。我们的研究结果表明，各种提示方法与（语境中的）少量学习相结合，显示出相当大的有效性，在专家提示下，GPT-4 的 F1 得分高达 65.49（比我们设定的基线高出约 5 分）。尽管取得了这些积极成果，但我们发现，经过指令微调的模型，无论其大小如何，其性能仍然优于经过完全微调的模型，即使它们的大小要小得多。这种差异凸显了 LLMs 的巨大改进空间。受低资源机器翻译方法的启发，我们还开发了一种利用合成数据的方法，该方法在两个标准阿拉伯语基准上的表现明显优于以前的模型。我们的最佳模型在阿拉伯语 GEC 上实现了新的 SOTA，在 2014 年和 2015 年 QALB 数据集上的 F1 分别为 73.29 和 73.26，超过了同行评审公布的基线。
{"title":"Beyond English: Evaluating LLMs for Arabic Grammatical Error Correction","authors":"S. Kwon, Gagan Bhatia, El Moatez Billah Nagoudi, M. Abdul-Mageed","doi":"10.18653/v1/2023.arabicnlp-1.9","DOIUrl":"https://doi.org/10.18653/v1/2023.arabicnlp-1.9","url":null,"abstract":"Large language models (LLMs) finetuned to follow human instruction have recently exhibited significant capabilities in various English NLP tasks. However, their performance in grammatical error correction (GEC), especially on languages other than English, remains significantly unexplored. In this work, we evaluate the abilities of instruction finetuned LLMs in Arabic GEC, a complex task due to Arabic’s rich morphology. Our findings suggest that various prompting methods, coupled with (in-context) few-shot learning, demonstrate considerable effectiveness, with GPT-4 achieving up to 65.49 F1 score under expert prompting (approximately 5 points higher than our established baseline). Despite these positive results, we find that instruction finetuned models, regardless of their size, are still outperformed by fully finetuned ones, even if they are significantly smaller in size. This disparity highlights substantial room for improvements for LLMs. Inspired by methods used in low-resource machine translation, we also develop a method exploiting synthetic data that significantly outperforms previous models on two standard Arabic benchmarks. Our best model achieves a new SOTA on Arabic GEC, with 73.29 and 73.26 F1 on the 2014 and 2015 QALB datasets, respectively, compared to peer-reviewed published baselines.","PeriodicalId":503921,"journal":{"name":"ARABICNLP","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-12-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139181302","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
In this paper, we highlight our approach for the “Arabic AI Tasks Evaluation (ArAiEval) Shared Task 2023”. We present our approaches for task 1-A and task 2-A of the shared task which focus on persuasion technique detection and disinformation detection respectively. Detection of persuasion techniques and disinformation has become imperative to avoid distortion of authentic information. The tasks use multigenre snippets of tweets and news articles for the given binary classification problem. We experiment with several transformer-based models that were pre-trained on the Arabic language. We fine-tune these state-of-the-art models on the provided dataset. Ensembling is employed to enhance the performance of the systems. We achieved a micro F1-score of 0.742 on task 1-A (8th rank on the leaderboard) and 0.901 on task 2-A (7th rank on the leaderboard) respectively.
本文重点介绍了我们针对 "阿拉伯语人工智能任务评估（ArAiEval）2023 共享任务 "所采用的方法。我们介绍了针对任务 1-A 和任务 2-A 的方法，这两个任务分别侧重于说服技术检测和虚假信息检测。为避免真实信息失真，检测劝诱技术和虚假信息已成为当务之急。这些任务使用推文和新闻文章的多源片段来解决给定的二元分类问题。我们试验了几种基于转换器的模型，这些模型已在阿拉伯语中进行了预先训练。我们在提供的数据集上对这些最先进的模型进行了微调。为了提高系统的性能，我们采用了集合的方法。我们在任务 1-A 和任务 2-A 上分别取得了 0.742（排行榜第 8 位）和 0.901（排行榜第 7 位）的微型 F1 分数。
{"title":"Mavericks at ArAIEval Shared Task: Towards a Safer Digital Space - Transformer Ensemble Models Tackling Deception and Persuasion","authors":"Sudeep Mangalvedhekar, Kshitij Deshpande, Yash Patwardhan, Vedant Deshpande, Ravindra Murumkar","doi":"10.48550/arXiv.2311.18730","DOIUrl":"https://doi.org/10.48550/arXiv.2311.18730","url":null,"abstract":"In this paper, we highlight our approach for the “Arabic AI Tasks Evaluation (ArAiEval) Shared Task 2023”. We present our approaches for task 1-A and task 2-A of the shared task which focus on persuasion technique detection and disinformation detection respectively. Detection of persuasion techniques and disinformation has become imperative to avoid distortion of authentic information. The tasks use multigenre snippets of tweets and news articles for the given binary classification problem. We experiment with several transformer-based models that were pre-trained on the Arabic language. We fine-tune these state-of-the-art models on the provided dataset. Ensembling is employed to enhance the performance of the systems. We achieved a micro F1-score of 0.742 on task 1-A (8th rank on the leaderboard) and 0.901 on task 2-A (7th rank on the leaderboard) respectively.","PeriodicalId":503921,"journal":{"name":"ARABICNLP","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-11-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139200020","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-11-15DOI: 10.48550/arXiv.2311.08844
Abdelrahman Mohamed, Fakhraddin Alwajih, El Moatez Billah Nagoudi, Alcides Alcoba Inciarte, M. Abdul-Mageed
Although image captioning has a vast array of applications, it has not reached its full potential in languages other than English. Arabic, for instance, although the native language of more than 400 million people, remains largely underrepresented in this area. This is due to the lack of labeled data and powerful Arabic generative models. We alleviate this issue by presenting a novel vision-language model dedicated to Arabic, dubbed Violet. Our model is based on a vision encoder and a Gemini text decoder that maintains generation fluency while allowing fusion between the vision and language components. To train our model, we introduce a new method for automatically acquiring data from available English datasets. We also manually prepare a new dataset for evaluation. Violet performs sizeably better than our baselines on all of our evaluation datasets. For example, it reaches a CIDEr score of 61.2 on our manually annotated dataset and achieves an improvement of 13 points on Flickr8k.
{"title":"Violet: A Vision-Language Model for Arabic Image Captioning with Gemini Decoder","authors":"Abdelrahman Mohamed, Fakhraddin Alwajih, El Moatez Billah Nagoudi, Alcides Alcoba Inciarte, M. Abdul-Mageed","doi":"10.48550/arXiv.2311.08844","DOIUrl":"https://doi.org/10.48550/arXiv.2311.08844","url":null,"abstract":"Although image captioning has a vast array of applications, it has not reached its full potential in languages other than English. Arabic, for instance, although the native language of more than 400 million people, remains largely underrepresented in this area. This is due to the lack of labeled data and powerful Arabic generative models. We alleviate this issue by presenting a novel vision-language model dedicated to Arabic, dubbed Violet. Our model is based on a vision encoder and a Gemini text decoder that maintains generation fluency while allowing fusion between the vision and language components. To train our model, we introduce a new method for automatically acquiring data from available English datasets. We also manually prepare a new dataset for evaluation. Violet performs sizeably better than our baselines on all of our evaluation datasets. For example, it reaches a CIDEr score of 61.2 on our manually annotated dataset and achieves an improvement of 13 points on Flickr8k.","PeriodicalId":503921,"journal":{"name":"ARABICNLP","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-11-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139272911","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：