印地语和马拉地语文学字数统计语料库的开发

Vivek Belhekar, Radhika Bhargava
{"title":"印地语和马拉地语文学字数统计语料库的开发","authors":"Vivek Belhekar,&nbsp;Radhika Bhargava","doi":"10.1016/j.acorp.2023.100070","DOIUrl":null,"url":null,"abstract":"<div><p><span>India has a huge diversity of languages, and Hindi and Marathi are the most spoken languages in the northern and western parts of India. Hindi and Marathi have more than 528 million and 83 million speakers, respectively. The paper describes the development of the Hindi Word Corpus (Hindi WordCorp) and the Marathi Word Corpus (Marathi WordCorp), reporting the frequency of single words (1-gram) used in written texts of the respective languages using the bag-of-words model (BoW). The word frequencies are provided for eleven decades (pre-1920, 1920 to 2020). These texts include books (fiction, non-fiction, history, autobiographies, etc.) and magazines. Academic and reference books were not used. The Hindi WordCorp and Marathi WordCorp used 640 and 712 texts, respectively. An analysis was employed to check whether the texts used were enough to stabilize the rank-order of the total frequencies of the words. Zipf's and Heaps’ law coefficients indicated the sufficiency of the texts. Researchers in various areas like linguistics, social sciences, text mining, machine learning, etc., can use the dataset to answer research questions about language and culture. Some demonstrative examples are provided for using the datasets in the two languages. The dataset is made available on an </span>open data<span> repository. The paper is an account of dataset creation for Hindi and Marathi WordCorp. Hence, no empirical results or conclusions are made based on the data created. A WebApp named Indian Languages Word Corpus (ILWC) has been developed for users. Future directions for text mining and language models are discussed.</span></p></div>","PeriodicalId":72254,"journal":{"name":"Applied Corpus Linguistics","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2023-08-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Development of word count data corpus for Hindi and Marathi literature\",\"authors\":\"Vivek Belhekar,&nbsp;Radhika Bhargava\",\"doi\":\"10.1016/j.acorp.2023.100070\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div><p><span>India has a huge diversity of languages, and Hindi and Marathi are the most spoken languages in the northern and western parts of India. Hindi and Marathi have more than 528 million and 83 million speakers, respectively. The paper describes the development of the Hindi Word Corpus (Hindi WordCorp) and the Marathi Word Corpus (Marathi WordCorp), reporting the frequency of single words (1-gram) used in written texts of the respective languages using the bag-of-words model (BoW). The word frequencies are provided for eleven decades (pre-1920, 1920 to 2020). These texts include books (fiction, non-fiction, history, autobiographies, etc.) and magazines. Academic and reference books were not used. The Hindi WordCorp and Marathi WordCorp used 640 and 712 texts, respectively. An analysis was employed to check whether the texts used were enough to stabilize the rank-order of the total frequencies of the words. Zipf's and Heaps’ law coefficients indicated the sufficiency of the texts. Researchers in various areas like linguistics, social sciences, text mining, machine learning, etc., can use the dataset to answer research questions about language and culture. Some demonstrative examples are provided for using the datasets in the two languages. The dataset is made available on an </span>open data<span> repository. The paper is an account of dataset creation for Hindi and Marathi WordCorp. Hence, no empirical results or conclusions are made based on the data created. A WebApp named Indian Languages Word Corpus (ILWC) has been developed for users. Future directions for text mining and language models are discussed.</span></p></div>\",\"PeriodicalId\":72254,\"journal\":{\"name\":\"Applied Corpus Linguistics\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2023-08-23\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Applied Corpus Linguistics\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://www.sciencedirect.com/science/article/pii/S2666799123000308\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Applied Corpus Linguistics","FirstCategoryId":"1085","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S2666799123000308","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

摘要

印度有多种多样的语言,印地语和马拉地语是印度北部和西部最常用的语言。印地语和马拉地语的使用者分别超过5.28亿和8300万。本文描述了印地语单词语料库(Hindi WordCorp)和马拉地语单词语料库(Marathi WordCorp)的发展,使用单词袋模型(BoW)报告了各自语言书面文本中单个单词(1克)的使用频率。提供了11年的词频(1920年以前、1920年至2020年)。这些文本包括书籍(小说、非小说、历史、自传等)和杂志。没有使用学术书籍和参考书。印地语词汇公司和马拉地语词汇公司分别使用了640和712个文本。通过分析来检查所使用的文本是否足以稳定单词总频率的等级顺序。齐夫定律系数和希普斯定律系数表明了文本的充分性。语言学、社会科学、文本挖掘、机器学习等各个领域的研究人员都可以使用该数据集来回答有关语言和文化的研究问题。提供了一些使用两种语言的数据集的示范示例。数据集在开放数据存储库上可用。这篇论文是关于印地语和马拉地语WordCorp数据集创建的一篇文章。因此,没有实证结果或结论是基于所创建的数据。为用户开发了一个名为印度语言词库(ILWC)的web应用程序。讨论了文本挖掘和语言模型的未来发展方向。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
查看原文
分享 分享
微信好友 朋友圈 QQ好友 复制链接
本刊更多论文
Development of word count data corpus for Hindi and Marathi literature

India has a huge diversity of languages, and Hindi and Marathi are the most spoken languages in the northern and western parts of India. Hindi and Marathi have more than 528 million and 83 million speakers, respectively. The paper describes the development of the Hindi Word Corpus (Hindi WordCorp) and the Marathi Word Corpus (Marathi WordCorp), reporting the frequency of single words (1-gram) used in written texts of the respective languages using the bag-of-words model (BoW). The word frequencies are provided for eleven decades (pre-1920, 1920 to 2020). These texts include books (fiction, non-fiction, history, autobiographies, etc.) and magazines. Academic and reference books were not used. The Hindi WordCorp and Marathi WordCorp used 640 and 712 texts, respectively. An analysis was employed to check whether the texts used were enough to stabilize the rank-order of the total frequencies of the words. Zipf's and Heaps’ law coefficients indicated the sufficiency of the texts. Researchers in various areas like linguistics, social sciences, text mining, machine learning, etc., can use the dataset to answer research questions about language and culture. Some demonstrative examples are provided for using the datasets in the two languages. The dataset is made available on an open data repository. The paper is an account of dataset creation for Hindi and Marathi WordCorp. Hence, no empirical results or conclusions are made based on the data created. A WebApp named Indian Languages Word Corpus (ILWC) has been developed for users. Future directions for text mining and language models are discussed.

求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
Applied Corpus Linguistics
Applied Corpus Linguistics Linguistics and Language
CiteScore
1.30
自引率
0.00%
发文量
0
审稿时长
70 days
期刊最新文献
Breach of pacta sunt servanda: A corpus-assisted analysis of newspaper discourse on the AUKUS agreement Identifying ChatGPT-generated texts in EFL students’ writing: Through comparative analysis of linguistic fingerprints English podcasts for schoolchildren and their vocabulary demands Capturing chronological variation in L2 speech through lexical measurements and regression analysis Investigating spoken classroom interactions in linguistically heterogeneous learning groups – An interdisciplinary approach to process video-based data in second language acquisition classrooms
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
已复制链接
已复制链接
快去分享给好友吧!
我知道了
×
扫码分享
扫码分享
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1