Google Scholar is one of the top search engines to access research articles across multiple disciplines for scholarly literature. Google scholar advance search option gives the privilege to extract articles based on phrases, publishers name, authors name, time duration etc. In this work, we collected Google Scholar data (2000-2021) for two different research domains in computer science: Data Mining and Software Engineering. The scholar database resources are powerful for network analysis, data mining, and identify links between authors via authorship network. We examined coauthor-ship network for each domain and studied their network structure. Extensive experiments are performed to analyze publications trend and identifying influential authors and affiliated organizations for each domain. The network analysis shows that the networks features are distinct from one another and exhibit small communities within the influential authors of a particular domain.
Google Scholar 是访问跨学科学术文献研究文章的顶级搜索引擎之一。谷歌学者的高级搜索选项提供了根据短语、出版商名称、作者姓名、时间长度等提取文章的特权。在这项工作中，我们收集了计算机科学领域两个不同研究领域的谷歌学术数据（2000-2021 年）：数据挖掘和软件工程。学者数据库资源具有强大的网络分析和数据挖掘功能，可通过作者关系网络识别作者之间的联系。我们检查了每个领域的合著者关系网络，并研究了它们的网络结构。我们进行了广泛的实验，以分析每个领域的论文发表趋势，并识别有影响力的作者和附属机构。网络分析结果表明，这些网络特征彼此不同，并在特定领域有影响力的作者中呈现出小社区的特征。
{"title":"Examining Different Research Communities: Authorship Network","authors":"Shrabani Ghosh","doi":"arxiv-2409.00081","DOIUrl":"https://doi.org/arxiv-2409.00081","url":null,"abstract":"Google Scholar is one of the top search engines to access research articles\u0000across multiple disciplines for scholarly literature. Google scholar advance\u0000search option gives the privilege to extract articles based on phrases,\u0000publishers name, authors name, time duration etc. In this work, we collected\u0000Google Scholar data (2000-2021) for two different research domains in computer\u0000science: Data Mining and Software Engineering. The scholar database resources\u0000are powerful for network analysis, data mining, and identify links between\u0000authors via authorship network. We examined coauthor-ship network for each\u0000domain and studied their network structure. Extensive experiments are performed\u0000to analyze publications trend and identifying influential authors and\u0000affiliated organizations for each domain. The network analysis shows that the\u0000networks features are distinct from one another and exhibit small communities\u0000within the influential authors of a particular domain.","PeriodicalId":501285,"journal":{"name":"arXiv - CS - Digital Libraries","volume":"59 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142219114","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
F. Asdrubali, A. Fronzetti Colladon, L. Segneri, D. M. Gandola
Research on Life Cycle Assessment (LCA) is being conducted in various sectors, from analyzing building materials and components to comprehensive evaluations of entire structures. However, reviews of the existing literature have been unable to provide a comprehensive overview of research in this field, leaving scholars without a definitive guideline for future investigations. This paper aims to fill this gap, mapping more than twenty years of research. Using an innovative methodology that combines social network analysis and text mining, the paper examined 8024 scientific abstracts. The authors identified seven key thematic groups, building and sustainability clusters (BSCs). To assess their significance in the broader discourse on building and sustainability, the semantic brand score (SBS) indicator was applied. Additionally, building and sustainability trends were tracked, focusing on the LCA concept. The major research topics mainly relate to building materials and energy efficiency. In addition to presenting an innovative approach to reviewing extensive literature domains, the article also provides insights into emerging and underdeveloped themes, outlining crucial future research directions.
{"title":"LCA and energy efficiency in buildings: mapping more than twenty years of research","authors":"F. Asdrubali, A. Fronzetti Colladon, L. Segneri, D. M. Gandola","doi":"arxiv-2409.00065","DOIUrl":"https://doi.org/arxiv-2409.00065","url":null,"abstract":"Research on Life Cycle Assessment (LCA) is being conducted in various\u0000sectors, from analyzing building materials and components to comprehensive\u0000evaluations of entire structures. However, reviews of the existing literature\u0000have been unable to provide a comprehensive overview of research in this field,\u0000leaving scholars without a definitive guideline for future investigations. This\u0000paper aims to fill this gap, mapping more than twenty years of research. Using\u0000an innovative methodology that combines social network analysis and text\u0000mining, the paper examined 8024 scientific abstracts. The authors identified\u0000seven key thematic groups, building and sustainability clusters (BSCs). To\u0000assess their significance in the broader discourse on building and\u0000sustainability, the semantic brand score (SBS) indicator was applied.\u0000Additionally, building and sustainability trends were tracked, focusing on the\u0000LCA concept. The major research topics mainly relate to building materials and\u0000energy efficiency. In addition to presenting an innovative approach to\u0000reviewing extensive literature domains, the article also provides insights into\u0000emerging and underdeveloped themes, outlining crucial future research\u0000directions.","PeriodicalId":501285,"journal":{"name":"arXiv - CS - Digital Libraries","volume":"15 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142219110","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Björn Schembera, Frank Wübbeling, Hendrik Kleikamp, Burkhard Schmidt, Aurela Shehu, Marco Reidelbach, Christine Biedinger, Jochen Fiedler, Thomas Koprucki, Dorothea Iglezakis, Dominik Göddeke
Mathematical models and algorithms are an essential part of mathematical research data, as they are epistemically grounding numerical data. In order to represent models and algorithms as well as their relationship semantically to make this research data FAIR, two previously distinct ontologies were merged and extended, becoming a living knowledge graph. The link between the two ontologies is established by introducing computational tasks, as they occur in modeling, corresponding to algorithmic tasks. Moreover, controlled vocabularies are incorporated and a new class, distinguishing base quantities from specific use case quantities, was introduced. Also, both models and algorithms can now be enriched with metadata. Subject-specific metadata is particularly relevant here, such as the symmetry of a matrix or the linearity of a mathematical model. This is the only way to express specific workflows with concrete models and algorithms, as the feasible solution algorithm can only be determined if the mathematical properties of a model are known. We demonstrate this using two examples from different application areas of applied mathematics. In addition, we have already integrated over 250 research assets from applied mathematics into our knowledge graph.
{"title":"Towards a Knowledge Graph for Models and Algorithms in Applied Mathematics","authors":"Björn Schembera, Frank Wübbeling, Hendrik Kleikamp, Burkhard Schmidt, Aurela Shehu, Marco Reidelbach, Christine Biedinger, Jochen Fiedler, Thomas Koprucki, Dorothea Iglezakis, Dominik Göddeke","doi":"arxiv-2408.10003","DOIUrl":"https://doi.org/arxiv-2408.10003","url":null,"abstract":"Mathematical models and algorithms are an essential part of mathematical\u0000research data, as they are epistemically grounding numerical data. In order to\u0000represent models and algorithms as well as their relationship semantically to\u0000make this research data FAIR, two previously distinct ontologies were merged\u0000and extended, becoming a living knowledge graph. The link between the two\u0000ontologies is established by introducing computational tasks, as they occur in\u0000modeling, corresponding to algorithmic tasks. Moreover, controlled vocabularies\u0000are incorporated and a new class, distinguishing base quantities from specific\u0000use case quantities, was introduced. Also, both models and algorithms can now\u0000be enriched with metadata. Subject-specific metadata is particularly relevant\u0000here, such as the symmetry of a matrix or the linearity of a mathematical\u0000model. This is the only way to express specific workflows with concrete models\u0000and algorithms, as the feasible solution algorithm can only be determined if\u0000the mathematical properties of a model are known. We demonstrate this using two\u0000examples from different application areas of applied mathematics. In addition,\u0000we have already integrated over 250 research assets from applied mathematics\u0000into our knowledge graph.","PeriodicalId":501285,"journal":{"name":"arXiv - CS - Digital Libraries","volume":"20 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142219112","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Peter Buneman, Dennis Dosso, Matteo Lissandrini, Gianmaria Silvello, He Sun
In disseminating scientific and statistical data, on-line databases have almost completely replaced traditional paper-based media such as journals and reference works. Given this, can we measure the impact of a database in the same way that we measure an author's or journal's impact? To do this, we need somehow to represent a database as a set of publications, and databases typically allow a large number of possible decompositions into parts, any of which could be treated as a publication. We show that the definition of the h-index naturally extends to hierarchies, so that if a database admits some kind of hierarchical interpretation we can use this as one measure of the importance of a database; moreover, this can be computed as efficiently as one can compute the normal h-index. This also gives us a decomposition of the database that might be used for other purposes such as giving credit to the curators or contributors to the database. We illustrate the process by analyzing three widely used databases.
在传播科学和统计数据方面，在线数据库几乎完全取代了期刊和参考文献等传统纸质媒体。既然如此，我们能否像衡量作者或期刊的影响力那样来衡量数据库的影响力呢？要做到这一点，我们需要以某种方式将数据库表示为一组出版物，而数据库通常允许大量可能的分解，其中任何一部分都可以被视为出版物。我们证明了 h 指数的定义可以自然地扩展到层次结构，因此，如果数据库允许某种层次结构的解释，我们就可以用它来衡量数据库的重要性；此外，它的计算效率与计算普通的 h 指数一样高。这也为我们提供了数据库的分解方法，可用于其他目的，例如为数据库的策划者或贡献者提供荣誉。我们通过分析三个广泛使用的数据库来说明这一过程。
{"title":"Can we measure the impact of a database?","authors":"Peter Buneman, Dennis Dosso, Matteo Lissandrini, Gianmaria Silvello, He Sun","doi":"arxiv-2408.09842","DOIUrl":"https://doi.org/arxiv-2408.09842","url":null,"abstract":"In disseminating scientific and statistical data, on-line databases have\u0000almost completely replaced traditional paper-based media such as journals and\u0000reference works. Given this, can we measure the impact of a database in the\u0000same way that we measure an author's or journal's impact? To do this, we need\u0000somehow to represent a database as a set of publications, and databases\u0000typically allow a large number of possible decompositions into parts, any of\u0000which could be treated as a publication. We show that the definition of the h-index naturally extends to hierarchies,\u0000so that if a database admits some kind of hierarchical interpretation we can\u0000use this as one measure of the importance of a database; moreover, this can be\u0000computed as efficiently as one can compute the normal h-index. This also gives\u0000us a decomposition of the database that might be used for other purposes such\u0000as giving credit to the curators or contributors to the database. We illustrate\u0000the process by analyzing three widely used databases.","PeriodicalId":501285,"journal":{"name":"arXiv - CS - Digital Libraries","volume":"4 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142219113","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Perhaps among the most prominent sites on which we always encourage professors to create accounts are (ORCID), (Reserach Gate), and (Google Scholar), and how to publish and promote their research through social media or through educational platforms, conferences, and scientific workshops Then we try to explain, in the course of the research, in a smooth manner, the ways to activate accounts on these platforms, supported by pictures and a comprehensive step-by-step explanation, as a gesture to encourage the spread of the culture of electronic publishing in light of the escalation of the digital and computing revolution and the desire to catch up with its accelerating pace
{"title":"Creating Publishing Accounts for University Professors on Global Scientific Websites (ORCID, Research Gate, Google Scholar)","authors":"Ahmed Shaker Alalaq","doi":"arxiv-2408.08936","DOIUrl":"https://doi.org/arxiv-2408.08936","url":null,"abstract":"Perhaps among the most prominent sites on which we always encourage\u0000professors to create accounts are (ORCID), (Reserach Gate), and (Google\u0000Scholar), and how to publish and promote their research through social media or\u0000through educational platforms, conferences, and scientific workshops Then we\u0000try to explain, in the course of the research, in a smooth manner, the ways to\u0000activate accounts on these platforms, supported by pictures and a comprehensive\u0000step-by-step explanation, as a gesture to encourage the spread of the culture\u0000of electronic publishing in light of the escalation of the digital and\u0000computing revolution and the desire to catch up with its accelerating pace","PeriodicalId":501285,"journal":{"name":"arXiv - CS - Digital Libraries","volume":"23 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142219111","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
There has been some concern about the impact of predatory publishers on scientific research for some time. Recently, publishers that might previously have been considered `predatory' have established their bona fides, at least to the extent that they are included in citation impact scores such as the field-weighted citation impact (FWCI). These are sometimes called `grey' publishers (MDPI, Frontiers, Hindawi). In this paper, we show that the citation landscape for these grey publications is significantly different from the mainstream landscape and that affording publications in these venues the same status as publications in mainstream journals may significantly distort metrics such as the FWCI.
{"title":"An Analysis of the Impact of Gold Open Access Publications in Computer Science","authors":"Padraig Cunningham, Barry Smyth","doi":"arxiv-2408.10262","DOIUrl":"https://doi.org/arxiv-2408.10262","url":null,"abstract":"There has been some concern about the impact of predatory publishers on\u0000scientific research for some time. Recently, publishers that might previously\u0000have been considered `predatory' have established their bona fides, at least to\u0000the extent that they are included in citation impact scores such as the\u0000field-weighted citation impact (FWCI). These are sometimes called `grey'\u0000publishers (MDPI, Frontiers, Hindawi). In this paper, we show that the citation\u0000landscape for these grey publications is significantly different from the\u0000mainstream landscape and that affording publications in these venues the same\u0000status as publications in mainstream journals may significantly distort metrics\u0000such as the FWCI.","PeriodicalId":501285,"journal":{"name":"arXiv - CS - Digital Libraries","volume":"69 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142219115","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Giuseppe De Gregorio, Simon Perrin, Rodrigo C. G. Pena, Isabelle Marthot-Santaniello, Harold Mouchère
The intersection of computer vision and machine learning has emerged as a promising avenue for advancing historical research, facilitating a more profound exploration of our past. However, the application of machine learning approaches in historical palaeography is often met with criticism due to their perceived ``black box'' nature. In response to this challenge, we introduce NeuroPapyri, an innovative deep learning-based model specifically designed for the analysis of images containing ancient Greek papyri. To address concerns related to transparency and interpretability, the model incorporates an attention mechanism. This attention mechanism not only enhances the model's performance but also provides a visual representation of the image regions that significantly contribute to the decision-making process. Specifically calibrated for processing images of papyrus documents with lines of handwritten text, the model utilizes individual attention maps to inform the presence or absence of specific characters in the input image. This paper presents the NeuroPapyri model, including its architecture and training methodology. Results from the evaluation demonstrate NeuroPapyri's efficacy in document retrieval, showcasing its potential to advance the analysis of historical manuscripts.
{"title":"NeuroPapyri: A Deep Attention Embedding Network for Handwritten Papyri Retrieval","authors":"Giuseppe De Gregorio, Simon Perrin, Rodrigo C. G. Pena, Isabelle Marthot-Santaniello, Harold Mouchère","doi":"arxiv-2408.07785","DOIUrl":"https://doi.org/arxiv-2408.07785","url":null,"abstract":"The intersection of computer vision and machine learning has emerged as a\u0000promising avenue for advancing historical research, facilitating a more\u0000profound exploration of our past. However, the application of machine learning\u0000approaches in historical palaeography is often met with criticism due to their\u0000perceived ``black box'' nature. In response to this challenge, we introduce\u0000NeuroPapyri, an innovative deep learning-based model specifically designed for\u0000the analysis of images containing ancient Greek papyri. To address concerns\u0000related to transparency and interpretability, the model incorporates an\u0000attention mechanism. This attention mechanism not only enhances the model's\u0000performance but also provides a visual representation of the image regions that\u0000significantly contribute to the decision-making process. Specifically\u0000calibrated for processing images of papyrus documents with lines of handwritten\u0000text, the model utilizes individual attention maps to inform the presence or\u0000absence of specific characters in the input image. This paper presents the\u0000NeuroPapyri model, including its architecture and training methodology. Results\u0000from the evaluation demonstrate NeuroPapyri's efficacy in document retrieval,\u0000showcasing its potential to advance the analysis of historical manuscripts.","PeriodicalId":501285,"journal":{"name":"arXiv - CS - Digital Libraries","volume":"1 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142219119","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Federico Simonetta, Rishav Mondal, Luca Andrea Ludovico, Stavros Ntalampiras
The Ricordi archive, a prestigious collection of significant musical manuscripts from renowned opera composers such as Donizetti, Verdi and Puccini, has been digitized. This process has allowed us to automatically extract samples that represent various musical elements depicted on the manuscripts, including notes, staves, clefs, erasures, and composer's annotations, among others. To distinguish between digitization noise and actual music elements, a subset of these images was meticulously grouped and labeled by multiple individuals into several classes. After assessing the consistency of the annotations, we trained multiple neural network-based classifiers to differentiate between the identified music elements. The primary objective of this study was to evaluate the reliability of these classifiers, with the ultimate goal of using them for the automatic categorization of the remaining unannotated data set. The dataset, complemented by manual annotations, models, and source code used in these experiments are publicly accessible for replication purposes.
{"title":"Optical Music Recognition in Manuscripts from the Ricordi Archive","authors":"Federico Simonetta, Rishav Mondal, Luca Andrea Ludovico, Stavros Ntalampiras","doi":"arxiv-2408.10260","DOIUrl":"https://doi.org/arxiv-2408.10260","url":null,"abstract":"The Ricordi archive, a prestigious collection of significant musical\u0000manuscripts from renowned opera composers such as Donizetti, Verdi and Puccini,\u0000has been digitized. This process has allowed us to automatically extract\u0000samples that represent various musical elements depicted on the manuscripts,\u0000including notes, staves, clefs, erasures, and composer's annotations, among\u0000others. To distinguish between digitization noise and actual music elements, a\u0000subset of these images was meticulously grouped and labeled by multiple\u0000individuals into several classes. After assessing the consistency of the\u0000annotations, we trained multiple neural network-based classifiers to\u0000differentiate between the identified music elements. The primary objective of\u0000this study was to evaluate the reliability of these classifiers, with the\u0000ultimate goal of using them for the automatic categorization of the remaining\u0000unannotated data set. The dataset, complemented by manual annotations, models,\u0000and source code used in these experiments are publicly accessible for\u0000replication purposes.","PeriodicalId":501285,"journal":{"name":"arXiv - CS - Digital Libraries","volume":"38 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142219117","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Giuseppe De Gregorio, Lavinia Ferretti, Rodrigo C. G. Pena, Isabelle Marthot-Santaniello, Maria Konstantinidou, John Pavlopoulos
The study of Greek papyri from ancient Egypt is fundamental for understanding Graeco-Roman Antiquity, offering insights into various aspects of ancient culture and textual production. Palaeography, traditionally used for dating these manuscripts, relies on identifying chronologically relevant features in handwriting styles yet lacks a unified methodology, resulting in subjective interpretations and inconsistencies among experts. Recent advances in digital palaeography, which leverage artificial intelligence (AI) algorithms, have introduced new avenues for dating ancient documents. This paper presents a comparative analysis between an AI-based computational dating model and human expert palaeographers, using a novel dataset named Hell-Date comprising securely fine-grained dated Greek papyri from the Hellenistic period. The methodology involves training a convolutional neural network on visual inputs from Hell-Date to predict precise dates of papyri. In addition, experts provide palaeographic dating for comparison. To compare, we developed a new framework for error analysis that reflects the inherent imprecision of the palaeographic dating method. The results indicate that the computational model achieves performance comparable to that of human experts. These elements will help assess on a more solid basis future developments of computational algorithms to date Greek papyri.
{"title":"A New Framework for Error Analysis in Computational Paleographic Dating of Greek Papyri","authors":"Giuseppe De Gregorio, Lavinia Ferretti, Rodrigo C. G. Pena, Isabelle Marthot-Santaniello, Maria Konstantinidou, John Pavlopoulos","doi":"arxiv-2408.07779","DOIUrl":"https://doi.org/arxiv-2408.07779","url":null,"abstract":"The study of Greek papyri from ancient Egypt is fundamental for understanding\u0000Graeco-Roman Antiquity, offering insights into various aspects of ancient\u0000culture and textual production. Palaeography, traditionally used for dating\u0000these manuscripts, relies on identifying chronologically relevant features in\u0000handwriting styles yet lacks a unified methodology, resulting in subjective\u0000interpretations and inconsistencies among experts. Recent advances in digital\u0000palaeography, which leverage artificial intelligence (AI) algorithms, have\u0000introduced new avenues for dating ancient documents. This paper presents a\u0000comparative analysis between an AI-based computational dating model and human\u0000expert palaeographers, using a novel dataset named Hell-Date comprising\u0000securely fine-grained dated Greek papyri from the Hellenistic period. The\u0000methodology involves training a convolutional neural network on visual inputs\u0000from Hell-Date to predict precise dates of papyri. In addition, experts provide\u0000palaeographic dating for comparison. To compare, we developed a new framework\u0000for error analysis that reflects the inherent imprecision of the palaeographic\u0000dating method. The results indicate that the computational model achieves\u0000performance comparable to that of human experts. These elements will help\u0000assess on a more solid basis future developments of computational algorithms to\u0000date Greek papyri.","PeriodicalId":501285,"journal":{"name":"arXiv - CS - Digital Libraries","volume":"60 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142219116","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Evaluating the quality of academic journal articles is a time consuming but critical task for national research evaluation exercises, appointments and promotion. It is therefore important to investigate whether Large Language Models (LLMs) can play a role in this process. This article assesses which ChatGPT inputs (full text without tables, figures and references; title and abstract; title only) produce better quality score estimates, and the extent to which scores are affected by ChatGPT models and system prompts. The results show that the optimal input is the article title and abstract, with average ChatGPT scores based on these (30 iterations on a dataset of 51 papers) correlating at 0.67 with human scores, the highest ever reported. ChatGPT 4o is slightly better than 3.5-turbo (0.66), and 4o-mini (0.66). The results suggest that article full texts might confuse LLM research quality evaluations, even though complex system instructions for the task are more effective than simple ones. Thus, whilst abstracts contain insufficient information for a thorough assessment of rigour, they may contain strong pointers about originality and significance. Finally, linear regression can be used to convert the model scores into the human scale scores, which is 31% more accurate than guessing.
{"title":"Evaluating Research Quality with Large Language Models: An Analysis of ChatGPT's Effectiveness with Different Settings and Inputs","authors":"Mike Thelwall","doi":"arxiv-2408.06752","DOIUrl":"https://doi.org/arxiv-2408.06752","url":null,"abstract":"Evaluating the quality of academic journal articles is a time consuming but\u0000critical task for national research evaluation exercises, appointments and\u0000promotion. It is therefore important to investigate whether Large Language\u0000Models (LLMs) can play a role in this process. This article assesses which\u0000ChatGPT inputs (full text without tables, figures and references; title and\u0000abstract; title only) produce better quality score estimates, and the extent to\u0000which scores are affected by ChatGPT models and system prompts. The results\u0000show that the optimal input is the article title and abstract, with average\u0000ChatGPT scores based on these (30 iterations on a dataset of 51 papers)\u0000correlating at 0.67 with human scores, the highest ever reported. ChatGPT 4o is\u0000slightly better than 3.5-turbo (0.66), and 4o-mini (0.66). The results suggest\u0000that article full texts might confuse LLM research quality evaluations, even\u0000though complex system instructions for the task are more effective than simple\u0000ones. Thus, whilst abstracts contain insufficient information for a thorough\u0000assessment of rigour, they may contain strong pointers about originality and\u0000significance. Finally, linear regression can be used to convert the model\u0000scores into the human scale scores, which is 31% more accurate than guessing.","PeriodicalId":501285,"journal":{"name":"arXiv - CS - Digital Libraries","volume":"4 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-08-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142219118","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：