{"title":"Towards understanding evolution of science through language model series","authors":"Junjie Dong, Zhuoqi Lyu, Qing Ke","doi":"arxiv-2409.09636","DOIUrl":null,"url":null,"abstract":"We introduce AnnualBERT, a series of language models designed specifically to\ncapture the temporal evolution of scientific text. Deviating from the\nprevailing paradigms of subword tokenizations and \"one model to rule them all\",\nAnnualBERT adopts whole words as tokens and is composed of a base RoBERTa model\npretrained from scratch on the full-text of 1.7 million arXiv papers published\nuntil 2008 and a collection of progressively trained models on arXiv papers at\nan annual basis. We demonstrate the effectiveness of AnnualBERT models by\nshowing that they not only have comparable performances in standard tasks but\nalso achieve state-of-the-art performances on domain-specific NLP tasks as well\nas link prediction tasks in the arXiv citation network. We then utilize probing\ntasks to quantify the models' behavior in terms of representation learning and\nforgetting as time progresses. Our approach enables the pretrained models to\nnot only improve performances on scientific text processing tasks but also to\nprovide insights into the development of scientific discourse over time. The\nseries of the models is available at https://huggingface.co/jd445/AnnualBERTs.","PeriodicalId":501285,"journal":{"name":"arXiv - CS - Digital Libraries","volume":"4 1","pages":""},"PeriodicalIF":0.0000,"publicationDate":"2024-09-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - CS - Digital Libraries","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2409.09636","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0
Abstract
We introduce AnnualBERT, a series of language models designed specifically to
capture the temporal evolution of scientific text. Deviating from the
prevailing paradigms of subword tokenizations and "one model to rule them all",
AnnualBERT adopts whole words as tokens and is composed of a base RoBERTa model
pretrained from scratch on the full-text of 1.7 million arXiv papers published
until 2008 and a collection of progressively trained models on arXiv papers at
an annual basis. We demonstrate the effectiveness of AnnualBERT models by
showing that they not only have comparable performances in standard tasks but
also achieve state-of-the-art performances on domain-specific NLP tasks as well
as link prediction tasks in the arXiv citation network. We then utilize probing
tasks to quantify the models' behavior in terms of representation learning and
forgetting as time progresses. Our approach enables the pretrained models to
not only improve performances on scientific text processing tasks but also to
provide insights into the development of scientific discourse over time. The
series of the models is available at https://huggingface.co/jd445/AnnualBERTs.