Transactions of the Association for Computational Linguistics最新文献

英文中文

Transparency Helps Reveal When Language Models Learn Meaning 透明度有助于揭示语言模型何时学习意义

IF 10.9 1区计算机科学 Q2 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Transactions of the Association for Computational Linguistics

Pub Date : 2022-10-14 DOI: 10.1162/tacl_a_00565

Zhaofeng Wu, Will Merrill, Hao Peng, Iz Beltagy, Noah A. Smith

Many current NLP systems are built from language models trained to optimize unsupervised objectives on large amounts of raw text. Under what conditions might such a procedure acquire meaning? Our systematic experiments with synthetic data reveal that, with languages where all expressions have context-independent denotations (i.e., languages with strong transparency), both autoregressive and masked language models successfully learn to emulate semantic relations between expressions. However, when denotations are changed to be context-dependent with the language otherwise unmodified, this ability degrades. Turning to natural language, our experiments with a specific phenomenon—referential opacity—add to the growing body of evidence that current language models do not represent natural language semantics well. We show this failure relates to the context-dependent nature of natural language form-meaning mappings.

许多当前的NLP系统都是由经过训练的语言模型构建的，这些模型用于优化大量原始文本上的无监督目标。在什么条件下，这样的程序才能获得意义？我们对合成数据的系统实验表明，在所有表达式都具有上下文无关指称的语言（即具有强透明度的语言）中，自回归和掩蔽语言模型都成功地学习了模拟表达式之间的语义关系。然而，当指称被更改为上下文相关，而语言在其他方面没有修改时，这种能力就会下降。谈到自然语言，我们对一种特定现象——指称不透明度——的实验增加了越来越多的证据，证明当前的语言模型不能很好地代表自然语言语义。我们表明，这种失败与自然语言形式-意义映射的上下文依赖性有关。

引用次数: 2

Explainable Abuse Detection as Intent Classification and Slot Filling 可解释的滥用检测:意图分类和槽填充

IF 10.9 1区计算机科学 Q2 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Transactions of the Association for Computational Linguistics

Pub Date : 2022-10-06 DOI: 10.1162/tacl_a_00527

Agostina Calabrese, Björn Ross, Mirella Lapata

Abstract To proactively offer social media users a safe online experience, there is a need for systems that can detect harmful posts and promptly alert platform moderators. In order to guarantee the enforcement of a consistent policy, moderators are provided with detailed guidelines. In contrast, most state-of-the-art models learn what abuse is from labeled examples and as a result base their predictions on spurious cues, such as the presence of group identifiers, which can be unreliable. In this work we introduce the concept of policy-aware abuse detection, abandoning the unrealistic expectation that systems can reliably learn which phenomena constitute abuse from inspecting the data alone. We propose a machine-friendly representation of the policy that moderators wish to enforce, by breaking it down into a collection of intents and slots. We collect and annotate a dataset of 3,535 English posts with such slots, and show how architectures for intent classification and slot filling can be used for abuse detection, while providing a rationale for model decisions.1

摘要为了主动为社交媒体用户提供安全的在线体验，需要一种能够检测有害帖子并及时提醒平台版主的系统。为了保证执行一致的政策，向主持人提供了详细的指导方针。相比之下，大多数最先进的模型从标记的例子中了解什么是滥用，因此他们的预测基于虚假的线索，例如组标识符的存在，这可能是不可靠的。在这项工作中，我们引入了策略感知滥用检测的概念，放弃了不切实际的期望，即系统可以通过单独检查数据来可靠地了解哪些现象构成滥用。我们提出了一种机器友好的策略表示，主持人希望通过将其分解为意图和槽的集合来执行该策略。我们收集并注释了一个包含3535篇带有此类空位的英语帖子的数据集，并展示了如何将意图分类和空位填充的架构用于滥用检测，同时为模型决策提供了依据。1

引用次数: 4

Domain-Specific Word Embeddings with Structure Prediction 具有结构预测的领域特定词嵌入

IF 10.9 1区计算机科学 Q2 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Transactions of the Association for Computational Linguistics

Pub Date : 2022-10-06 DOI: 10.1162/tacl_a_00538

Stephanie Brandl, D. Lassner, A. Baillot, S. Nakajima

Complementary to finding good general word embeddings, an important question for representation learning is to find dynamic word embeddings, for example, across time or domain. Current methods do not offer a way to use or predict information on structure between sub-corpora, time or domain and dynamic embeddings can only be compared after post-alignment. We propose novel word embedding methods that provide general word representations for the whole corpus, domain- specific representations for each sub-corpus, sub-corpus structure, and embedding alignment simultaneously. We present an empirical evaluation on New York Times articles and two English Wikipedia datasets with articles on science and philosophy. Our method, called Word2Vec with Structure Prediction (W2VPred), provides better performance than baselines in terms of the general analogy tests, domain-specific analogy tests, and multiple specific word embedding evaluations as well as structure prediction performance when no structure is given a priori. As a use case in the field of Digital Humanities we demonstrate how to raise novel research questions for high literature from the German Text Archive.

作为寻找良好的通用单词嵌入的补充，表示学习的一个重要问题是寻找动态单词嵌入，例如，跨时间或域。当前的方法没有提供一种使用或预测子语料库、时间或域之间的结构信息的方法，并且动态嵌入只能在后对齐后进行比较。我们提出了新的单词嵌入方法，为整个语料库提供通用的单词表示，为每个子语料库提供特定领域的表示，同时提供子语料库结构和嵌入对齐。我们对《纽约时报》的文章和维基百科的两个英文数据集进行了实证评估，其中包含了关于科学和哲学的文章。我们的方法称为Word2Verc with Structure Prediction（W2VPred），在一般类比测试、特定领域类比测试、多个特定单词嵌入评估以及在没有先验结构的情况下的结构预测性能方面，它比基线提供了更好的性能。作为数字人文领域的一个用例，我们展示了如何从德国文本档案馆为高级文学提出新颖的研究问题。

{"title":"Domain-Specific Word Embeddings with Structure Prediction","authors":"Stephanie Brandl, D. Lassner, A. Baillot, S. Nakajima","doi":"10.1162/tacl_a_00538","DOIUrl":"https://doi.org/10.1162/tacl_a_00538","url":null,"abstract":"Complementary to finding good general word embeddings, an important question for representation learning is to find dynamic word embeddings, for example, across time or domain. Current methods do not offer a way to use or predict information on structure between sub-corpora, time or domain and dynamic embeddings can only be compared after post-alignment. We propose novel word embedding methods that provide general word representations for the whole corpus, domain- specific representations for each sub-corpus, sub-corpus structure, and embedding alignment simultaneously. We present an empirical evaluation on New York Times articles and two English Wikipedia datasets with articles on science and philosophy. Our method, called Word2Vec with Structure Prediction (W2VPred), provides better performance than baselines in terms of the general analogy tests, domain-specific analogy tests, and multiple specific word embedding evaluations as well as structure prediction performance when no structure is given a priori. As a use case in the field of Digital Humanities we demonstrate how to raise novel research questions for high literature from the German Text Archive.","PeriodicalId":33559,"journal":{"name":"Transactions of the Association for Computational Linguistics","volume":"11 1","pages":"320-335"},"PeriodicalIF":10.9,"publicationDate":"2022-10-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"43780471","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering 面向开放领域问答的检索增强生成(RAG)模型的领域适应性改进

IF 10.9 1区计算机科学 Q2 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Transactions of the Association for Computational Linguistics

Pub Date : 2022-10-06 DOI: 10.1162/tacl_a_00530

Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen, Tharindu Kaluarachchi, R. Rana, Suranga Nanayakkara

Retrieval Augment Generation (RAG) is a recent advancement in Open-Domain Question Answering (ODQA). RAG has only been trained and explored with a Wikipedia-based external knowledge base and is not optimized for use in other specialized domains such as healthcare and news. In this paper, we evaluate the impact of joint training of the retriever and generator components of RAG for the task of domain adaptation in ODQA. We propose RAG-end2end, an extension to RAG that can adapt to a domain-specific knowledge base by updating all components of the external knowledge base during training. In addition, we introduce an auxiliary training signal to inject more domain-specific knowledge. This auxiliary signal forces RAG-end2end to reconstruct a given sentence by accessing the relevant information from the external knowledge base. Our novel contribution is that, unlike RAG, RAG-end2end does joint training of the retriever and generator for the end QA task and domain adaptation. We evaluate our approach with datasets from three domains: COVID-19, News, and Conversations, and achieve significant performance improvements compared to the original RAG model. Our work has been open-sourced through the HuggingFace Transformers library, attesting to our work’s credibility and technical consistency.

检索增强生成(RAG)是开放域问答(ODQA)技术的最新进展。RAG仅通过基于wikipedia的外部知识库进行了培训和探索，并未针对医疗保健和新闻等其他专业领域进行优化。在本文中，我们评估了RAG的检索器和生成器组件的联合训练对ODQA领域自适应任务的影响。我们提出了RAG-end2end，这是RAG的扩展，可以通过在培训期间更新外部知识库的所有组件来适应特定于领域的知识库。此外，我们引入了辅助训练信号来注入更多的领域特定知识。这个辅助信号迫使RAG-end2end通过访问外部知识库中的相关信息来重构给定的句子。我们的新贡献是，与RAG不同，RAG-end2end对最终QA任务和域适应的检索者和生成器进行联合训练。我们用来自三个领域的数据集评估了我们的方法:COVID-19、新闻和对话，与原始RAG模型相比，实现了显著的性能改进。我们的工作已经通过HuggingFace Transformers库开源，证明了我们工作的可信度和技术一致性。

{"title":"Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering","authors":"Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen, Tharindu Kaluarachchi, R. Rana, Suranga Nanayakkara","doi":"10.1162/tacl_a_00530","DOIUrl":"https://doi.org/10.1162/tacl_a_00530","url":null,"abstract":"Retrieval Augment Generation (RAG) is a recent advancement in Open-Domain Question Answering (ODQA). RAG has only been trained and explored with a Wikipedia-based external knowledge base and is not optimized for use in other specialized domains such as healthcare and news. In this paper, we evaluate the impact of joint training of the retriever and generator components of RAG for the task of domain adaptation in ODQA. We propose RAG-end2end, an extension to RAG that can adapt to a domain-specific knowledge base by updating all components of the external knowledge base during training. In addition, we introduce an auxiliary training signal to inject more domain-specific knowledge. This auxiliary signal forces RAG-end2end to reconstruct a given sentence by accessing the relevant information from the external knowledge base. Our novel contribution is that, unlike RAG, RAG-end2end does joint training of the retriever and generator for the end QA task and domain adaptation. We evaluate our approach with datasets from three domains: COVID-19, News, and Conversations, and achieve significant performance improvements compared to the original RAG model. Our work has been open-sourced through the HuggingFace Transformers library, attesting to our work’s credibility and technical consistency.","PeriodicalId":33559,"journal":{"name":"Transactions of the Association for Computational Linguistics","volume":"55 1","pages":"1-17"},"PeriodicalIF":10.9,"publicationDate":"2022-10-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"64440765","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 10

FRMT: A Benchmark for Few-Shot Region-Aware Machine Translation 少镜头区域感知机器翻译的基准

IF 10.9 1区计算机科学 Q2 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Transactions of the Association for Computational Linguistics

Pub Date : 2022-10-01 DOI: 10.1162/tacl_a_00568

Parker Riley, Timothy Dozat, Jan A. Botha, Xavier García, Dan Garrette, Jason Riesa, Orhan Firat, Noah Constant

We present FRMT, a new dataset and evaluation benchmark for Few-shot Region-aware Machine Translation, a type of style-targeted translation. The dataset consists of professional translations from English into two regional variants each of Portuguese and Mandarin Chinese. Source documents are selected to enable detailed analysis of phenomena of interest, including lexically distinct terms and distractor terms. We explore automatic evaluation metrics for FRMT and validate their correlation with expert human evaluation across both region-matched and mismatched rating scenarios. Finally, we present a number of baseline models for this task, and offer guidelines for how researchers can train, evaluate, and compare their own models. Our dataset and evaluation code are publicly available: https://bit.ly/frmt-task.

我们提出了FRMT，这是一种新的数据集和评估基准，用于少镜头区域感知机器翻译，一种风格目标翻译。该数据集包括从英语到葡萄牙语和普通话两种地区变体的专业翻译。选择源文档可以对感兴趣的现象进行详细分析，包括词汇上不同的术语和干扰词。我们探索了FRMT的自动评估指标，并在区域匹配和不匹配的评级场景中验证了它们与专家人工评估的相关性。最后，我们为这项任务提供了一些基线模型，并为研究人员如何训练、评估和比较自己的模型提供了指导。我们的数据集和评估代码是公开的：https://bit.ly/frmt-task.

引用次数: 6

Meta-Learning a Cross-lingual Manifold for Semantic Parsing 元学习——用于语义分析的跨语言流形

IF 10.9 1区计算机科学 Q2 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Transactions of the Association for Computational Linguistics

Pub Date : 2022-09-26 DOI: 10.1162/tacl_a_00533

Tom Sherborne, Mirella Lapata

Localizing a semantic parser to support new languages requires effective cross-lingual generalization. Recent work has found success with machine-translation or zero-shot methods, although these approaches can struggle to model how native speakers ask questions. We consider how to effectively leverage minimal annotated examples in new languages for few-shot cross-lingual semantic parsing. We introduce a first-order meta-learning algorithm to train a semantic parser with maximal sample efficiency during cross-lingual transfer. Our algorithm uses high-resource languages to train the parser and simultaneously optimizes for cross-lingual generalization to lower-resource languages. Results across six languages on ATIS demonstrate that our combination of generalization steps yields accurate semantic parsers sampling ≤10% of source training data in each new language. Our approach also trains a competitive model on Spider using English with generalization to Chinese similarly sampling ≤10% of training data.1

本地化语义解析器以支持新语言需要有效的跨语言泛化。最近的研究发现，机器翻译或零射击方法取得了成功，尽管这些方法很难模拟母语人士提问的方式。我们考虑如何有效地利用新语言中最小的带注释的示例来进行几次跨语言语义解析。我们引入一阶元学习算法，在跨语言迁移过程中以最大的样本效率训练语义解析器。我们的算法使用高资源语言来训练解析器，同时优化跨语言泛化到低资源语言。ATIS在六种语言上的结果表明，我们的泛化步骤组合产生了准确的语义解析器，对每种新语言的源训练数据采样≤10%。我们的方法还在Spider上训练了一个竞争模型，使用英语对中文进行类似的推广，采样≤10%的训练数据

引用次数: 7

OPAL: Ontology-Aware Pretrained Language Model for End-to-End Task-Oriented Dialogue 面向端到端任务对话的本体感知预训练语言模型

IF 10.9 1区计算机科学 Q2 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Transactions of the Association for Computational Linguistics

Pub Date : 2022-09-10 DOI: 10.1162/tacl_a_00534

Zhi Chen, Yuncong Liu, Lu Chen, Su Zhu, Mengyue Wu, Kai Yu

This paper presents an ontology-aware pretrained language model (OPAL) for end-to-end task-oriented dialogue (TOD). Unlike chit-chat dialogue models, task-oriented dialogue models fulfill at least two task-specific modules: Dialogue state tracker (DST) and response generator (RG). The dialogue state consists of the domain-slot-value triples, which are regarded as the user’s constraints to search the domain-related databases. The large-scale task-oriented dialogue data with the annotated structured dialogue state usually are inaccessible. It prevents the development of the pretrained language model for the task-oriented dialogue. We propose a simple yet effective pretraining method to alleviate this problem, which consists of two pretraining phases. The first phase is to pretrain on large-scale contextual text data, where the structured information of the text is extracted by the information extracting tool. To bridge the gap between the pretraining method and downstream tasks, we design two pretraining tasks: ontology-like triple recovery and next-text generation, which simulates the DST and RG, respectively. The second phase is to fine-tune the pretrained model on the TOD data. The experimental results show that our proposed method achieves an exciting boost and obtains competitive performance even without any TOD data on CamRest676 and MultiWOZ benchmarks.

提出了一种面向端到端任务对话(TOD)的基于本体感知的预训练语言模型(OPAL)。与闲聊对话模型不同，面向任务的对话模型至少实现两个特定于任务的模块:对话状态跟踪器(DST)和响应生成器(RG)。对话状态由域-槽-值三元组组成，作为用户搜索域相关数据库的约束。具有带注释的结构化对话状态的面向任务的大规模对话数据通常是不可访问的。它阻碍了面向任务对话的预训练语言模型的发展。我们提出了一种简单而有效的预训练方法来缓解这一问题，该方法包括两个预训练阶段。第一阶段是对大规模上下文文本数据进行预训练，通过信息提取工具提取文本的结构化信息。为了弥补预训练方法与下游任务之间的差距，我们设计了两个预训练任务:类本体三重恢复和下一代文本生成，分别模拟了DST和RG。第二阶段是对TOD数据的预训练模型进行微调。实验结果表明，在CamRest676和MultiWOZ基准测试中，即使没有任何TOD数据，我们提出的方法也取得了令人兴奋的提升，并获得了具有竞争力的性能。

{"title":"OPAL: Ontology-Aware Pretrained Language Model for End-to-End Task-Oriented Dialogue","authors":"Zhi Chen, Yuncong Liu, Lu Chen, Su Zhu, Mengyue Wu, Kai Yu","doi":"10.1162/tacl_a_00534","DOIUrl":"https://doi.org/10.1162/tacl_a_00534","url":null,"abstract":"This paper presents an ontology-aware pretrained language model (OPAL) for end-to-end task-oriented dialogue (TOD). Unlike chit-chat dialogue models, task-oriented dialogue models fulfill at least two task-specific modules: Dialogue state tracker (DST) and response generator (RG). The dialogue state consists of the domain-slot-value triples, which are regarded as the user’s constraints to search the domain-related databases. The large-scale task-oriented dialogue data with the annotated structured dialogue state usually are inaccessible. It prevents the development of the pretrained language model for the task-oriented dialogue. We propose a simple yet effective pretraining method to alleviate this problem, which consists of two pretraining phases. The first phase is to pretrain on large-scale contextual text data, where the structured information of the text is extracted by the information extracting tool. To bridge the gap between the pretraining method and downstream tasks, we design two pretraining tasks: ontology-like triple recovery and next-text generation, which simulates the DST and RG, respectively. The second phase is to fine-tune the pretrained model on the TOD data. The experimental results show that our proposed method achieves an exciting boost and obtains competitive performance even without any TOD data on CamRest676 and MultiWOZ benchmarks.","PeriodicalId":33559,"journal":{"name":"Transactions of the Association for Computational Linguistics","volume":"11 1","pages":"68-84"},"PeriodicalIF":10.9,"publicationDate":"2022-09-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"44597473","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 4

Investigating Reasons for Disagreement in Natural Language Inference 探究自然语言推理中分歧的原因

IF 10.9 1区计算机科学 Q2 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Transactions of the Association for Computational Linguistics

Pub Date : 2022-09-07 DOI: 10.1162/tacl_a_00523

Nan Jiang, M. Marneffe

Abstract We investigate how disagreement in natural language inference (NLI) annotation arises. We developed a taxonomy of disagreement sources with 10 categories spanning 3 high- level classes. We found that some disagreements are due to uncertainty in the sentence meaning, others to annotator biases and task artifacts, leading to different interpretations of the label distribution. We explore two modeling approaches for detecting items with potential disagreement: a 4-way classification with a “Complicated” label in addition to the three standard NLI labels, and a multilabel classification approach. We found that the multilabel classification is more expressive and gives better recall of the possible interpretations in the data.

摘要本文研究了自然语言推理(NLI)注释中分歧的产生。我们开发了一个有10个类别的分歧来源的分类法，跨越3个高级类。我们发现，一些分歧是由于句子意义的不确定性，另一些是由于注释者偏见和任务人为因素，导致对标签分布的不同解释。我们探索了两种用于检测潜在不一致项的建模方法:除了三个标准NLI标签外，还有一个“复杂”标签的四向分类方法，以及一个多标签分类方法。我们发现，多标签分类更具表现力，并能更好地回忆数据中可能的解释。

引用次数: 19

Adapting to the Long Tail: A Meta-Analysis of Transfer Learning Research for Language Understanding Tasks. 适应长尾:语言理解任务迁移学习研究的元分析。

IF 10.9 1区计算机科学 Q2 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Transactions of the Association for Computational Linguistics

Pub Date : 2022-09-07 eCollection Date: 2022-10-01 DOI: 10.1162/tacl_a_00500

Aakanksha Naik, Jill Lehman, Carolyn Rosé

Natural language understanding (NLU) has made massive progress driven by large benchmarks, but benchmarks often leave a long tail of infrequent phenomena underrepresented. We reflect on the question: Have transfer learning methods sufficiently addressed the poor performance of benchmark-trained models on the long tail? We conceptualize the long tail using macro-level dimensions (underrepresented genres, topics, etc.), and perform a qualitative meta-analysis of 100 representative papers on transfer learning research for NLU. Our analysis asks three questions: (i) Which long tail dimensions do transfer learning studies target? (ii) Which properties of adaptation methods help improve performance on the long tail? (iii) Which methodological gaps have greatest negative impact on long tail performance? Our answers highlight major avenues for future research in transfer learning for the long tail. Lastly, using our meta-analysis framework, we perform a case study comparing the performance of various adaptation methods on clinical narratives, which provides interesting insights that may enable us to make progress along these future avenues.

在大型基准测试的推动下，自然语言理解(NLU)已经取得了巨大的进步，但是基准测试通常会留下一个不常见现象的长尾。我们反思了这样一个问题:迁移学习方法是否充分解决了长尾基准训练模型表现不佳的问题?我们使用宏观层面的维度(未被充分代表的类型、主题等)对长尾进行了概念化，并对100篇关于NLU迁移学习研究的代表性论文进行了定性元分析。我们的分析提出了三个问题:(i)迁移学习研究的目标是哪些长尾维度?(ii)适应方法的哪些特性有助于提高长尾的性能?(iii)哪些方法上的差距对长尾绩效的负面影响最大?我们的回答强调了长尾迁移学习未来研究的主要途径。最后，利用我们的荟萃分析框架，我们进行了一个案例研究，比较了各种适应方法在临床叙述中的表现，这提供了有趣的见解，可能使我们在这些未来的途径上取得进展。

{"title":"Adapting to the Long Tail: A Meta-Analysis of Transfer Learning Research for Language Understanding Tasks.","authors":"Aakanksha Naik, Jill Lehman, Carolyn Rosé","doi":"10.1162/tacl_a_00500","DOIUrl":"https://doi.org/10.1162/tacl_a_00500","url":null,"abstract":"Natural language understanding (NLU) has made massive progress driven by large benchmarks, but benchmarks often leave a long tail of infrequent phenomena underrepresented. We reflect on the question: Have transfer learning methods sufficiently addressed the poor performance of benchmark-trained models on the long tail? We conceptualize the long tail using macro-level dimensions (underrepresented genres, topics, etc.), and perform a qualitative meta-analysis of 100 representative papers on transfer learning research for NLU. Our analysis asks three questions: (i) Which long tail dimensions do transfer learning studies target? (ii) Which properties of adaptation methods help improve performance on the long tail? (iii) Which methodological gaps have greatest negative impact on long tail performance? Our answers highlight major avenues for future research in transfer learning for the long tail. Lastly, using our meta-analysis framework, we perform a case study comparing the performance of various adaptation methods on clinical narratives, which provides interesting insights that may enable us to make progress along these future avenues.","PeriodicalId":33559,"journal":{"name":"Transactions of the Association for Computational Linguistics","volume":"10 ","pages":"956-980"},"PeriodicalIF":10.9,"publicationDate":"2022-09-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9590102/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"40667339","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 2

Efficient Methods for Natural Language Processing: A Survey 自然语言处理的有效方法综述

IF 10.9 1区计算机科学 Q2 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Transactions of the Association for Computational Linguistics

Pub Date : 2022-08-31 DOI: 10.1162/tacl_a_00577

Marcos Vinícius Treviso, Tianchu Ji, Ji-Ung Lee, Betty van Aken, Qingqing Cao, Manuel R. Ciosici, Michael Hassid, Kenneth Heafield, Sara Hooker, Pedro Henrique Martins, André F. T. Martins, Peter Milder, Colin Raffel, Edwin Simpson, N. Slonim, Niranjan Balasubramanian, Leon Derczynski, Roy Schwartz

Abstract Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data; however, using only scale to improve performance means that resource consumption also grows. Such resources include data, time, storage, or energy, all of which are naturally limited and unevenly distributed. This motivates research into efficient methods that require fewer resources to achieve similar results. This survey synthesizes and relates current methods and findings in efficient NLP. We aim to provide both guidance for conducting NLP under limited resources, and point towards promising research directions for developing more efficient methods.

近年来，自然语言处理(NLP)在缩放模型参数和训练数据方面取得了引人注目的成果;但是，仅使用扩展来提高性能意味着资源消耗也会增加。这些资源包括数据、时间、存储或能量，所有这些资源自然都是有限的，而且分布不均。这促使人们研究需要更少资源来获得类似结果的有效方法。本调查综合并联系了高效自然语言处理的现有方法和发现。我们的目标是为在有限资源下进行自然语言处理提供指导，并为开发更有效的方法指出有希望的研究方向。

引用次数: 38

首页上一页

下一页尾页

类型

全部化学•材料生命科学医学物理工程技术环境•农林材料科学地球科学法学管理学化学环境科学与生态学计算机科学教育学经济学农林科学人文科学生物学数学物理与天体物理心理学综合性期刊其他工业工程理学历史学农学文学信息工程

数据库

全部 ACS Publications Elsevier ieeexplore Springer The Royal Society of Chemistry Wiley

期刊

Transactions of the Association for Computational Linguistics

全部 Acc. Chem. Res. ACS Applied Bio Materials ACS Appl. Electron. Mater. ACS Appl. Energy Mater. ACS Appl. Mater. Interfaces ACS Appl. Nano Mater. ACS Appl. Polym. Mater. ACS BIOMATER-SCI ENG ACS Catal. ACS Cent. Sci. ACS Chem. Biol. ACS Chemical Health & Safety ACS Chem. Neurosci. ACS Comb. Sci. ACS Earth Space Chem. ACS Energy Lett. ACS Infect. Dis. ACS Macro Lett. ACS Mater. Lett. ACS Med. Chem. Lett. ACS Nano ACS Omega ACS Photonics ACS Sens. ACS Sustainable Chem. Eng. ACS Synth. Biol. Anal. Chem. BIOCHEMISTRY-US Bioconjugate Chem. BIOMACROMOLECULES Chem. Res. Toxicol. Chem. Rev. Chem. Mater. CRYST GROWTH DES ENERG FUEL Environ. Sci. Technol. Environ. Sci. Technol. Lett. Eur. J. Inorg. Chem. IND ENG CHEM RES Inorg. Chem. J. Agric. Food. Chem. J. Chem. Eng. Data J. Chem. Educ. J. Chem. Inf. Model. J. Chem. Theory Comput. J. Med. Chem. J. Nat. Prod. J PROTEOME RES J. Am. Chem. Soc. LANGMUIR MACROMOLECULES Mol. Pharmaceutics Nano Lett. Org. Lett. ORG PROCESS RES DEV ORGANOMETALLICS J. Org. Chem. J. Phys. Chem. J. Phys. Chem. A J. Phys. Chem. B J. Phys. Chem. C J. Phys. Chem. Lett. Analyst Anal. Methods Biomater. Sci. Catal. Sci. Technol. Chem. Commun. Chem. Soc. Rev. CHEM EDUC RES PRACT CRYSTENGCOMM Dalton Trans. Energy Environ. Sci. ENVIRON SCI-NANO ENVIRON SCI-PROC IMP ENVIRON SCI-WAT RES Faraday Discuss. Food Funct. Green Chem. Inorg. Chem. Front. Integr. Biol. J. Anal. At. Spectrom. J. Mater. Chem. A J. Mater. Chem. B J. Mater. Chem. C Lab Chip Mater. Chem. Front. Mater. Horiz. MEDCHEMCOMM Metallomics Mol. Biosyst. Mol. Syst. Des. Eng. Nanoscale Nanoscale Horiz. Nat. Prod. Rep. New J. Chem. Org. Biomol. Chem. Org. Chem. Front. PHOTOCH PHOTOBIO SCI PCCP Polym. Chem.

﹀