Accelerating BERT inference with GPU-efficient exit prediction

IF 3.4 3区计算机科学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS Frontiers of Computer Science Pub Date : 2024-01-22 DOI:10.1007/s11704-022-2341-9

{"title":"Accelerating BERT inference with GPU-efficient exit prediction","authors":"","doi":"10.1007/s11704-022-2341-9","DOIUrl":null,"url":null,"abstract":"<h3>Abstract</h3> <p>BERT is a representative pre-trained language model that has drawn extensive attention for significant improvements in downstream Natural Language Processing (NLP) tasks. The complex architecture and massive parameters bring BERT competitive performance but also result in slow speed at model inference time. To speed up BERT inference, FastBERT realizes adaptive inference with an acceptable drop in accuracy based on knowledge distillation and the early-exit technique. However, many factors may limit the performance of FastBERT, such as the teacher classifier that is not knowledgeable enough, the batch size shrinkage and the redundant computation of student classifiers. To overcome these limitations, we propose a new BERT inference method with GPU-Efficient Exit Prediction (GEEP). GEEP leverages the <em>shared exit loss</em> to simplify the training process of FastBERT from two steps into only one step and makes the teacher classifier more knowledgeable by feeding diverse Transformer outputs to the teacher classifier. In addition, the <em>exit layer prediction</em> technique is proposed to utilize a GPU hash table to handle the token-level exit layer distribution and to sort test samples by predicted exit layers. In this way, GEEP can avoid batch size shrinkage and redundant computation of student classifiers. Experimental results on twelve public English and Chinese NLP datasets prove the effectiveness of the proposed approach. The source codes of GEEP will be released to the public upon paper acceptance.</p>","PeriodicalId":12640,"journal":{"name":"Frontiers of Computer Science","volume":"58 3 1","pages":""},"PeriodicalIF":3.4000,"publicationDate":"2024-01-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Frontiers of Computer Science","FirstCategoryId":"94","ListUrlMain":"https://doi.org/10.1007/s11704-022-2341-9","RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"COMPUTER SCIENCE, INFORMATION SYSTEMS","Score":null,"Total":0}

引用次数: 0

Abstract

BERT is a representative pre-trained language model that has drawn extensive attention for significant improvements in downstream Natural Language Processing (NLP) tasks. The complex architecture and massive parameters bring BERT competitive performance but also result in slow speed at model inference time. To speed up BERT inference, FastBERT realizes adaptive inference with an acceptable drop in accuracy based on knowledge distillation and the early-exit technique. However, many factors may limit the performance of FastBERT, such as the teacher classifier that is not knowledgeable enough, the batch size shrinkage and the redundant computation of student classifiers. To overcome these limitations, we propose a new BERT inference method with GPU-Efficient Exit Prediction (GEEP). GEEP leverages the shared exit loss to simplify the training process of FastBERT from two steps into only one step and makes the teacher classifier more knowledgeable by feeding diverse Transformer outputs to the teacher classifier. In addition, the exit layer prediction technique is proposed to utilize a GPU hash table to handle the token-level exit layer distribution and to sort test samples by predicted exit layers. In this way, GEEP can avoid batch size shrinkage and redundant computation of student classifiers. Experimental results on twelve public English and Chinese NLP datasets prove the effectiveness of the proposed approach. The source codes of GEEP will be released to the public upon paper acceptance.

查看原文

微信好友朋友圈 QQ好友复制链接

本刊更多论文

利用 GPU 高效出口预测加速 BERT 推断

摘要 BERT 是一种具有代表性的预训练语言模型，因其在下游自然语言处理（NLP）任务中的显著改进而受到广泛关注。复杂的架构和庞大的参数为 BERT 带来了极具竞争力的性能，但同时也导致模型推理速度缓慢。为了加快 BERT 的推理速度，FastBERT 基于知识提炼和早期退出技术实现了自适应推理，同时降低了可接受的准确度。然而，许多因素可能会限制 FastBERT 的性能，例如教师分类器的知识不够丰富、批量规模缩减以及学生分类器的冗余计算。为了克服这些限制，我们提出了一种采用 GPU 高效出口预测（GEEP）的新 BERT 推断方法。GEEP 利用共享出口损失，将 FastBERT 的训练过程从两步简化为一步，并通过向教师分类器提供不同的 Transformer 输出，使教师分类器的知识更丰富。此外，还提出了出口层预测技术，利用 GPU 哈希表来处理令牌级出口层分布，并根据预测的出口层对测试样本进行排序。通过这种方法，GEEP 可以避免批量缩减和学生分类器的冗余计算。在 12 个公开的中英文 NLP 数据集上的实验结果证明了所提方法的有效性。GEEP 的源代码将在论文被接受后公开发布。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文去求助

来源期刊

Frontiers of Computer Science COMPUTER SCIENCE, INFORMATION SYSTEMS-COMPUTER SCIENCE, SOFTWARE ENGINEERING

CiteScore

8.60

自引率

2.40%

发文量

799

审稿时长

6-12 weeks

期刊介绍： Frontiers of Computer Science aims to provide a forum for the publication of peer-reviewed papers to promote rapid communication and exchange between computer scientists. The journal publishes research papers and review articles in a wide range of topics, including: architecture, software, artificial intelligence, theoretical computer science, networks and communication, information systems, multimedia and graphics, information security, interdisciplinary, etc. The journal especially encourages papers from new emerging and multidisciplinary areas, as well as papers reflecting the international trends of research and development and on special topics reporting progress made by Chinese computer scientists.