{"title":"AtRec:加速 CPU 上的推荐模型训练","authors":"Siqi Wang;Tianyu Feng;Hailong Yang;Xin You;Bangduo Chen;Tongxuan Liu;Zhongzhi Luan;Depei Qian","doi":"10.1109/TPDS.2024.3381186","DOIUrl":null,"url":null,"abstract":"The popularity of recommendation models and the enhanced AI processing capability of CPUs have provided massive performance opportunities to deliver satisfactory experiences to a large number of users. Unfortunately, existing recommendation model training methods fail to achieve high efficiency due to unique challenges such as dynamic shape and high parallelism. To address the above limitations, we comprehensively study the distinctive characteristics of recommendation models and discover several unexploited optimization opportunities. To exploit such opportunities, we propose \n<italic>AtRec</i>\n, a high-performant recommendation model training engine that significantly accelerates the training process on CPUs. Specifically, \n<italic>AtRec</i>\n presents comprehensive approach of training that employs operator-level and graph-level joint optimizations and runtime optimization. At the operator-level, \n<italic>AtRec</i>\n identifies and optimizes the time-consuming operators, which enables further efficient graph-level optimizations. At the graph-level, \n<italic>AtRec</i>\n conducts an in-depth analysis of the inefficiencies in several frequently used subgraphs, enables further performance improvement via eliminating redundant computations and memory accesses. In addition, to achieve better runtime performance, \n<italic>AtRec</i>\n also identifies inefficiencies prevalent in the current scheduling and proposes runtime batching. The experiment results demonstrate that \n<italic>AtRec</i>\n can significantly outperform state-of-the-art recommendation model training engines. We have open sourced the implementation and corresponding data of \n<italic>AtRec</i>\n to boost research in this direction.","PeriodicalId":13257,"journal":{"name":"IEEE Transactions on Parallel and Distributed Systems","volume":null,"pages":null},"PeriodicalIF":5.6000,"publicationDate":"2024-03-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"AtRec: Accelerating Recommendation Model Training on CPUs\",\"authors\":\"Siqi Wang;Tianyu Feng;Hailong Yang;Xin You;Bangduo Chen;Tongxuan Liu;Zhongzhi Luan;Depei Qian\",\"doi\":\"10.1109/TPDS.2024.3381186\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"The popularity of recommendation models and the enhanced AI processing capability of CPUs have provided massive performance opportunities to deliver satisfactory experiences to a large number of users. Unfortunately, existing recommendation model training methods fail to achieve high efficiency due to unique challenges such as dynamic shape and high parallelism. To address the above limitations, we comprehensively study the distinctive characteristics of recommendation models and discover several unexploited optimization opportunities. To exploit such opportunities, we propose \\n<italic>AtRec</i>\\n, a high-performant recommendation model training engine that significantly accelerates the training process on CPUs. Specifically, \\n<italic>AtRec</i>\\n presents comprehensive approach of training that employs operator-level and graph-level joint optimizations and runtime optimization. At the operator-level, \\n<italic>AtRec</i>\\n identifies and optimizes the time-consuming operators, which enables further efficient graph-level optimizations. At the graph-level, \\n<italic>AtRec</i>\\n conducts an in-depth analysis of the inefficiencies in several frequently used subgraphs, enables further performance improvement via eliminating redundant computations and memory accesses. In addition, to achieve better runtime performance, \\n<italic>AtRec</i>\\n also identifies inefficiencies prevalent in the current scheduling and proposes runtime batching. The experiment results demonstrate that \\n<italic>AtRec</i>\\n can significantly outperform state-of-the-art recommendation model training engines. We have open sourced the implementation and corresponding data of \\n<italic>AtRec</i>\\n to boost research in this direction.\",\"PeriodicalId\":13257,\"journal\":{\"name\":\"IEEE Transactions on Parallel and Distributed Systems\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":5.6000,\"publicationDate\":\"2024-03-25\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"IEEE Transactions on Parallel and Distributed Systems\",\"FirstCategoryId\":\"94\",\"ListUrlMain\":\"https://ieeexplore.ieee.org/document/10478579/\",\"RegionNum\":2,\"RegionCategory\":\"计算机科学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"COMPUTER SCIENCE, THEORY & METHODS\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"IEEE Transactions on Parallel and Distributed Systems","FirstCategoryId":"94","ListUrlMain":"https://ieeexplore.ieee.org/document/10478579/","RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, THEORY & METHODS","Score":null,"Total":0}
引用次数: 0
摘要
推荐模型的普及和 CPU 人工智能处理能力的增强为向大量用户提供满意的体验提供了巨大的性能机遇。遗憾的是,由于动态形状和高并行性等独特挑战,现有的推荐模型训练方法无法实现高效率。为了解决上述局限性,我们全面研究了推荐模型的显著特征,并发现了几个尚未开发的优化机会。为了利用这些机会,我们提出了 AtRec,这是一个高性能的推荐模型训练引擎,能显著加速 CPU 上的训练过程。具体来说,AtRec 提出了全面的训练方法,其中包括操作员级和图级联合优化以及运行时优化。在算子级,AtRec 会识别并优化耗时的算子,从而进一步实现高效的图级优化。在图层面,AtRec 深入分析了几个常用子图中的低效问题,通过消除冗余计算和内存访问进一步提高了性能。此外,为了获得更好的运行时性能,AtRec 还识别了当前调度中普遍存在的低效问题,并提出了运行时批处理建议。实验结果表明,AtRec 的性能明显优于最先进的推荐模型训练引擎。我们已将 AtRec 的实现和相应数据开源,以促进该方向的研究。
AtRec: Accelerating Recommendation Model Training on CPUs
The popularity of recommendation models and the enhanced AI processing capability of CPUs have provided massive performance opportunities to deliver satisfactory experiences to a large number of users. Unfortunately, existing recommendation model training methods fail to achieve high efficiency due to unique challenges such as dynamic shape and high parallelism. To address the above limitations, we comprehensively study the distinctive characteristics of recommendation models and discover several unexploited optimization opportunities. To exploit such opportunities, we propose
AtRec
, a high-performant recommendation model training engine that significantly accelerates the training process on CPUs. Specifically,
AtRec
presents comprehensive approach of training that employs operator-level and graph-level joint optimizations and runtime optimization. At the operator-level,
AtRec
identifies and optimizes the time-consuming operators, which enables further efficient graph-level optimizations. At the graph-level,
AtRec
conducts an in-depth analysis of the inefficiencies in several frequently used subgraphs, enables further performance improvement via eliminating redundant computations and memory accesses. In addition, to achieve better runtime performance,
AtRec
also identifies inefficiencies prevalent in the current scheduling and proposes runtime batching. The experiment results demonstrate that
AtRec
can significantly outperform state-of-the-art recommendation model training engines. We have open sourced the implementation and corresponding data of
AtRec
to boost research in this direction.
期刊介绍:
IEEE Transactions on Parallel and Distributed Systems (TPDS) is published monthly. It publishes a range of papers, comments on previously published papers, and survey articles that deal with the parallel and distributed systems research areas of current importance to our readers. Particular areas of interest include, but are not limited to:
a) Parallel and distributed algorithms, focusing on topics such as: models of computation; numerical, combinatorial, and data-intensive parallel algorithms, scalability of algorithms and data structures for parallel and distributed systems, communication and synchronization protocols, network algorithms, scheduling, and load balancing.
b) Applications of parallel and distributed computing, including computational and data-enabled science and engineering, big data applications, parallel crowd sourcing, large-scale social network analysis, management of big data, cloud and grid computing, scientific and biomedical applications, mobile computing, and cyber-physical systems.
c) Parallel and distributed architectures, including architectures for instruction-level and thread-level parallelism; design, analysis, implementation, fault resilience and performance measurements of multiple-processor systems; multicore processors, heterogeneous many-core systems; petascale and exascale systems designs; novel big data architectures; special purpose architectures, including graphics processors, signal processors, network processors, media accelerators, and other special purpose processors and accelerators; impact of technology on architecture; network and interconnect architectures; parallel I/O and storage systems; architecture of the memory hierarchy; power-efficient and green computing architectures; dependable architectures; and performance modeling and evaluation.
d) Parallel and distributed software, including parallel and multicore programming languages and compilers, runtime systems, operating systems, Internet computing and web services, resource management including green computing, middleware for grids, clouds, and data centers, libraries, performance modeling and evaluation, parallel programming paradigms, and programming environments and tools.