FullPack: Full Vector Utilization for Sub-Byte Quantized Matrix-Vector Multiplication on General Purpose CPUs

IF 1.4 3区计算机科学 Q4 COMPUTER SCIENCE, HARDWARE & ARCHITECTURE IEEE Computer Architecture Letters Pub Date : 2024-02-27 DOI:10.1109/LCA.2024.3370402

Hossein Katebi;Navidreza Asadi;Maziar Goudarzi

引用次数: 0

Abstract

Sub-byte quantization on popular vector ISAs suffers from heavy waste of vector as well as memory bandwidth. The latest methods pack a number of quantized data in one vector, but have to pad them with empty bits to avoid overflow to neighbours. We remove even these empty bits and provide full utilization of the vector and memory bandwidth by our data-layout/compute co-design scheme. We implemented FullPack on TFLite for Vector-Matrix multiplication and showed up to

$6.7\times$

speedup,

$2.75\times$

on average on single layers, which translated to

$1.56-2.11\times$

end-to-end speedup on DeepSpeech.

查看原文

微信好友朋友圈 QQ好友复制链接

本刊更多论文

FullPack：通用 CPU 上子字节量化矢量矩阵乘法的全矢量利用率

流行的矢量 ISA 上的子字节量化严重浪费了矢量和内存带宽。最新的方法将大量量化数据打包到一个矢量中，但必须填充空位以避免溢出到邻域。我们通过数据布局/计算协同设计方案，去掉了这些空位，从而充分利用了矢量和内存带宽。我们在 TFLite 上实现了 FullPack 的矢量-矩阵乘法，结果显示速度提高了 6.7 倍，单层平均提高了 2.75 倍，在 DeepSpeech 上的端到端速度提高了 1.56-2.11 倍。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文去求助

来源期刊

IEEE Computer Architecture Letters COMPUTER SCIENCE, HARDWARE & ARCHITECTURE-

CiteScore

4.60

自引率

4.30%

发文量

期刊介绍： IEEE Computer Architecture Letters is a rigorously peer-reviewed forum for publishing early, high-impact results in the areas of uni- and multiprocessor computer systems, computer architecture, microarchitecture, workload characterization, performance evaluation and simulation techniques, and power-aware computing. Submissions are welcomed on any topic in computer architecture, especially but not limited to: microprocessor and multiprocessor systems, microarchitecture and ILP processors, workload characterization, performance evaluation and simulation techniques, compiler-hardware and operating system-hardware interactions, interconnect architectures, memory and cache systems, power and thermal issues at the architecture level, I/O architectures and techniques, independent validation of previously published results, analysis of unsuccessful techniques, domain-specific processor architectures (e.g., embedded, graphics, network, etc.), real-time and high-availability architectures, reconfigurable systems.