Mixed Precision Quantization with Hardware-Friendly Activation Functions for Hybrid ViT Models

Abstract

As hardware devices have advanced recently, various artificial intelligence tasks including convolutional neural networks (CNNs) have achieved high accuracy. Especially in computer vision tasks, vision transformer (ViT) based models have achieved unprecedented progress, and CNN + ViT hybrid models have also been proposed that take advantage of both CNNs and ViTs. However, the numerous parameters of hybrid ViTs are unsuitable for resource-constrained mobile/edge environments. In addition, the nonlinear activation functions in hybrid ViTs (e.g., GeLU and Swish) require more resources and computational cost compared to integer operation functions (e.g., ReLU) when using dedicated hardware accelerators. To address these issues, we propose a technique to efficiently compress the prominent hybrid ViT model, MobileViT, by applying the mixed precision quantization and the Shift-Swish activation function. Compressing the MobileViT-s, MobileViT-xs, and MobileViT-xxs models with the proposed method on the ImageNet dataset resulted in minimal accuracy drops of 0.41%, 0.18%, and 0.86%, respectively, while achieving effective quantization and activation function approximation at the average 7.9-bit level.