Deploying Pre-Quantized Deep Learning Models on Heterogeneous Platforms with Operator Flow Recognition and Quantization Parameter Optimization

Abstract

Quantized deep learning models are suitable for the embedded devices with limited computation resource. For computation-intensive neural network operators such as convolution, heterogeneous platforms with a set of processing units of different types become common in the embedded devices. These embedded devices usually operate on fixed-point calculations; moreover, they rely on customized kernel functions to deploy deep learning models. In this paper, a flow of deploying pre-quantized deep learning models on heterogeneous platforms using TVM is presented. We propose an optimization to convert quantization parameters. To leverage customized kernel functions, we propose the operator flow recognition. To demonstrate our flow, we utilize embARC Machine Learning Inference (embARC MLI), an open-source software library targeted for low-power applications. A set of pre-quantized deep learning models are deployed on a heterogeneous platform comprising x86 and embARC MLI. Experimental results show that for each model, the accuracy obtained from the heterogeneous platform is much the same as the one obtained from an x86 platform.