Optimization of Deep Learning Inference on Edge Devices

Abstract

Concerning Artificial Intelligence (AI)-based applications, it is necessary to reduce latency in real-time inference. This paper implements and compares two separate models, Inception V3 and Mobilenet, using Intel Neural Compute Stick (NCS) 2 and Raspberry Pi 4 as the edge devices. The Model Optimizer (MO), which generates an Intermediate Rep- resentation (IR) of the network, is used for optimizing these models. Then, the IR models are inferences in the edge device. Finally, the comparison of frame per second speed (FPS) and precision is provided. The results show that the speed on Inception V3 is 9 frames per second, while that on Mobilenet is 24 frames per second. Simultaneously, the accuracy reaches 41.28% on Inception V3, but misclassifies for Nissan Altima 2014, and reaches 71.29% on Mobilenet with right classification for Toyota Camry 2014.