Characterizing Resource Heterogeneity in Edge Devices for Deep Learning Inferences

Significant advances in hardware capabilities and the availability of enormous data sets have led to the rise and penetration of artificial intelligence (AI) and deep learning (DL) in various domains. Considerable efforts have been put forth in academia and industry to make these computationally demanding DL tasks work on resource-constrained edge devices. However, performing DL tasks on edge devices is still challenging due to the diversity of DNN (Deep Neural Networks) architectures and heterogeneity of edge devices. This study evaluates and characterizes the performance and resource heterogeneity in various edge devices for performing DL tasks. We benchmark various DNN models for image classification on a set of edge devices ranging from the widely popular and relatively less powerful Raspberry Pi to GPU-equipped high-performance edge devices like Jetson Xavier NX. We also compare and contrast the performance of three widely-used DL frameworks when used in these edge devices. We report DL inference throughput, CPU and memory usage, power consumption, and frameworks' initialization overhead, which are the most critical factors for characterizing DL tasks on edge devices. Additionally, we provide our insights and findings, which will provide a better idea of how compatible or feasible edge devices are for running DL applications.