BlinkNet: Software-Defined Deep Learning Analytics with Bounded Resources

Abstract

Deep neural networks (DNNs) have recently gained unprecedented success in various domains. In resource-constrained edge systems (e.g., mobile devices and IoT devices) QoS-aware DNNs are required to meet latency and memory/storage requirements of mission-critical deep learning applications. However, none of the existing DNNs has been de-signed to satisfy both latency and memory bounds simultaneously as specified by end-users in the resource-constrained systems. This paper proposes a runtime system, BlinkNet, which can guarantee both latency and memory/storage bounds for one or multiple DNNs via efficient QoS-aware per-layer approximation. We implement BlinkNet in Apache TVM and evaluate it using CaffeNet, CIFAR-10-quick, and VGG16 network models on both CPU and GPU platforms. Our experimental results show that BlinkNet can enforce various latency and memory bounds set by end-users with real-world datasets.