A DNN Inference Latency-aware GPU Power Management Scheme

Graphics Processing Units (GPUs) are widely used for deep learning training as well as inference due to their high processing speed and programmability. Modern GPUs dynamically adjust the clock frequency according to their power management scheme. However, under the default scheme, the clock frequency of a GPU is only determined by utilization rate while being blind to target latency SLO, leading to unnecessary high clock frequency which causes excessive power consumption. In this paper, we propose a method to increase the energy efficiency of a GPU while satisfying latency SLO through performance scaling. It dynamically monitors the queue length of the inference engine to determine the optimal clock that can satisfy latency SLO. We implemented an efficient inference service using GPU DVFS on the existing inference engine. According to the result of experiments on inference over image classification models using three types of GPUs, all the 99th percentile latency in our method satisfied latency SLO while exhibiting better power efficiency. In particular, when processing the VGG19 model on Titan RTX, the energy consumption of the GPU is reduced by up to 49.5% compared to the default clock management when processing the same request rates.