How Much Cache is Enough? A Cache Behavior Analysis for Machine Learning GPU Architectures

Abstract

Graphic Processing Units (GPUs) are highly parallel, power hungry devices with large numbers of transistors devoted to the cache hierarchy. Machine learning is a target application field of these devices, which take advantage of their high levels of parallelism to hide long latency memory access dependencies. Even though parallelism is the main source of performance in these devices, a large number of transistors is still devoted to the cache memory hierarchy. Upon detailed analysis, we measure the real impact of the cache hierarchy on the overall performance. Targeting Machine Learning applications, we observed that most of the successful cache accesses happen in a very reduced number of blocks.With this in mind, we propose a different cache configuration for the GPU, resulting in 25% of the leakage power consumption and 10% of the dynamic energy per access of the original cache configuration, with minimal impact on the overall performance.