Scaling Computation on GPUs Using Powerlists

With the explosion of big data analytics, scaling linear algebra packages has become extremely important. Inthe context of GPUs, cuBLAS API provides a highly efficientpackage for linear algebra subroutines on a single GPU. Dueto inputs of large dimensions, it often becomes necessary tocompute over clusters. However, the package does not provide facilities for computing over a 'cluster of GPUs' efficiently. Inthis paper, we demonstrate a high level framework for scaling linear algebra computations across a cluster of GPUs, through matrix multiplication problem. In particular, we describe amethod of specifying matrices using powerlists that captures both parallelism and recursion succinctly, and automatically schedule partitioned matrices over a GPU cluster to gain the advantages of cuBLAS for computing the product of partitioned matrices over a cluster of GPUs. Our experimental results show significant performance gains, of the order ofat least 132% for large matrices over that of a single GPUcomputation. The method reflects the map-reduce paradigmwhere the matrices are mapped to appropriate partitioned matrices and sent to appropriate members of the clusters andthe results are collected to obtain the resultant matrix.