Improving compressed matrix multiplication using control variate method

Abstract

The seminal work by Pagh [1] proposed a matrix multiplication algorithm for real-valued squared matrices called Compressed Matrix Multiplication (CMM) having a sparse matrix output product. The algorithm is based on a popular sketching technique called Count-Sketch [2] and Fast Fourier Transform (FFT). For input square matrices A and B of order n and the product matrix AB with Frobenius norm $\left|\right|\mathrm{AB}|{|}_F$, the algorithm offers an unbiased estimate for each entry, i.e., ${\left(\mathrm{AB}\right)}_{i,j}$ of the product matrix AB with a variance bounded by $\left|\right|\mathrm{AB}|{|}_F^2/b$, where b is the compressed bucket size. Thus, the variance will eventually become high for a small bucket size. In this work, we address the high variance problem of CMM with the help of a simple and practical technique based on classical variance reduction methods in statistics. Our techniques rely on the Control Variate (CV) method. We suggest rigorous theoretical analysis for variance reduction and complement it via supporting empirical evidence.