STOCHASTIC ALTERNATING DIRECTION METHOD OF MULTIPLIERS FOR STRUCTURED REGULARIZATION

In this paper, we present stochastic optimization variants of the alternating direction method of multipliers (ADMM). ADMM is a useful method to solve a regularized risk minimization problem where the regularization term is complicated and not easily dealt with in an ordinary manner. For example, structured regularization is one of the typical applications of such regularization in which ADMM is effective. It includes group lasso regularization, low rank tensor regularization, and fused lasso regularization. Since ADMM is a general method and has wide applications, it is intensively studied and refined these days. However, ADMM is not suited to optimization problems with huge data. To resolve this problem, online stochastic optimization variants and a batch stochastic optimization variant of ADMM are presented. All the presented methods can be easily implemented and have wide applications. Moreover, the theoretical guarantees of the methods are given.