Efficient Mini-batch Training for Echo State Networks

Abstract

Echo state networks (ESNs) are generally optimized by the ordinary recursive least squares (ORLS) algorithm. Although ORLS has fast convergence, it can process only one sample per iteration, which makes ESNs difficult to scale to large datasets. To tackle this problem, a novel mini-batch RLS (MRLS) algorithm is proposed in this paper. On this basis, to prevent overfitting in the ESN training, an L1regularization method is suggested for MRLS. In addition, to make ESNs more suitable for time-varying tasks, an adaptive method of the forgetting factor is also introduced for MRLS. Experimental results of two time-series problems show that ESNs have faster processing speed and better convergence performance with MRLS than with ORLS. CCS CONCEPTS • Computing methodologies → Machine learning; Machine learning approaches; Neural networks; Machine learning; Learning settings; Batch learning.