Sequential Covariance-Matrix Estimation with Application to Mitigating Catastrophic Forgetting

Abstract

Catastrophic forgetting is a problem encountered with neural networks as well as other learning systems whereby past representations are lost as new representations are learned. It has been shown that catastrophic forgetting can be mitigated in neural networks by using a neuron selection technique, dubbed "cluster-select," which performs online clustering over the network inputs to partition the network such that only a subset of neurons are used for a given input vector. Cluster-select can benefit by using Mahalanobis distance which relies on an inverse covariance estimate. Unfortunately, covariance estimation is problematic when lacking a very large number of samples relative to the number of input dimensions. One way to tackle this problem is through the use of a shrinkage estimator that offers a compromise between the sample covariance matrix and a well-conditioned matrix with the aim of minimizing the mean-squared error (MSE). In online environments, such as those in which catastrophic forgetting can occur, data arrives sequentially, requiring the covariance matrix to be estimated sequentially. Therefore, in this work we derive sequential update rules for the shrinkage estimator and approximate it's related inverse. The online covariance estimator is applied to the cluster-select technique with results that demonstrate further improvements in terms of effectively mitigating catastrophic forgetting.