Incremental Learning on Decorrelated Approximators

Abstract

In general, designing an incremental learning system for a particular task at least consists of choosing an appropriate approximation structure and learning algorithm. Common Linear In the Parameters (LIP) approximation structures are for example polynomials, radial basis functions or grid-based lookup tables. Typical learning algorithms accompanying them are for example Passive-Aggressive (PA) or Recursive Least Squares (RLS). Usually, these two choices are not independent as not every learning algorithm is able to handle any approximation structure well. Here we present a formalism that allows the designer to treat these two design aspects independently from each other. By decorrelating the basis functions of the approximator we form a new set of basis functions that can be handled by any learning algorithm. We develop design guidelines in order to make our approach an easy to use tool and to support the designer in making the learning progress reliable at design time. Further, we look at the properties of our approach as an extension to LIP approximators and investigate its implications for the behavior of the incremental learning system using artificial, benchmark and real world data sets for regression tasks.