An Online Two-Stage Classification Based on Projections

Abstract

Kernel-based online classification algorithms, such as the Perceptron, NORMA, and passive-aggressive, are renowned for their computational efficiency but have been criticized for slow convergence. However, the parallel projection algorithm, within the adaptive projected subgradient method framework, exhibits accelerated convergence and enhanced noise resilience. Despite these advantages, a specific sparsification procedure for the parallel projection algorithm is currently absent. Additionally, existing online classification algorithms, including those mentioned earlier, heavily rely on the kernel width parameter, rendering them sensitive to its choices. In an effort to bolster the performance of these algorithms, we propose a two-stage classification algorithm within the Cartesian product space of reproducing kernel Hilbert spaces. In the initial stage, we introduce an online double-kernel classifier with parallel projection. This design aims not only to improve convergence but also to address the sensitivity to kernel width. In the subsequent stage, the component with a larger kernel width remains fixed, while the component with a smaller kernel width undergoes updates. To promote sparsity and mitigate model complexity, we incorporate the projection-along-subspace technique. Moreover, for enhanced computational efficiency, we integrate the set-membership technique into the updates, selectively exploiting informative vectors to improve the classifier. The monotone approximation of the proposed classifier, based on the designed \( \epsilon \)-insensitive function, is presented. Finally, we apply the proposed algorithm to equalize a nonlinear channel. Simulation results demonstrate that the proposed classifier achieves faster convergence and lower misclassification error with comparable model complexity.