Design of a differentiable L-1 norm for pattern recognition and machine learning

Abstract

In various applications of pattern recognition, feature selection, and machine learning, L-1 norm is used as either an objective function or a regularizer. Mathematically, L-1 norm has unique characteristics that make it attractive in machine learning, feature selection, optimization, and regression. Computationally, however, L-1 norm presents a hurdle as it is non-differentiable, making the process of finding a solution difficult. Existing approach therefore relies on numerical approaches. In this work we designed an L-1 norm that is differentiable and, thus, has an analytical solution. The differentiable L-1 norm removes the absolute sign in the conventional definition and is everywhere differentiable. The new L-1 norm is almost everywhere linear, a desirable feature that is also present in the conventional L-1 norm. The only limitation of the new L-1 norm is that near zero, its behavior is not linear, hence we consider the new L-1 norm quasi-linear. Being differentiable, the new L-1 norm and its quasi-linear variation make them amenable to analytic solutions. Hence, it can facilitate the development and implementation of many algorithms involving L-1 norm. Our tests validate the capability of the new L-1 norm in various applications.