Histogram-Based Dimensionality Reduction of Term Vector Space

Abstract

One of the most vital problems of free-text document processing is the curse of dimensionality. The paper presents a dimensionality reduction algorithm based on informed feature selection. Terms describing the document are based on histogram-like statistics which can be computed as well as incrementally updated at low complexity. The document representation can adapt to changing document collection characteristics. Along with the fundamental concepts we present an empirical verification of the approach.