Contextual Entropy and Text Categorization

Abstract

In this paper we describe a new approach to text categorization, our focus is in the amount of information (the entropy) in the text. The entropy is computed with the empirical distribution of words in the text. We provide the system with a manually segmented collection of documents in different categories. For each category a separate empirical distribution of words is computed, we use this empirical distribution for categorization purposes. If we compute the entropy of the test document for each empirical distribution the correct category shows as a maximum. For example, if we compute the entropy of a sports document using the politics or the sports empirical word distributions then the computed entropy is higher in sports than in politics. Our text categorization approach is simple, easy to code and needs no training time (aside from histogram computations). The classification time is linear on the size of the document and the number of document categories. We support our claims with extensive experimentation