The Use of Discriminant Analysis to Select Content-Bearing Words

Abstract

This article presents a method for identifying good indexing terms from frequently occurring stems. The method uses discriminant analysis to distinguish terms that refer to topics from general terms that do not refer to topics. The steps in the method are the selection of discriminating variables, the calibration of predefined groups and the derivation of discriminant functions from them, and the classification of a second, unknown set of terms and its evaluation. The method is tested by applying it to the Harris Survey Question database, which covers 121 different surveys and includes the text of over 12, 000 Individual questions. The evaluation demonstrates the success of the method.