The Least Common Bigram: A dictionary arrangement technique for computerized natural-language text searching

Abstract

The LCB (Least Common Bigram) method is a dictionary arrangement technique for computerized natural-language searching, based upon certain statistical characteristics of a data base. Once the distribution of elements (such as bigrams) in the data base is known, the LCB method allows each dictionary entry to be sorted on the element least likely to be found in the data base. Thus the LCB method minimizes the number of dictionary entries examined during text searching. Statistical analysis of a dictionary and data base used in literature searching by IIT Research Institute's Computer Search Center shows the LCB method yields a 400% improvement over the traditional method of dictionary lookup. Applications in machine-assisted translation, indexing, linguistic analysis, and other information storage and retrieval areas are foreseen.