How shall we evaluate prototype natural language processors

Abstract

Recent years have seen important advances in computational linguistics and artificial intelligence. Although many problems remain, the goal of providing limited English-processing facilities for non-technical computer users is within sight. By the end of the decade, numerous systems providing limited coverage of "natural language" will be available for business and home use. Several systems (e.g. TQA [16]) have already become operational. One system (ROBOT [7]) has been supporting natural language inputs in a dozen or so different commercial database applications for at least three years. Many other systems have been developed to the prototype stage and will soon be able to be transferred, with varying degrees of effort, from a research to a production environment. Each system tends to provide special features of its own, and the future prospects for database, office, instructional, and other environments are quite exciting.