A Machine Learning Algorithm in Automated Text Categorization of Legacy Archives

Abstract

The goal of this research is to develop an algorithm to automatically retrieve critical information from raw data files in NASA’s airborne measurement data archive. The product has to meet specific metrics in term of accuracy, robustness and usability, as the initial decision-tree based development has shown limited applicability due to its resource intensive characteristics. We have developed an innovative solution that is much less resource intensive while offering comparable performance. As with many practical applications, the data available are noisy and correlated; and there is a wide range of features that are associated with the information to be retrieved. The proposed algorithm uses a decision tree to select features and determine their weights. A weighted Naive Bayes is used due to the presence of highly correlated inputs. The development has been successfully deployed in an industrial scale, and the results show that the development is well-balanced in term of performance and resource requirements.