Natural Language Processing and Classification Methods for the Maintenance and Optimization of US Weapon Systems

Abstract

The Logistics Management Institute (LMI) works with the US Department of Defense (DoD) in analyzing maintenance logs on US weapons systems. A major issue in processing this data is determining how to extract useful information from disorganized short-form texts in order to optimize the maintenance of these systems. Unlike text from other corpora, these text entries are only a few words in length and do not conform to lexical convention. LMI has provided a subset of about 10 million of these maintenance logs, each labeled with action-object pairs. The goals of this research are to construct a model that predicts action-object pairs and provide a metric to assess its validity. Prior to analysis, the entries are vectorized by either TFIDF and TSVD, or Word2vec. Several models are applied, including logistic regression, k-NN, SVM, decision trees, LSA, and DBSCAN clustering. Unsupervised models are tested in addition to supervised models due to the ambiguity regarding the validity of the provided ground truth values. The results of these tests yield accuracy scores of about 0.53 for action words and 0.73 for object words. Furthermore, the results from clustering provides evidence for discrepancies in the ground truth values. Taking this into consideration, prior models are adjusted and accuracy scores increased to 0.78 for action words.