Blind Recognition of Text Input on Mobile Devices via Natural Language Processing

Abstract

In this paper, we investigate how to retrieve meaningful English text input on mobile devices from recorded videos while the text is illegible in the videos. In our previous work, we were able to retrieve random passwords with high success rate at a certain distance. When the distance increases, the success rate of recovering passwords decreases. However, if the input is meaningful text such as email messages, we can further increase the success rate via natural language processing techniques since the text follows spelling and grammar rules and is context sensitive. The process of retrieving the text from videos can be modeled as noisy channels. We first derive candidate words for each word of the input sentence, model the whole sentence with a Hidden Markov model and then apply the trigram language model to derive the original sentence. Our experiments validate our technique of retrieving meaningful English text input on mobile devices from recorded videos.