Big Brother is Watching You—But in a Good Way

Abstract

In any modern desktop environment the glyph compositor-where raw text information is combined with font information and other attributes to render rasterized component images-is part of the software's core functionality. In this paper we present work that shows it is computationally feasible to apply full-text indexing in real-time to the live stream of glyph compositor operations generated by a user's interaction with their desktop environment. By embedding indexing functionality at such a level, we effectively get to “see” (and more importantly remember) all the text that is drawn on the user's screen. With elements reminiscent of the Memex, we illustrate the technique in use through a personal digital library we have developed that enriches (through text-searching and context) the user's desktop experience by letting them go back in time to view information that had previously been displayed. We achieved this by augmenting our dynamically updated text index with time-stamped snapshots of the desktop. By recording the (x, y) positions of the text at the time it is rendered, the snapshots have a semi-live feel, whereby text can be selected for copy-and-paste operations for further use. Moreover, windows-even if they were hidden behind others at the time the text was rendered-can be brought to the front and their text accessed.