Coupling data understanding with software reuse

Abstract

Reuse of information requires an ability to understand data gathered from the web and to integrate that data with knowledge and reusable programs. We describe systems that allow a user to capture and understand data from the web and rapidly and easily write programs to analyze the data and combine it with other data. A data grokker parses data, inferring the data types of its fields both from field names and from values of the data itself; this produces both a local set of usable data and a set of data type descriptions that link the data to known types. The known types have knowledge and reusable procedures that can be inherited and used with the data. Web pages that perform calculations or data lookup can be treated as remote procedure calls, allowing calculations, proprietary data and real-time data to be used. We have developed a graphical programming system that can specialize reusable programs for use with data from the web, allowing rapid and easy construction of programs for custom analysis of web data. These systems are illustrated with examples.