A visual language for interactive data exploration and analysis

The analysis of large amounts of data to extract generalizations, exceptions, trends, and hidden relationships is a common activity in the business and scientific communities. While some kinds of "knowledge" can be extracted automatically with preselected algorithms or data mining techniques, others require an experienced human, often an expert in analysis, the business or scientific context, or both. We have found that such humans combine exploration, the search for a relevant subset or view of the data, with analysis, statistical or other techniques for measurement. We designed and implemented a visual language, IDEA, to assist the data analyst in these two intertwined tasks. The language is a convenient representation for data analysis and provides environmental support for keeping track of sequences of operations, reuse of the data analysis itself, and enforced semantics between operations and data.