Towards a hierarchy of visual languages

Formalisms for visual language specification have been investigated for more than two decades. However, there has been little attempt to develop a systematic and comprehensive hierarchy of visual languages based on formal properties. Given the importance of the Chomsky hierarchy in the theory of textual languages and the difficulty of comparing the different visual language formalisms, it is clear that there is a need for such a hierarchy. We develop a hierarchy for VLs and investigate the expressiveness and the cost of parsing for classes defined therein. Although the hierarchy is based on the constraint multiset grammar formalism, we sketch how other visual language specification formalisms can be mapped into constraint multiset grammars. One consequence of our work is that a large class of "naturally occurring" visual languages are inherently context-sensitive, so that the core of such a hierarchy has to be built around different forms of context-sensitivity.