Beyond the vision paradigm: design strategies for crossmodal interaction with dynamic digital displays

The purpose of our research is to develop new interaction paradigms for dynamic digital displays (DDDs) in public space. Our lab is currently developing an ontological framework that comprises seven different interaction paradigms for DDDs. Still in its budding stages, this framework is intended to assist HCI practitioners in the conception and evaluation of architectural scale DDD installations. This paper theoretically discusses crossmodal interaction as one of these seven interaction paradigms. We used an architectural approach that draws on medium specificity --- a fine arts concept foreign to HCI --- to conduct a phenomenological analysis of DDDs that have been deployed in an actual public space in Montréal, Canada. After interviewing DDD design artists and performing our preliminary observational analyses, we found four design strategies that were used to produce crossmodal interaction on an architectural scale, helping shift the experience of DDDs beyond the vision paradigm.