Exploiting multimodal integration in adaptive interactive systems and game-based learning interfaces

Abstract

The main purpose of my work is to investigate multisensory and multimodal integration in the design and development of adaptive systems and interfaces for game-based learning applications in the areas of education and rehabilitation. To this aim, I contributed to the creation of a multimodal dataset of violin performances, integrating motion capture, video, audio, and on-body sensors (accelerometers and EMG), and I worked closely with psychophysicists and educators on the design of paradigms and technologies for multisensory and embodied learning of mathematics in primary school children. Main theoretical foundations of my research are multisensory processing and integration, psychophysics analysis, embodied cognition theories, computational models of non-verbal and emotion communication in full-body movement, and human-computer interaction models for adaptive interfaces and serious-games.