Is a picture worth a thousand words A computational investigation of the modality effect

Abstract

The modality effect is a term that refers to differences in learning performance in relation to the mode of presentation. It is an interesting phenomenon that impacts education, online-learning, and marketing among many other areas of life. In this study, we use electroencephalography (EEG alpha, beta, and theta) and computational modelling of comprehension to study the modality effect in text and multimedia. First, we provide a framework for evaluating learning performance, working-memory, and emotions during learning. Second, we apply these tools to investigate the modality effect computationally focusing on text in contrast to multimedia. This study is based on a dataset that we have collected through a human experiment involving 16 participants. Our results are important for future learning systems that incorporate learning performance, working-memory, and emotions in a continuous feedback system that measures and optimises learning during learning and not after.