Ongoing Challenges of Evaluating Mulsemedia QoE

Multimedia applications are usually limited to stimulating only two human senses: vision and hearing. Recent studies seek to expand the definition of multimedia applications to include stimuli for other human senses. In this way, sensory effects that should be triggered in synchrony with the audiovisual content being presented are included in the applications. By including sensory effects in multimedia, we aim to improve the Quality of Experience (QoE) with these mulsemedia environments. Usually, two approaches are being used for performing QoE evaluations these environments. The first, more common, is performed by subjective evaluation approaches, i.e. through questionnaires, interrogations, oral responses, etc. The second, rarer but growing, uses objective approaches by collecting physiological data from the user when dealing with the system being evaluated. Such data is gathered in real time or not, however, it is considered objective because it is "involuntary", that is, data is not the result of the user’s intention. This paper will address both the these methods to evaluate QoE and what the respective obstacles are when dealing with in mulsemedia systems.