Comparison of two mixed methods approaches for multimodal quality evaluations: Open Profiling of Quality and Conventional Profiling

To guide the practitioner's work in choosing between assessment methods, defined criteria to compare their benefits, costs, and limitations are needed. The goal of this paper is two-fold. Firstly, we develop an extensive comparison model to guide between-method comparisons based on a literature review. The model is composed of four main criteria, called economy, excellence, implementation and assessment with a total of 24 sub-criteria. Secondly, we conduct a comparison study between two mixed methods in which a subset of criteria of the model is examined. We compare Open Profiling of Quality utilizing individuals' own vocabulary and Conventional Profiling utilizing fixed vocabulary in their descriptive evaluation. The study is conducted with naïve participants with varying 3D video qualities on a mobile device. The results compare both methods on a subset of comparison criteria and show that operationalization of the developed comparison model can provide a tool for holistic methods comparison.