Task-based subject validation: Reliability metrics

Abstract

The quality obtained from a CCTV or a telemedicine system has to be good enough to correctly and with high probability recognize a face or a medical pathology. In order to measure “good enough” term subjective experiments are used. One of the problems related to subjective experiment data analysis is the subjects' reliability or, more precisely, each individual subject's reliability. In this paper two metrics which can be used to detect unreliable subjects in case of a task-based subjective experiment are presented. A task-based subjective experiment is an experiment in which a subject performs a task (recognizes a person, characters, pathology, etc.). In this paper metrics enabling measurement of a subject's reliability are referred to as reliability metrics. Reliability metrics should be considered as a part of a standard of a task based subjective experiment like ITU P.912 [1].