Subtle consumer-photo quality evaluation

Subjective test methodologies are morphing to enable researchers to answer questions relevant to rapidly evolving technologies in an efficient and reliable manner. This paper is an exploration of how subjective testing that employs crowdsourcing can be refined to drive stability and reliability in subjective results. We investigate how various design decisions can lead to disparate subjective responses; motivated by the need for efficient acquisition of large volumes of data, and the need to understand and mitigate pitfalls in online tests, when the stimuli are complex and subtle as is the case in popular consumer scenarios.