A Technique for Ranking and Visualization of Crowd-Powered Subjective Evaluations

Abstract

We previously presented a crowd-powered digital contents evaluation system. This system shows a lot of pictures to the answerers and ask them to input the evaluations. It preferentially selects pictures which are predicted to be highly or poorly evaluated to the answerers, based on our assumption that high or poor evaluations are more informative results comparing with moderate evaluations. We have applied an interactive genetic algorithm in our system to select such pictures. This paper presents a technique for ranking and visualization for the evaluation results collected by our system. The presented technique calculates scores of all contents and uses for the ranking. Here, it may happen that some pictures are shown to no answerers while using our evaluation system. Our technique presented in this paper estimates the evaluation of such pictures shown to no answerers, and finally complete the ranking of all the pictures. The paper also presents the visualization tool for the ranking of pictures, and our experiment to demonstrate the effectiveness of our technique.