Empirical Study on Assessment Algorithms with Confidence in Crowdsourcing

Abstract

Evaluating the quality of workers is very important in crowdsourcing system and impactful methods are required in order to obtain the most appropriate quality. Previous work have introduced confidence intervals to estimate the quality of workers. However, we have found the size of the confidence interval is wide through analysis of experimental results, which leads to inaccurate worker error rates. In this paper, we propose an optimized algorithm of confidence interval to reduce the size of the confidence interval as narrow as possible and to estimate the quality of workers more precise. We verify our algorithm using the simulated data from our own crowdsourcing platform under realistic settings.