Robust annotation aggregation in crowdsourcing via enhanced worker ability modeling

Abstract

Truth inference in crowdsourcing, which studies how to aggregate noisy and biased annotations from workers with varied expertise, is a fundamental technology powering the quality of crowdsourced annotations. Generally, confusion-matrix-based methods are more promising and worker better, as they model each worker’s ability using a confusion matrix rather than a single real value. However, the imbalanced classes and the insufficient training data caused by the $K\times K$ pattern ($K$ refers to the number of classes) are still two major issues for the learning of confusion matrices, which call for a robust modeling structure of workers’ confusion matrices. In this article, we propose in response a Fine-Grained Bayesian Classifier Combination model (FGBCC), in which a combination of $K$ univariate Gaussian distributions and the standard softmax function is exploited with an aim to improve the estimation of workers’ abilities. Compared to existing methods, FGBCC is capable of learning extensive worker behaviors and is less susceptible to these issues that previous methods suffer from, owing to its stronger generalization ability. Moreover, Considering the exact solution to the complex posterior is unavailable, we devise a computationally efficient algorithm to approximate the posterior. Extensive experiments on 24 real-world datasets covering a wide range of domains, verify the clear advantages of FGBCC over 11 state-of-the-art benchmark methods.