Integration of multiple annotators by aggregating experts and filtering novices

Learning from noisy labels obtained from multiple annotators and without access to any true labels is an increasingly important problem in bioinformatics and biomedicine. In our method, this challenge is addressed by iteratively filtering low-quality annotators and estimating the consensus labels based only on the remaining experts that provide higher-quality annotations. Experiments on biomedical text classification and CASP9 protein disorder prediction tasks provide evidence that the proposed algorithm is more accurate than the majority voting and previously developed multi-annotator approaches. The benefit of using the new method is particularly large when low-quality annotators dominate. Moreover, the new algorithm also suggests the most relevant annotators for each instance, thus paving the way for understanding the behaviors of each annotator and building more reliable predictive models for bioinformatics applications.