{"title":"通过聚合专家和过滤新手来集成多个注释器","authors":"Ping Zhang, Z. Obradovic","doi":"10.1109/BIBM.2012.6392657","DOIUrl":null,"url":null,"abstract":"Learning from noisy labels obtained from multiple annotators and without access to any true labels is an increasingly important problem in bioinformatics and biomedicine. In our method, this challenge is addressed by iteratively filtering low-quality annotators and estimating the consensus labels based only on the remaining experts that provide higher-quality annotations. Experiments on biomedical text classification and CASP9 protein disorder prediction tasks provide evidence that the proposed algorithm is more accurate than the majority voting and previously developed multi-annotator approaches. The benefit of using the new method is particularly large when low-quality annotators dominate. Moreover, the new algorithm also suggests the most relevant annotators for each instance, thus paving the way for understanding the behaviors of each annotator and building more reliable predictive models for bioinformatics applications.","PeriodicalId":6392,"journal":{"name":"2012 IEEE International Conference on Bioinformatics and Biomedicine Workshops","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2012-10-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"7","resultStr":"{\"title\":\"Integration of multiple annotators by aggregating experts and filtering novices\",\"authors\":\"Ping Zhang, Z. Obradovic\",\"doi\":\"10.1109/BIBM.2012.6392657\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"Learning from noisy labels obtained from multiple annotators and without access to any true labels is an increasingly important problem in bioinformatics and biomedicine. In our method, this challenge is addressed by iteratively filtering low-quality annotators and estimating the consensus labels based only on the remaining experts that provide higher-quality annotations. Experiments on biomedical text classification and CASP9 protein disorder prediction tasks provide evidence that the proposed algorithm is more accurate than the majority voting and previously developed multi-annotator approaches. The benefit of using the new method is particularly large when low-quality annotators dominate. Moreover, the new algorithm also suggests the most relevant annotators for each instance, thus paving the way for understanding the behaviors of each annotator and building more reliable predictive models for bioinformatics applications.\",\"PeriodicalId\":6392,\"journal\":{\"name\":\"2012 IEEE International Conference on Bioinformatics and Biomedicine Workshops\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2012-10-04\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"7\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"2012 IEEE International Conference on Bioinformatics and Biomedicine Workshops\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.1109/BIBM.2012.6392657\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"2012 IEEE International Conference on Bioinformatics and Biomedicine Workshops","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/BIBM.2012.6392657","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
Integration of multiple annotators by aggregating experts and filtering novices
Learning from noisy labels obtained from multiple annotators and without access to any true labels is an increasingly important problem in bioinformatics and biomedicine. In our method, this challenge is addressed by iteratively filtering low-quality annotators and estimating the consensus labels based only on the remaining experts that provide higher-quality annotations. Experiments on biomedical text classification and CASP9 protein disorder prediction tasks provide evidence that the proposed algorithm is more accurate than the majority voting and previously developed multi-annotator approaches. The benefit of using the new method is particularly large when low-quality annotators dominate. Moreover, the new algorithm also suggests the most relevant annotators for each instance, thus paving the way for understanding the behaviors of each annotator and building more reliable predictive models for bioinformatics applications.