更好的众包编码:提高众包内容分析准确性的策略

IF 6.3 1区文学 Q1 COMMUNICATION Communication Methods and Measures Pub Date : 2021-04-03 DOI:10.1080/19312458.2021.1895977

Ceren Budak, R. Garrett, Daniel J. Sude

{"title":"更好的众包编码:提高众包内容分析准确性的策略","authors":"Ceren Budak, R. Garrett, Daniel J. Sude","doi":"10.1080/19312458.2021.1895977","DOIUrl":null,"url":null,"abstract":"ABSTRACT In this work, we evaluate different instruction strategies to improve the quality of crowdcoding for the concept of civility. We test the effectiveness of training, codebooks, and their combination through 2 × 2 experiments conducted on two different populations – students and Amazon Mechanical Turk workers. In addition, we perform simulations to evaluate the trade-off between cost and performance associated with different instructional strategies and the number of human coders. We find that training improves crowdcoding quality, while codebooks do not. We further show that relying on several human coders and applying majority rule to their assessments significantly improves performance.","PeriodicalId":47552,"journal":{"name":"Communication Methods and Measures","volume":"15 1","pages":"141 - 155"},"PeriodicalIF":6.3000,"publicationDate":"2021-04-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1080/19312458.2021.1895977","citationCount":"3","resultStr":"{\"title\":\"Better Crowdcoding: Strategies for Promoting Accuracy in Crowdsourced Content Analysis\",\"authors\":\"Ceren Budak, R. Garrett, Daniel J. Sude\",\"doi\":\"10.1080/19312458.2021.1895977\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"ABSTRACT In this work, we evaluate different instruction strategies to improve the quality of crowdcoding for the concept of civility. We test the effectiveness of training, codebooks, and their combination through 2 × 2 experiments conducted on two different populations – students and Amazon Mechanical Turk workers. In addition, we perform simulations to evaluate the trade-off between cost and performance associated with different instructional strategies and the number of human coders. We find that training improves crowdcoding quality, while codebooks do not. We further show that relying on several human coders and applying majority rule to their assessments significantly improves performance.\",\"PeriodicalId\":47552,\"journal\":{\"name\":\"Communication Methods and Measures\",\"volume\":\"15 1\",\"pages\":\"141 - 155\"},\"PeriodicalIF\":6.3000,\"publicationDate\":\"2021-04-03\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"https://sci-hub-pdf.com/10.1080/19312458.2021.1895977\",\"citationCount\":\"3\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Communication Methods and Measures\",\"FirstCategoryId\":\"98\",\"ListUrlMain\":\"https://doi.org/10.1080/19312458.2021.1895977\",\"RegionNum\":1,\"RegionCategory\":\"文学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"COMMUNICATION\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Communication Methods and Measures","FirstCategoryId":"98","ListUrlMain":"https://doi.org/10.1080/19312458.2021.1895977","RegionNum":1,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMMUNICATION","Score":null,"Total":0}

引用次数: 3

摘要

在这项工作中，我们评估了不同的教学策略，以提高文明概念的众包编码质量。我们通过对两种不同人群(学生和亚马逊土耳其机械工人)进行的2 × 2实验来测试培训、密码本及其组合的有效性。此外，我们执行模拟来评估与不同教学策略和人类编码器数量相关的成本和性能之间的权衡。我们发现，培训提高了众包编码的质量，而代码本却没有。我们进一步表明，依赖几个人类编码人员并将多数决原则应用于他们的评估，可以显著提高性能。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

查看原文

微信好友朋友圈 QQ好友复制链接

本刊更多论文

Better Crowdcoding: Strategies for Promoting Accuracy in Crowdsourced Content Analysis

ABSTRACT In this work, we evaluate different instruction strategies to improve the quality of crowdcoding for the concept of civility. We test the effectiveness of training, codebooks, and their combination through 2 × 2 experiments conducted on two different populations – students and Amazon Mechanical Turk workers. In addition, we perform simulations to evaluate the trade-off between cost and performance associated with different instructional strategies and the number of human coders. We find that training improves crowdcoding quality, while codebooks do not. We further show that relying on several human coders and applying majority rule to their assessments significantly improves performance.

求助全文

通过发布文献求助，成功后即可免费获取论文全文。去求助

来源期刊

Communication Methods and Measures COMMUNICATION-

CiteScore

21.10

自引率

1.80%

发文量

期刊介绍： Communication Methods and Measures aims to achieve several goals in the field of communication research. Firstly, it aims to bring attention to and showcase developments in both qualitative and quantitative research methodologies to communication scholars. This journal serves as a platform for researchers across the field to discuss and disseminate methodological tools and approaches. Additionally, Communication Methods and Measures seeks to improve research design and analysis practices by offering suggestions for improvement. It aims to introduce new methods of measurement that are valuable to communication scientists or enhance existing methods. The journal encourages submissions that focus on methods for enhancing research design and theory testing, employing both quantitative and qualitative approaches. Furthermore, the journal is open to articles devoted to exploring the epistemological aspects relevant to communication research methodologies. It welcomes well-written manuscripts that demonstrate the use of methods and articles that highlight the advantages of lesser-known or newer methods over those traditionally used in communication. In summary, Communication Methods and Measures strives to advance the field of communication research by showcasing and discussing innovative methodologies, improving research practices, and introducing new measurement methods.