Crowdvoting Judgment: An Analysis of Modern Peer Review

In this paper, we propose and analyze models of self policing in online communities, in which assessment activities, typically handled by firm employees, are shifted to the “crowd.” Our underlying objective is to maximize firm value by maintaining the quality of the online community to prevent attrition, which, given a parsimonious model of voter participation, we show can be achieved by efficiently utilizing the crowd of volunteer voters. To do so, we focus on minimizing the number of voters needed for each assessment, subject to service-level constraints, which depends on a voting aggregation rule. We focus our attention on classes of voting aggregators that are simple, interpretable, and implementable, which increases the chance of adoption in practice. We consider static and dynamic variants of simple majority-rule voting, with which each vote is treated equally. We also study static and dynamic variants of a more sophisticated voting rule that allows more accurate voters to have a larger influence in determining the aggregate decision. We consider both independent and correlated voters and show that correlation is detrimental to performance. Finally, we take a system view and characterize the limit of a costless crowdvoting system that relies solely on volunteer voters. If this limit does not satisfy target service levels, then costly firm employees are needed to supplement the crowd.