MELODY: A Long-Term Dynamic Quality-Aware Incentive Mechanism for Crowdsourcing

Abstract

Crowdsourcing allows requesters to allocate tasks to a group of workers on the Internet to make use of their collective intelligence. Quality control is a key design objective in incentive mechanisms for crowdsourcing as requesters aim at obtaining answers of high quality under a given budget. However, when measuring workers' long-term quality, existing mechanisms either fail to utilize workers' historical information, or treat workers' quality as stable and ignore its temporal characteristics, hence performing poorly in a long run. In this paper we propose MELODY, a long-term dynamic quality-aware incentive mechanism for crowdsourcing. MELODY models interaction between requesters and workers as reverse auctions that run continuously. In each run of MELODY, we design a truthful, individual rational, budget feasible and quality-aware algorithm for task allocation with polynomial-time computation complexity and O(1) performance ratio. Moreover, taking into consideration the long-term characteristics of workers' quality, we propose a novel framework in MELODY for quality inference and parameters learning based on Linear Dynamical Systems at the end of each run, which takes full advantage of workers' historical information and predicts their quality accurately. Through extensive simulations, we demonstrate that MELODY outperforms existing work in terms of both quality estimation (reducing estimation error by 17.6% ~ 24.2%) and social performance (increasing requester's utility by 18.2% ~ 46.6%) in long-term scenarios.