Dynamic Weighted Majority Approach for Detecting Malicious Crowd Workers

Abstract

Crowdsourcing is a paradigm that utilizes human intelligence to solve problems that computers cannot yet solve. However, the introduction of human intelligence into computations has also resulted in new challenges in quality control. These challenges originate from the malicious behaviors of crowd workers. Malicious workers are workers with hidden motives, who either simply sabotage a task or provide arbitrary responses to attain some monetary compensation. Recently, many studies have tried to detect and reduce the impact of malicious workers. The mechanisms vary from using ground truth to peer review by experts. Although the use of such mechanisms may increase the overall quality of outputs, it also imposes overhead costs in terms of money and/or time, with such costs being often remarkable and contradictory to the philosophy of crowdsourcing. In this paper, a novel dynamic weighted majority method is introduced to detect malicious workers based on a new malicious metric. Effectiveness of the proposed methodology is then showed by presenting the experimental results.