Who Should Take This Task?: Dynamic Decision Support for Crowd Workers

Abstract

Context: The success of crowdsourced software development (CSD) depends on a large crowd of trustworthy software workers who are registering and submitting for their interested tasks in exchange of financial gains. Preliminary analysis on software worker behaviors reveals an alarming task-quitting rate of 82.9%. Goal: The objective of this study is to empirically investigate worker decision factors and provide better decision support in order to improve the success and efficiency of CSD. Method: We propose a novel problem formulation, DCW-DS, and an analytics-based decision support methodology to guide workers in acceptance of offered development tasks. DCS-DS is evaluated using more than one year's real-world data from TopCoder, the leading CSD platform. Results: Applying Random Forest based machine learning with dynamic updates, we can predict a worker as being a likely quitter with 99% average precision and 99% average recall accuracy. Similarly, we achieved 78% average precision and 88% average recall for the worker winner class. For workers just following the top three task recommendations, we have shown that the average quitting rate goes down below 6%. Conclusions: In total, the proposed method can be used to improve total success rate as well as reduce quitting rate of tasks performed.