Predicting Crowdworkers' Performance as Human-Sensors for Robot Navigation

This paper provides and evaluates a new paradigm for collaborative human-robot operation in search and rescue-like settings with information asymmetry. In particular, we focus on settings where the human, a crowdworker in our case, is used as a sensor, providing the route-planning module with essential environmental information. In such settings, the ability to predict the expected performance of the collaborating crowdworker in real-time is instrumental for maintaining a continuously high level of performance. Through an extensive set of experiments with crowdworkers recruited and interacted through Amazon Mechanical Turk, we show that effective online prediction is indeed possible, however only if distinguishing between two subpopulations of crowdworkers, termed ”operators” and ”sensors”, applying a different prediction model to each. Furthermore, we show that even the classification of crowdworkers to the two types can be carried out successfully in real-time, based merely on the first two minutes of collaboration. Finally, we demonstrate how the above abilities can be used for a more effective workers’ recruiting process, resulting in a substantially improved overall performance.