Algorithmic Improvement of Crowdsourced Data: Intrinsic Quality Measures, Local Optima, and Consensus

Abstract

Raw crowdsourced data is often of questionable quality. The typical solution to this is redundancy: ask multiple independent participants the same question and take some form of majority answer. However, this can be wasteful in terms of human effort. In this paper we show that algorithmic analysis of the data is able to get higher quality results out of a given amount of crowd effort (or alternatively, that less crowd effort would have sufficed for the same level of quality). Our case study is based on a publicly available crowdsourced data set by the New York Public Library, featuring building footprints in historical insurance atlases. Besides evaluating the quality improvement achieved by our methods, we provide both a command line interface for batch-mode processing and an interactive web interface; both work with standard data formats and are available as open source software.