Crowdsourcing experiments with a video analytics system

Abstract

The need for more experimental data, but also quicker and cheaper, lead us beyond traditional lab experiments, approaching a new subject pool via a crowdsourcing platform. SocialSkip is an open system that leverages the video clickstream data for extracting useful information about the video content and the viewers. The difficulty of embedding a pre-existing system as a task demands a carefully designed interface, adjusting experiments and be aware of workers' cheating behavior. We present a replicable task design and by analyzing crowdsourced results, we highlight problems in experimental procedure and propose potential solutions for future crowdsourcing experiments. The proposed crowdsourcing methodology achieved the collection of a significant amount of video clickstream data, in a timely manner and with affordable cost. Our findings indicate that future social media analytics systems should include an integrated crowdsourcing module. Further research should focus on collecting more data by controlling the random worker behavior a priori.