Using crowdsourced data for empirical research in information systems: What it is and how to do it safe?

Abstract

Industry-relevant information systems (IS) and software engineering (SE) research assumes practitioners' involvement, be it in the exploration of the state-of-the-art practice or in the investigation of real-life problems experienced in organizations. Crowdsourcing is an appealing concept for collecting practitioners' perceptions on an industry-relevant phenomenon that is of interest to researchers. As practitioners-generated contents are easily available in social media platforms such as practitioners' blogs or professional discussion groups in LinkedIn, researchers face the opportunity to use this crowdsourced information for the purpose of gaining understanding of a situation from the point of view of the professionals involved therein. While there are many benefits of using crowdsourcing for data collection, there are also challenges, all of which pose validity threats of various degrees to the empirical results obtained. This tutorial will provide a systematic understanding of the use of practitioners' crowdsourced data for empirical research purposes, and of the possible ways to safely apply it in IS and SE research. The tutorial leverages the tutor's experience and lessons learned from using practitioners' blogs articles for qualitative research in business-IT alignment and in large scale online games. At the end of the tutorial, attendees should be able to critically reason about (i) the possible choices in designing a crowdsourcing-based research process, (ii) the quality criteria for judging their research designs, and (iii) the criteria for evaluating their studies.