The Challenge of Evaluating Virtual Communities of Practice: A Systematic Mapping Study

Abstract

Aim/Purpose: This paper presents a study of Virtual Communities of Practice (VCoP) evaluation methods that aims to identify their current status and impact on knowledge sharing. The purposes of the study are as follows: (i) to identify trends and research gaps in VCoP evaluation methods; and, (ii) to assist researchers to position new research activities in this domain. Background: VCoP have become a popular knowledge sharing mechanism for both individuals and organizations. Their evaluation process is complex; however, it is recognized as an essential means to provide evidences of community effectiveness. Moreover, VCoP have introduced additional features to face to face Communities of Practice (CoP) that need to be taken into account in evaluation processes, such as geographical dispersion. The fact that VCoP rely on Information and Communication Technologies (ICT) to execute their practices as well as storing artifacts virtually makes more consistent data analysis possible; thus, the evaluation process can apply automatic data gathering and analysis. Methodology: A systematic mapping study, based on five research questions, was carried out in order to analyze existing studies about VCoP evaluation methods and frameworks. The mapping included searching five research databases resulting in the selection of 1,417 papers over which a formal analysis process was applied. This process led to the preliminary selection of 39 primary studies for complete reading. After reading them, we select 28 relevant primary studies from which data was extracted and synthesized to answer the proposed research questions. Contribution: The authors of the primary studies analyzed along this systematic mapping propose a set of methods and strategies for evaluating VCoP, such as frameworks, processes and maturity models. Our main contribution is the identification of some research gaps present in the body of studies, in order to stimulate projects that can improve VCoP evaluation methods and support its important role in social learning. Findings: The systematic mapping led to the conclusion that most of the approaches for VCoP evaluation do not consider the combination of data structured and unstructured metrics. In addition, there is a lack of guidelines to support community operators’ actions based on evaluation metrics.