Nested effects testing: a multidimensional approach for evaluating CSCW systems

Abstract

Testing the extent to which the human interface of software is 'usable' for the individual user knows a well-developed methodology (see e.g. [6]). However there are many more issues that determine the value of CSCW systems. Among these issues are whether the system improves access to information and Supports interpersonal communicative behaviour and co-operation, whether it contributes to group member's purposes and to the organisation's objectives. Assessment of side effects, related to e.g. quality of work, privacy or power(re)distribution is also relevant. An adequate evaluation approach should include these issues in a comprehensive approach. We developed such an approach, the 'Nested Effects Testing-approach', developed in a European Union sponsored project (MEGATAQ 1), called Methods and Guidelines for the Assessment of Telematics Applications Quality (MEGATAQ) contains a framework aimed at covering most of these aspects. The approach It is based on a combination of various theoretical approaches, stemming from four research traditions [1 ]:• human-computer-interaction,• communication theory,• group dynamics• organisational effectivenessThe techniques focus on the user-centred aspects of evaluation, including interface usability, human communicative behaviour, collaborative work, and teamwork within the social context.Many of these theories adhere to one of two basic perspectives on the effectiveness of CSCW systems, i.e. the 'media match' perspective (technical systems have to fit the task and the context) or the 'adaptation' perspective (users adapt the system, the context and themselves to each other [see e.g. 3, 4, 5].The objective of MEGATAQ is to provide so called Telematics Applications Projects with guidelines and techniques for effective evaluation. These techniques can be used when TA Project consortia are developing and validating Telematics Applications. The techniques focus on the user-centred aspects of evaluation, including interface usability, human communicative behaviour, collaborative work, and teamwork within the social context.