Using randomized controlled trials to evaluate socially complex services: problems, challenges and recommendations

Abstract

Background:

Following the lead of evidence-based medicine, practice based on effectiveness research has become the new gold standard of contemporary public policy. Studies of this sort are increasingly demanded to evaluate services provided by mental health, social services and criminal justice systems.

Aims:

The paper questions whether the simple randomized controlled trial (RCT) paradigm as applied in clinical trials can be used ‘off the rack’ to evaluate socially complex service (SCS) interventions. These are services that are characterized by complex, diverse and non-standardized staffing arrangements; ambiguous protocols; hard-to-define study samples and unevenly motivated subjects and dependence on broader social environments. The difficulty of ensuring precise protocols, equivalent groups (tied to a meaningful target population) and neutral and equivalent trial environments under real world conditions are explored, as are the implications of not achieving standardization and equivalence.

Methods:

Limitations of effectiveness research as a research tool and information source are examined by comparing the assumptions underpinning the simple RCT to the characteristics of SCS interventions, as illustrated by programs targeted to mentally disordered offenders in Britain.

Results:

SCSs violate the assumptions underpinning the simple RCT model in ways that draw into sharp question the validity, reliability and generalizability of inferences of SCS trials.

Discussion:

The RCT is not a panacea. Effectiveness research of SCS interventions that is based on the RCT model is unlikely to yield valid, reliable and generalizable inferences without becoming more complex in design and more sensitive to issues of selection bias, unmeasured variables and endogeneity. Ten recommendations are offered for stylizing the RCT design to the characteristics of socially complex services.

Implications:

It remains an empirical issue whether RCT-based services effectiveness research can inform mental health policy. Without major design innovations, it is more likely that the information generated by this research will have limited practical use, especially if the RCT model is unable to control for the effect of social complexity and the interaction between social complexity and dynamic systemic change. Scientific evaluations of services make clinical and economic sense so long as they are designed to meet the challenges of the services of which they promise greater knowledge. Copyright © 2000 John Wiley & Sons, Ltd.