The productivity of mental health care: an instrumental variable approach

Background

Like many other medical technologies and treatments, there is a lack of reliable evidence on treatment effectiveness of mental health care. Increasingly, data from non-experimental settings are being used to study the effect of treatment. However, as in a number of studies using non-experimental data, a simple regression of outcome on treatment shows a puzzling negative and significant impact of mental health care on the improvement of mental health status, even after including a large number of potential control variables. The central problem in interpreting evidence from real-world or non-experimental settings is, therefore, the potential ‘selection bias’ problem in observational data set. In other words, the choice/quantity of mental health care may be correlated with other variables, particularly unobserved variables, that influence outcome and this may lead to a bias in the estimate of the effect of care in conventional models.

Aims of the Study

This paper addresses the issue of estimating treatment effects using an observational data set. The information in a mental health data set obtained from two waves of data in Puerto Rico is explored. The results using conventional models—in which the potential selection bias is not controlled—and that from instrumental variable (IV) models—which is what was proposed in this study to correct for the contaminated estimation from conventional models—are compared.

Methods

Treatment effectiveness is estimated in a production function framework. Effectiveness is measured as the improvement in mental health status. To control for the potential selection bias problem, IV approaches are employed. The essence of the IV method is to use one or more instruments, which are observable factors that influence treatment but do not directly affect patient outcomes, to isolate the effect of treatment variation that is independent of unobserved patient characteristics. The data used in this study are the first (1992–1993) and second (1993–1994) wave of the ongoing longitudinal study Mental Health Care Utilization Among Puerto Ricans, which includes information for an island-wide probability sample of over 3000 adults living in poor areas of Puerto Rico. The instrumental variables employed in this study are travel distance and health insurance sources.

Results

It is very noticeable that in this study, treatment effects were found to be negative in all conventional models (in some cases, highly significant). However, after the IV method was applied, the estimated marginal effects of treatment became positive. Sensitivity analysis partly supports this conclusion. According to the IV estimation results, treatment is productive for the group in most need of mental health care. However, estimations do not find strong enough evidence to demonstrate treatment effects on other groups with less or no need. The results in this paper also suggest an important impact of the following factors on the probability of improvement in mental health status: baseline mental health status, previous treatment, sex, marital status and education.

Discussion

The IV approach provides a practical way to reduce the selection bias due to the confounding of treatment with unmeasured variables. The limitation of this study is that the instruments explored did not perform well enough in some IV equations, therefore the predictive power remains questionable. The most challenging part of applying the IV approach is on finding ‘good’ instruments which influence the choice/quantity of treatment yet do not introduce further bias by being directly correlated with treatment outcome.

Conclusions

The results in this paper are supportive of the concerns on the credibility of evaluation results using observation data set when the endogeneity of the treatment variable is not controlled. Unobserved factors contribute to the downward bias in the conventional models. The IV approach is shown to be an appropriate method to reduce the selection bias for the group in most need for mental health care, which is also the group of most policy and treatment concern.

Implications for Health Care Provision and Use

The results of this work have implications for resource allocation in mental health care. Evidence is found that mental health care provided in Puerto Rico is productive, and is most helpful for persons in most need for mental health care. According to what estimated from the IV models, on the margin, receiving formal mental health care significantly increases the probability of obtaining a better mental health outcome by 19.2%, and one unit increase in formal treatment increased the probability of becoming healthier by 6.2% to 8.4%. Consistent with other mental health literature, an individual’s baseline mental health status is found to be significantly related to the probability of improvement in mental health status: individuals with previous treatment history are less likely to improve. Among demographic factors included in the production function, being female, married, and high education were found to contribute to a higher probability of improvement.

Implication for Further Research

In order to provide accurate evidence of treatment effectiveness of medical technologies to support decision making, it is important that the selection bias be controlled as rigorously as possible when using information from a non-experimental setting. More data and a longer panel are also needed to provide more valid evidence. Copyright © 1999 John Wiley & Sons, Ltd.