Are Web-Based Valuation Surveys for Preference-Based Measures as Reliable as Face-to-Face Surveys? TTO, DCE and DCE with Duration

Abstract

Background

Valuation surveys of preference-based measures are typically conducted face-to-face or on web panels. In this survey, we considered whether face-to-face and online surveys were reliable using three tasks: composite time trade-off (cTTO), discrete choice experiment (DCE), and DCE with duration.

Methods

Respondents (aged 20–69 years) for both face-to-face (N = 1000, target sample size) and web surveys were selected through quota sampling by sex and age from each panel of the general population in Japan. They were then allocated to one of the three tasks and divided into six groups (two survey modes × three tasks, N = 334 per group). For the cTTO, respondents were asked to rate ten health states described by the EQ-5D-5L. For the DCE and DCE with duration surveys, respondents were asked about 15 health–state pairs. For all participants, as in the second-stage survey, a similar process was repeated two weeks after the first survey. Reliability was evaluated by calculating the percentage of agreement and intraclass correlation coefficients.

Results

The cTTO scores of the face-to-face and web surveys were systematically different. Between the face-to-face and web surveys, the agreement of the TTO survey was not good. The intraclass correlation coefficient (ICC) was 0.37 for the face-to-face test-test and 0.59 for the web test-retest. Discrete choice experiment (DCE) and DCE with duration had similarly good agreement (more than 70%), regardless of face-to-face or web surveys. However, between the first and second surveys (test-retest) of DCE and DCE with duration, the agreement depends on whether the positions of the two cards (health states) are identical.

Conclusion

If the face-to-face cTTO score is the gold standard, a web-based survey of cTTO is not recommended regardless of the ICC. If a DCE survey is performed, positioning effects should be considered.