Survey Research Methods最新文献

An increasing number of telephone surveys are introducing a web option. Limited experimental research has explored the implications of such design changes and how they vary by question characteristics. Using an experimental design and propensity score matching techniques, this study examines the effects of mode (web vs. telephone) and transitioning to a mixed-mode design (from telephone-only to the choice of web or telephone) on survey estimates and how effects differ by question characteristics. We draw upon an experiment embedded in the National Study of Caregiving (NSOC), in which half of NSOC-eligible caregivers were randomized to a telephone-only design and the other half to a sequential mixed-mode design offering web and telephone options. For each categorical survey item, we test whether responses differ significantly by mode (after adjusting for selection effects using propensity score matching) and by design (experimentally assigned). We find that for the matched sample, significant differences by mode are evident for 10% of categorical survey items. These differences are larger for the approximately 75% of subjective (vs. 25% objective) questions and 45% non-binary (vs. 55% binary) questions, but are nevertheless negligible in size (<0.07 phi coefficient or Cramer's V). Because most effects are small and only about half of those randomized to the mixed-mode design opted for web, these mode effects rarely result in differences in estimates (3%) between the telephone-only and mixed-mode designs and differences in estimates are on average negligible in size (<0.03). We demonstrate that even if web take-up rates reach 90%, significant differences in estimates between telephone-only and mixed-mode designs remain rare (5%) and on average negligible in size (0.08) for the mix of categorical questions in NSOC. We discuss implications for designing future mixed-mode surveys.

Wearable electronic devices are increasingly used to collect physical activity (PA) data. Most wearables provide PA feedback to users. The feedback has the potential for biasing measurements as users may increase their PA level in response to the feedback. On the other hand, the feedback can also be a desirable property as it can be employed to promote PA across settings. Studies examining the causal feedback effect independent of other factors have been limited. This study analyzed the causal effect of PA feedback provided by wearables. We implemented 4 field experiments over a period of 8 months. In each experiment, we recruited participants from a probability-based internet panel and asked them to wear continuously: (i) a PA device that does not provide feedback for 7 consecutive days (control condition), (ii) a PA device that provides feedback for another 7 consecutive days (treated condition), (iii) both devices for another 1 or 2 consecutive days. After 6 PM each experimental day, participants completed a short online survey asking them about their experiences participating in the study. Of 120 eligible participants assessed, 81 provided valid and complete data while 39 were lost to follow-up. Participants with valid and complete data (n=81) accumulated 7% more PA on a given day when they wore a PA device providing feedback relative to when they wore a PA device that does not provide feedback (p-value<0.001). The feedback effect was robust to the inclusion of additional factors that might influence PA. Use of research-grade PA devices that provide no feedback is warranted for studies whose primary goals are to collect population PA data with minimal measurement errors, while wearables with feedback are most suited for PA intervention studies. When using PA devices that provide feedback, one needs to be aware of the bias that may result.

Young adults are generally hard to survey, presenting researchers with numerous difficulties. They are hard to locate and contact due to high mobility. They are hard to persuade and exhibit high levels of resistance to survey participation. As a result, they pose a greater challenge for longitudinal surveys. This paper explores the role of mode of data collection in young adults' decisions to stay in a longitudinal panel. We draw on data from the National Young Adult Health Survey (NYAHS). NYAHS is a longitudinal study (three annual waves and 2 brief between-wave follow-up surveys) of adults aged 18-34 initially recruited in 2019 through RDD sampling of cell phone numbers nationwide. All sampled cell phone numbers were randomly assigned to one of three experimental conditions; the conditions differed in mode of data collection used in subsequent interviews once screened in. In the first condition, young adults continue all rounds of interviews by telephone ("telephone only" condition). The second group of young adults completed one round of interview by web and the rest by telephone ("telephone mostly" condition). The last third was asked to complete three interviews online and two interviews by telephone ("web mostly" condition). We examined the impact of mode switching on young adults' likelihood of participating in later surveys and on nonresponse bias in key survey outcomes. We found that switching young adults from telephone to web had an immediate negative effect on their likelihood of participating in that web survey, but it did not have a continued negative effect. Switching them from web to telephone increased response rates and reduced nonresponse bias. The findings have important practical implications on how to survey young adults.

The U.S. Panel Study of Income Dynamics (PSID) made a planned transition to a web-first mixed-mode data collection design in 2021 (web and computer-assisted telephone interviewing [CATI]), following nearly five decades of collecting data primarily using CATI with professional interviewers. To evaluate potential effects of mode on fieldwork outcomes, two sequential mixed-mode protocols were introduced using an experimental design. One protocol randomized sample families to a "web-first" treatment, which encouraged response through an online interview, followed by an offer of telephone to complete the interview; a second protocol randomized sample families to a "CATI-first" treatment until the last phase of fieldwork when the option to complete a web interview was offered. This paper examines the comparative effects of the two protocols on fieldwork outcomes, including response rates, interviewer contact attempts, fieldwork duration, and cost. Comparisons are also made with fieldwork outcomes and characteristics of non-responding sample members from the prior-wave when a traditional telephone design was used. We found that the web-first design compared to the CATI-first design led to comparably high response rates, and faster interview completion with lower effort and cost. With some notable exceptions, compared to the prior wave, the mixed-mode design reduced effort and had generally similar patterns of non-response among key respondent subgroups. The results provide new empirical evidence on the effects of mixing modes on fieldwork outcomes and costs and contribute to the small body of experimental evidence on the use of mixed-mode designs in household panel studies.