Improving survey quality using paradata: Lessons from a field survey in India

Abstract

Motivation

When collecting evidence from the field, the quality of the data determines the reliability of the analysis. When data are collected in the field by enumerators, the latter's performance needs to be monitored to avoid errant behaviour that could compromise data quality.

Purpose

We show how paradata on the process of data collection itself can improve enumerator performance, using a household survey in India as a case study.

Approach and methods

We conducted action research to improve data quality in the India Working Study conducted in early 2020 in Karnataka and Rajasthan. We designed indicators (flags) from the paradata to mark potential deviant enumerator behaviour in the early stages of the survey. Flagged enumerators were contacted by supervisors who provided constructive feedback. We then measured the performance of the flagged enumerators over the remainder of the survey.

We were able to benchmark specific groups of enumerators facing similar field conditions, namely location and gender of respondents. This allowed us to compare enumerators to a subset of their peers, rather than the entire set of enumerators.

Findings

Our feedback improved enumerator behaviour in the field: flagged enumerators subsequently spent more time on a core module of the questionnaire.

Policy implications

In any survey, two objectives compete: completing a fixed number of interviews per day to reduce costs versus enumerators spending enough time with each respondent to collect meaningful data. To strike a balance between these competing demands, we recommend tracking three paradata indicators: count of completed interviews; average time per completed interview; and ratio of completed to initiated interviews.

We recommend using paradata to improve the quality of data when surveying, thereby reducing standard errors for estimates based on the data and leading to more reliable analysis.