The 2009 FDA PRO guidance, Potential Type I error, Descriptive Statistics and Pragmatic estimation of the number of interviews for item elicitation.

Abstract

A statistical methodology named "capture recapture", a Kaplan-Meier Summary Statistic, and an urn model framework are presented to describe the elicitation, then estimate both the number of interviews and the total number of items ("codes") that will be elicited during patient interviews, and present a summary graphical statistic that "saturation" has occurred. This methodology is developed to address a gap in the FDA 2009 PRO and 2012 PFDD guidance for determining the number of interviews (sample size). This estimate of the number of interviews (sample size) uses a two-step procedure. The estimate of the total number of items is then used to estimate the number of interviews to elicit all items. A framework called an urn model is a framework for describing the elicitation and demonstrate the algorithm for declaring saturation "first interview with zero new codes". A caveat emptor is that due to independence assumptions, the urn model is not used as a method for estimating probabilities. The URN model provides a framework to demonstrate that an algorithm such as "first interview with zero new codes" may establish that all codes have been elicited. The limitations of the Urn model, capture recapture, and Kaplan-Meier are summarized. The statistical methods and the estimates supplement but do not replace expert judgement and declaration of "saturation." A graphical summary statistic is presented to summarize "saturation," after expert declaration for two algorithms. An example of a capture-recapture estimate, using simulated data is provided. The example suggests that the estimate of total number of codes may be accurate when prepared as early as the second interview. A second simulation is presented with an URN model, under a strong assumption of independence that an algorithm such as 'first interview with zero new codes" may fail to identify all codes. Potential errors in declaration of saturation are presented. Recommendations are presented for additional research and the use of the algorithm "first interview with zero new codes."