Estimating the Diagnostic Accuracy from Multiple Raters Based on a Bivariate Random Effects Model

In clinical investigations designed to demonstrate the efficacy of a diagnostic procedure, the procedure is usually evaluated by multiple independent raters. Although the sensitivity and specificity may be estimated by considering consensus evaluations to treat results from multiple raters as if there were a single rater, raters are not consid-ered independent in consensus evaluations. Typically, estimation methods are based on an “average rater” or a “majority rater” to account for multiple raters. In this paper, we propose a method for summarizing sensitivities and specificities evaluated from multiple independent raters based on a bivariate random effects model (BVRM) to account between-rater variance and correlation between sensitivity and specificity. In addition, we propose methods to draw joint confidence regions of sensitivity and specificity based on the BVRM. Simulation results show that the differences in the biases between the proposed method and the average rater method are small and that the empirical coverage probabilities of the proposed joint confidence regions are close to the nominal level. The proposed methods are illustrated using data from florbetapir F 18 positron emission tomographic imaging to predict the presence of β -amyloid in the brains of subjects with Alzheimer’s We an If each of exceeded , the was to otherwise, it was to zero. Similarly, if each element of the matrix for the specificities exceeded z p b ) , the outcome of the element was set to one; otherwise, it was set to zero. Here, z − p ) is the 100 p percentile of the standard distribution. our simulation study, 10,000 data sets were