Methodology for establishing a 'gold standard' (for a medical expert system)

Presents a statistical method for establishing a consensus diagnosis among several experts, which is the common 'gold standard' used for evaluating the diagnostic performance of a medical expert system. This method involves the use of the extended kappa statistic developed by J.L. Fleiss (1971) and R.J. Light (1971) in the social sciences for the study of a similar problem. The method is carried out in two stages. First, the existence of an overall agreement or disagreement among the medical experts in their diagnoses of a sample of patients is established. Second, in the case of overall agreement, one selects the particular disorders that have a significant level of agreement among the experts, and uses the experts' diagnoses as the 'gold standard' for the sample of patients that were classified in those disorders.<>