Interrater agreement in headache diagnoses

Background: Diagnosing headache disorders comprises the collection and interpretation of information. This study estimates agreement and bias in the latter. Methods: Physicians and medical students diagnosed eight patients’ headaches using the International Classification of Headache Disorders. We calculated Cohen’s Kappa for all participants and subgroups (board-certified neurologists, physicians working in a neurology department). Moreover, we asked how sure they felt about their diagnoses. Finally, participants estimated the number of different headache diagnoses a patient receives when consulting many physicians for the same headache and indicated the highest acceptable number. Results: The data of 63 participants entered the analysis, of whom 18 were neurologists (18/63, 28.6%), and 41 were currently working at a neurology clinic (41/63, 66.7%). Cohen’s Kappa decreased (0.706, 0.566, and 0.408) with increasing levels of the classification hierarchy. Interrater agreement was highest among neurologists. Physicians not working in a neurology clinic tended to diagnose secondary headaches more often were less confident about their diagnoses. Conclusions: Physicians with less experience in headache disorders struggle more to diagnose headaches than neurologists do; they suspect secondary headaches, disagree, and feel insecure more often. Thus, interpreting a headache history is prone to error and bias.