An analysis of interobserver variability in the recording of maxillary sinusitis in human osteoarchaeological remains

The accurate comparison of prevalence rates of disease between different groups/populations is vital if we seek to contextualize our understanding of the impact of risk factors on health in the past. The majority of bioarchaeological studies of maxillary sinusitis employ the methods and “diagnostic criteria” outlined by Boocock and colleagues in 1995. However, until now, few attempts have been made to assess the inter-rater reliability of these methods. This study presents the results of the analysis of interobserver variability in the recording of bone changes related to maxillary sinusitis among three observers within three human osteoarchaeological populations. The results of Cohen's kappa coefficient tests indicated variability in agreement between different observers. The agreement on the presence/absence of maxillary sinusitis in different osteoarchaeological populations ranged from κ = 0.433 (“moderate” agreement) to κ = 0.629 (“substantial” agreement). The agreement on the type of bone change present within affected sinuses was often poor, with almost no to negative agreement for pitting (Observers 1 and 2) and remodeled spicules (Observers 1 and 3). Methodological problems that can impact consistency of results between researchers include variability in sinus preservation, duration of observer experience, the use of different endoscopic equipment, lack of clarity in the original method descriptions, and a deficit in clinical corroborations for bone changes. The results of this study highlight the need to improve standards for the recording of bone changes related to sinusitis to allow for meaningful comparisons of past maxillary sinusitis frequency rates. Further investigations of interobserver variability, incorporating a greater range of variables, are also required.