Relationships of Cohen's Kappa, Sensitivity, and Specificity for Unbiased Annotations

For the binary classification tasks in supervised learning, the labels of data have to be available for classifier development. Cohen's kappa is usually employed as a quality measure for data annotation, which is inconsistent with its true functionality of assessing the inter-annotator consistency. However, the derived relationship functions of Cohen's kappa, sensitivity, and specificity in the literature are complicated, thus are unable to be employed to interpret classification performance from kappa values. In this study, based on an annotation generation model, we develop simple relationships of kappa, sensitivity, and specificity when there is no bias in the annotations. A relationship between kappa and Youden's J statistic, a performance metric for binary classification, is further obtained. The derived relationships are evaluated on a synthetic dataset using linear regression analysis. The results demonstrate the accuracy of the derived relationships. It suggests the potential of estimating classification performance from kappa values when bias is absent in the annotations.