Comprehensive guidelines for emotion annotation

Emotions are psychological traits which are associated with an individuals' thoughts, feelings, behavioral responses, and experiences of pleasure and displeasure. The ability to recognise a conversational partner's emotional state from their speech (and respond accordingly) is a longstanding requirement of a fully capable intelligent virtual agent. However, despite the fact that current approaches to emotion recognition primarily depend upon supervised machine learning models, there are no comprehensive guidelines for emotion label annotation of the corpora used to train such models. We present comprehensive guidelines for consistent and effective annotation of text corpora with emotion labels. In particular, our proposal directly addresses the requirements of multi-label emotion recognition, and we demonstrate how an implementation of our proposed guidelines led to substantially (30%) higher agreement score among human annotators.