GHA: A Gated Hierarchical Attention Mechanism for the Detection of Abusive Language in Social Media

Abstract

The use of attention mechanisms in deep learning solutions has become popular within natural language processing tasks. The use of these mechanisms allows managing the relevance of the elements of a sequence in accordance with their context, however, this relevance has been observed independently between the pairs of elements of a sequence (self-attention) or between the application domain of a sequence (contextual attention), leading to the loss of relevant information and limiting the representation of the sequences. To tackle these particular issues, we propose a dual attention mechanism, which trades off the previous limitations, by considering the internal and contextual relationships between the elements of the sequence. Additionally, we propose the extension of the dual attention mechanism into a multi-layer perspective, through the weighted fusion of the different encoding layers of deep architectures. As the interpretation of abusive language is highly context-dependent, its identification is an ideal task to evaluate the proposed attention mechanism. Accordingly, we considered six standard collections for the abusive language identification task. The obtained results are encouraging; the proposed hierarchical attention mechanism outperformed the current self-attention and contextual attention mechanisms coupled with recurrent neural networks and Transformers, as well as, state-of-the-art approaches in detecting abusive language.