A novel hybrid approach for text encoding: Cognitive Attention To Syntax model to detect online misinformation

Abstract

Most approaches for text encoding rely on the attention mechanism, at the core of the transformers architecture and large language models. The understanding of this mechanism is still limited and present inconvenients such as lack of interpretability, large requirements of data and low generalization. Based on current understanding of the attention mechanism, we propose CATS (Cognitive Attention To Syntax), a neurosymbolic attention encoding approach based on the syntactic understanding of texts. This approach has on-par to better performance compared to classical attention and displays expected advantages of neurosymbolic AI such as better functioning with little data and better explainability. This layer has been tested on the task of misinformation detection but is general and could be used in any task involving natural language processing.