DGNet: A Handwritten Mathematical Formula Recognition Network Based on Deformable Convolution and Global Context Attention

Abstract

The Handwritten Mathematical Expression Recognition (HMER) task aims to generate corresponding LATEX sequences from images of handwritten mathematical expressions. Currently, the encoder-decoder architecture has made significant progress in this task. However, the architecture based on the DenseNet encoder fails to adequately consider the unique features of handwritten mathematical expressions (HME) and the similarity between different characters. Additionally, the decoder, with its small receptive field during the decoding process, fails to effectively capture the spatial positional information of the targets, resulting in a lack of global contextual information during decoding. To address these issues, this paper proposes a neural network called DGNet based on deformable convolution and global contextual attention. Our network takes into full consideration the sparse nature of handwritten mathematical formulas and utilizes the properties of deformable convolution, allowing the convolution kernel to deform based on the content of the neighborhood. This enables our model to better adapt to geometric changes and other deformations in handwritten mathematical expressions. Simultaneously, we introduce GCAttention in optimizing the feature part to fully aggregate global contextual features of both position and channel. In experiments, our model achieved accuracies of 58.51%, 56.32%, and 56.1% on the CROHME 2014, 2016, and 2019 datasets, respectively.