EGO-LM: An efficient, generic, and out-of-the-box language model for handwritten text recognition

The language model (LM) plays a crucial role in post-processing handwritten text recognition (HTR) by capturing linguistic patterns. However, traditional rule-based LMs are inefficient, and recent end-to-end LMs require customized training for each HTR model. To address these limitations, we propose an Efficient, Generic, and Out-of-the-box Language Model (EGO-LM) for HTR. To unlock the out-of-the-box capability of the end-to-end LM, we introduce a vision-limited proxy task that focuses on visual-pattern-agnostic linguistic dependencies during training, enhancing the robustness and generality of the LM. The enhanced capabilities also enable EGO-LM to iteratively refine its output for a further accuracy boost without additional tuning. Moreover, we introduce a Diverse-Corpus Online Handwriting dataset (DCOH-120K) with more diverse corpus types and more samples than existing datasets, including 83,142 Chinese and 39,398 English text lines. Extensive experiments demonstrate that EGO-LM can attain state-of-the-art performance while achieving up to 613$\times$ acceleration. The DCOH-120K dataset is available at .