Handwritten Document Recognition Using Pre-trained Vision Transformers

Handwritten document recognition (HDR) is a rapidly growing field that aims to automate the process of document transcription, streamlining the traditional stages of segmentation and block recognition inherent in handwritten text recognition systems. However, the efficacy of HDR systems often encounters performance hurdles due to limited data availability, constraining their practical applicability. In addressing this challenge, our study delves into leveraging pre-trained vision transformer models, proposing two distinct architectures. One architecture integrates a transformer decoder, while the other relies on a decoder-free connectionist temporal classification approach. Consequently, we present a comprehensive analysis focused on adapting and optimizing weights to enhance performance efficiently. Our methodology encompasses exploring an optimal blend of hyperparameters through grid search, implementing freezing strategies, utilizing gradient accumulation, and employing hard negative mining techniques. This analysis demonstrates that our proposed efficient fine-tuning workflow significantly mitigates error rates and enhances inference speed, surpassing state-of-the-art results across most extended HDR benchmark datasets. Specifically, we outperform previous methods on the READ 2016 single-page and double-page with a word error rate (WER) of 11.49% and 11.86%, the RIMES 2009 with a WER of 9.58%, and MAURDOR C3 and C4 with a WER of 16.21% and 13.32%. The results of this study have important implications for the development of HDR systems, where our approach can lead to improved system performance and enable wider adoption of this technology. Source code is publicly available at https://github.com/dparres/Pretrained-Document-Recognition- Transformers .