Optical Character Recognition (OCR) for Indian languages is challenging due to diverse scripts, complex characters, and varied writing styles. This work presents HindiOCR-VLM, a Vision-Language Model adapted for Hindi OCR using Low-Rank Adaptation (LoRA) on a model pre-trained for Chinese and English. We explore two adaptation strategies: (i) a single-stage model that directly predicts text at the page level—bypassing intermediate word or line detection—focused on printed documents, and (ii) a two-stage model for multi-domain scenarios (printed, handwritten, and scene text), which first detects words and then recognizes them individually. Inspired by human learning processes, we propose a progressive learning approach—a training strategy to the single-stage model to enhance language acquisition and accelerate convergence. Leveraging the vision encoder’s rich representations, our method enables effective multi-domain training. Experimental results and ablation studies show that HindiOCR-VLM handles the complexities of the Devanagari script well and outperforms domain-specific models, offering a unified and robust solution for Hindi OCR. Code and resources are available at:  https://hindiocr-vlm.github.io/ .

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Adapting Vision-Language Models for Hindi OCR

  • Shaon Bhattacharyya,
  • Souvik Ghosh,
  • Prantik Deb,
  • Ajoy Mondal,
  • C. V. Jawahar

摘要

Optical Character Recognition (OCR) for Indian languages is challenging due to diverse scripts, complex characters, and varied writing styles. This work presents HindiOCR-VLM, a Vision-Language Model adapted for Hindi OCR using Low-Rank Adaptation (LoRA) on a model pre-trained for Chinese and English. We explore two adaptation strategies: (i) a single-stage model that directly predicts text at the page level—bypassing intermediate word or line detection—focused on printed documents, and (ii) a two-stage model for multi-domain scenarios (printed, handwritten, and scene text), which first detects words and then recognizes them individually. Inspired by human learning processes, we propose a progressive learning approach—a training strategy to the single-stage model to enhance language acquisition and accelerate convergence. Leveraging the vision encoder’s rich representations, our method enables effective multi-domain training. Experimental results and ablation studies show that HindiOCR-VLM handles the complexities of the Devanagari script well and outperforms domain-specific models, offering a unified and robust solution for Hindi OCR. Code and resources are available at:  https://hindiocr-vlm.github.io/ .