The digitization of Vietnamese certificates-characterized by a mixture of printed templates, handwritten entries, diverse fonts with diacritics, and official legal stamps-poses unique challenges for automated document analysis systems. This paper presents ViCertNet, a unified multimodal framework designed to address three core tasks: (1) accurate named entity recognition (NER) from hybrid printed-handwritten layouts, (2) robust preservation of Vietnamese diacritical marks across variable font styles, and (3) legal stamp authentication under realistic conditions. ViCertNet integrates LayoutLMv2 for layout-text structural encoding and ViBERTgrid for visual-semantic fusion, enhanced through a cascaded gated fusion module that aligns and merges dual OCR outputs (PaddleOCR for printed, TrOCR for handwritten text) via spatial attention. A diacritic-aware CRF layer is introduced to enforce Vietnamese orthographic constraints at character level, improving recognition accuracy. For legal validation, ViCertNet employs a three-stage stamp verification pipeline, including Mask R-CNN for localization, OCR-based microtext extraction, and a Siamese network for signature consistency analysis. Evaluated on VietCER, a curated dataset of over 3,000 manually annotated certificates from Vietnamese government and academic institutions, ViCertNet achieves 89.7% F1-score for NER, 93.5% stamp verification accuracy, and 1.9 s per page in processing time-reducing verification errors by 37% compared to strong baselines. This work demonstrates the effectiveness of combining multimodal learning with linguistic constraints for Southeast Asian document analysis and offers a scalable solution for real-world certificate verification.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Vietnamese Certificate Analysis: Addressing Hybrid Document Challenges Through Multimodal Layout-Text Fusion

  • Nam Duong Ho,
  • Duy Tran Ngoc Bao,
  • Quan Thi Khac,
  • Dang Le Binh

摘要

The digitization of Vietnamese certificates-characterized by a mixture of printed templates, handwritten entries, diverse fonts with diacritics, and official legal stamps-poses unique challenges for automated document analysis systems. This paper presents ViCertNet, a unified multimodal framework designed to address three core tasks: (1) accurate named entity recognition (NER) from hybrid printed-handwritten layouts, (2) robust preservation of Vietnamese diacritical marks across variable font styles, and (3) legal stamp authentication under realistic conditions. ViCertNet integrates LayoutLMv2 for layout-text structural encoding and ViBERTgrid for visual-semantic fusion, enhanced through a cascaded gated fusion module that aligns and merges dual OCR outputs (PaddleOCR for printed, TrOCR for handwritten text) via spatial attention. A diacritic-aware CRF layer is introduced to enforce Vietnamese orthographic constraints at character level, improving recognition accuracy. For legal validation, ViCertNet employs a three-stage stamp verification pipeline, including Mask R-CNN for localization, OCR-based microtext extraction, and a Siamese network for signature consistency analysis. Evaluated on VietCER, a curated dataset of over 3,000 manually annotated certificates from Vietnamese government and academic institutions, ViCertNet achieves 89.7% F1-score for NER, 93.5% stamp verification accuracy, and 1.9 s per page in processing time-reducing verification errors by 37% compared to strong baselines. This work demonstrates the effectiveness of combining multimodal learning with linguistic constraints for Southeast Asian document analysis and offers a scalable solution for real-world certificate verification.