Scholarlink

<p>With the advancement of smart re-baking factories and the accelerating digital transformation of agriculture, tobacco re-baking is facing an urgent need to shift from manual inspection to intelligent automated systems for quality traceability and information management. However, traditional cigarette box label information extraction methods still suffer from limited efficiency, insufficient accuracy, and poor robustness under real industrial conditions. To address these challenges, we propose MLDP (Multimodal Learning for Document Processing), a multimodal framework tailored for robust information extraction from tobacco box labels.MLDP integrates textual and visual information in a unified framework. For the text modality, multiple DeBERTa-based variants, including BiLSTM-DeBERTa, Multi-Sample Dropout-DeBERTa, Distil-DeBERTa, Two-Stage Dropout-DeBERTa, and LongMax-Dropout-DeBERTa, are combined through an Optuna-driven weighted ensemble strategy to balance model diversity and computational efficiency. For the image modality, PaddleOCR is adopted for text detection and recognition, and a Levenshtein distance-based correction mechanism is introduced to reduce OCR errors, especially in numeric fields. To further enhance cross-modal complementarity, MLDP employs a dynamic weighted fusion strategy together with a decision matrix for modality-level error correction, improving extraction robustness under noisy and complex industrial conditions.Experiments on a real-world multimodal tobacco box label dataset show that MLDP achieves a Micro-F1 of 96.51%, outperforming the text-only baseline (MLDP-T) and image-only baseline (MLDP-V) by 4.34% and 5.73%, respectively. Compared with classical document understanding baselines, including LayoutLMv3, UDOP, and Donut, and recent advanced document parsing or multimodal baselines such as MinerU2.5, PaddleOCR-VL, and Qwen2.5-VL, MLDP achieves the best overall performance and provides a favorable accuracy-efficiency trade-off.In addition, on the DocBank benchmark, MLDP yields 1.18%-1.55% gains on non-textual elements, demonstrating promising generalization ability beyond the target industrial scenario. Nevertheless, the current framework still depends on OCR quality and has been validated mainly on tobacco box labels, so its robustness under more diverse industrial document distributions requires further study. Furthermore, we construct and open-source an industrial tobacco box label multimodal dataset containing 9,719 graphically aligned samples, providing a useful benchmark resource for this task.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Robust and accurate multimodal information extraction from tobacco box labels using MLDP

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年7月24日

查看全文
  • Songling Huang,
  • Shuai Zhang,
  • Minghua Han,
  • Jianping Yang,
  • Yuyan Bai

摘要

With the advancement of smart re-baking factories and the accelerating digital transformation of agriculture, tobacco re-baking is facing an urgent need to shift from manual inspection to intelligent automated systems for quality traceability and information management. However, traditional cigarette box label information extraction methods still suffer from limited efficiency, insufficient accuracy, and poor robustness under real industrial conditions. To address these challenges, we propose MLDP (Multimodal Learning for Document Processing), a multimodal framework tailored for robust information extraction from tobacco box labels.MLDP integrates textual and visual information in a unified framework. For the text modality, multiple DeBERTa-based variants, including BiLSTM-DeBERTa, Multi-Sample Dropout-DeBERTa, Distil-DeBERTa, Two-Stage Dropout-DeBERTa, and LongMax-Dropout-DeBERTa, are combined through an Optuna-driven weighted ensemble strategy to balance model diversity and computational efficiency. For the image modality, PaddleOCR is adopted for text detection and recognition, and a Levenshtein distance-based correction mechanism is introduced to reduce OCR errors, especially in numeric fields. To further enhance cross-modal complementarity, MLDP employs a dynamic weighted fusion strategy together with a decision matrix for modality-level error correction, improving extraction robustness under noisy and complex industrial conditions.Experiments on a real-world multimodal tobacco box label dataset show that MLDP achieves a Micro-F1 of 96.51%, outperforming the text-only baseline (MLDP-T) and image-only baseline (MLDP-V) by 4.34% and 5.73%, respectively. Compared with classical document understanding baselines, including LayoutLMv3, UDOP, and Donut, and recent advanced document parsing or multimodal baselines such as MinerU2.5, PaddleOCR-VL, and Qwen2.5-VL, MLDP achieves the best overall performance and provides a favorable accuracy-efficiency trade-off.In addition, on the DocBank benchmark, MLDP yields 1.18%-1.55% gains on non-textual elements, demonstrating promising generalization ability beyond the target industrial scenario. Nevertheless, the current framework still depends on OCR quality and has been validated mainly on tobacco box labels, so its robustness under more diverse industrial document distributions requires further study. Furthermore, we construct and open-source an industrial tobacco box label multimodal dataset containing 9,719 graphically aligned samples, providing a useful benchmark resource for this task.