Pre-training for Document Structure Extraction with Lightweight Model Architecture

Document Structured Extraction (DSE) aims at converting raw documents into machine-readable structured text, which is a key technology for knowledge base construction and retrieval-augmented generation. However, existing multimodal large language models (MLLMs) generally lack high-quality training data and customized pre-training tasks, resulting in sub-optimal performance when parsing documents with diverse layout formats. To address this challenge, we design six novel pre-training tasks, including Optical Character Recognition (OCR), layout analysis, text2box, box2text, image caption, and figure/table caption, which boost the document understanding by guiding the model to focus attention on the document-level region. In addition, we adopt a shape adaptive cropping method that supports processing images with arbitrary resolutions, thereby avoiding deformation and blurring issues caused by image compression. Furthermore, a cross-attention based visual token compression method is proposed, which compresses the number of visual tokens to 384 per image, thus effectively improving the inference efficiency of the model without compromising performance. By leveraging novel data construction strategy, tens of millions of training samples have been generated, laying a foundation for effective model training. Compared with the current state-of-the-art algorithm, the trained multimodal large language model achieves an approximately 7% performance improvement in the F1 score. The experimental results verify the superiority of our method.