Scholarlink

<p>Document retrieval plays a vital role in the financial domain, particularly in investment decision-making, risk assessment, and market regulation. Financial documents often contain complex multimodal data, including text, tables, and charts, and there are still errors in the parsing and question answering of multimodal financial documents. Firstly, to address the insufficient semantic relevance between the responses of large language models and corresponding queries in complex Chinese financial long-text scenarios, we propose a bidirectional and autoregressive transformers (BART) based named entity recognition (NER) approach combined with a prompt-guided strategy. By explicitly capturing and modeling entity and relation information, the model improves the accuracy of entity recognition and semantic understanding, while also enhancing its logical reasoning and interpretability. Secondly, to address the issue of parsing errors in multimodal tabular data, we introduce a financial domain-specific table structure recognition model that improves the accuracy of table parsing, significantly reduces GPU memory consumption, and ultimately enhances the answer accuracy of the multimodal retrieval-augmented generation (RAG) system. In addition, to address the lack of high-quality named entity annotation data in the financial domain, we constructed a Chinese financial multimodal NER dataset to support multimodal RAG models. Experimental results demonstrate the effectiveness of our approach in enhancing both table parsing performance and answer generation for multimodal financial documents. Our table structure recognition method requires only 1.5 GB of GPU memory, and the RAG approach achieves an Answer Correctness score of 48% on Ragas. More information and access to our code are available at our GitHub repository: <a href="https://github.com/LeKit089/NER_MultimodalRAG">https://github.com/LeKit089/NER_MultimodalRAG</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Multimodal RAG for financial documents: BART-based financial named entity recognition and attention-based table parsing for financial QA enhancement

  • 期刊论文

  • 发表日期：

    2026年4月13日

查看全文
  • Ying Ni,
  • Xiaoli Wang,
  • Hanghang Peng,
  • Pengle Zhang,
  • Xin Liu,
  • Bin Sheng,
  • Jin Huang

摘要

Document retrieval plays a vital role in the financial domain, particularly in investment decision-making, risk assessment, and market regulation. Financial documents often contain complex multimodal data, including text, tables, and charts, and there are still errors in the parsing and question answering of multimodal financial documents. Firstly, to address the insufficient semantic relevance between the responses of large language models and corresponding queries in complex Chinese financial long-text scenarios, we propose a bidirectional and autoregressive transformers (BART) based named entity recognition (NER) approach combined with a prompt-guided strategy. By explicitly capturing and modeling entity and relation information, the model improves the accuracy of entity recognition and semantic understanding, while also enhancing its logical reasoning and interpretability. Secondly, to address the issue of parsing errors in multimodal tabular data, we introduce a financial domain-specific table structure recognition model that improves the accuracy of table parsing, significantly reduces GPU memory consumption, and ultimately enhances the answer accuracy of the multimodal retrieval-augmented generation (RAG) system. In addition, to address the lack of high-quality named entity annotation data in the financial domain, we constructed a Chinese financial multimodal NER dataset to support multimodal RAG models. Experimental results demonstrate the effectiveness of our approach in enhancing both table parsing performance and answer generation for multimodal financial documents. Our table structure recognition method requires only 1.5 GB of GPU memory, and the RAG approach achieves an Answer Correctness score of 48% on Ragas. More information and access to our code are available at our GitHub repository: https://github.com/LeKit089/NER_MultimodalRAG.