Complexly structured data present in documents and web content pose significant challenges for accurate MLLM reasoning. Although MLLMs have advanced substantially, they continue to struggle with intricate data formats such as nested tables and multi-dimensional charts, often leading to hallucinations. This paper explores the capabilities of LLMs and MLLMs in understanding and answering questions from complex data found in PDF documents by leveraging a pre-processing pipeline consisting of industrial and open-source tools. Our results showcase that incorporating RAG and pre-processing tools enables MLLMs to achieve approximately 5% higher accuracy than direct multi-modal inference, while also reducing their overall cost since only text-mode is used. Our code is available at: https://github.com/manulife-ai/financialqa .

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Understanding Multi-Structured Documents via LLMs’

  • Shivani Upadhyay,
  • Messiah Ataey,
  • Syed Shariyar Murtaza,
  • Yifan Nie,
  • Anirudh Aggarwal,
  • Jimmy Lin

摘要

Complexly structured data present in documents and web content pose significant challenges for accurate MLLM reasoning. Although MLLMs have advanced substantially, they continue to struggle with intricate data formats such as nested tables and multi-dimensional charts, often leading to hallucinations. This paper explores the capabilities of LLMs and MLLMs in understanding and answering questions from complex data found in PDF documents by leveraging a pre-processing pipeline consisting of industrial and open-source tools. Our results showcase that incorporating RAG and pre-processing tools enables MLLMs to achieve approximately 5% higher accuracy than direct multi-modal inference, while also reducing their overall cost since only text-mode is used. Our code is available at: https://github.com/manulife-ai/financialqa .