HM-Transformer: Hierarchical Multi-modal Transformer for Long Document Image Understanding

Transformer plays a massive role in document image understanding. However, it has difficulty handling text in long document images due to the increasing quadratic complexity along the text length. To solve this problem, we propose the hierarchical multi-modal transformer (HM-Transformer) for long document image understanding. HM-Transformer hierarchically models document images. It learns the block representation first and then the document image representation. Specifically, we first use the pre-trained model LayoutLMv3 to generate a block representation containing image, layout, and text information. Then we utilize a document multimodal transformer to model the global multimodal document text representations and image representations. We then reconstruct the in-block representation in the block multimodal Transformer with the help of global multimodal document text and image representation. Finally we conduct different operations depending on the downstream tasks. Experimental results show that HM-Transformer achieves new state-of-the-art performance on two downstream document image understanding tasks, including FUNSD and CORD. The code and models are publicly available at https://github.com/dx233333/picture_project .