Reformulating Key-Information Extraction as Next Sentence Prediction for Hierarchical Data

We present a reformulation of the Key-Information Extraction (KIE) problem from document images, as a Next-Sentence Prediction (NSP) task for identifying information in hierarchically structured data. KIE implemented as a Key-Value extraction task, is limited to one-to-one (single key mapping to single value) information extraction and thus does not apply to hierarchical information e.g. information present in complex semi-structured or unstructured tables. The Visual-Question-Answering (VQA) approach tries to solve information extraction from such semi-structured formats, but use visual information extraction backbone architectures along with heavy language models. In the proposed work, we use only a backbone language feature extractor for semantic entity extraction. Unlike, the four entity types in FUNSD (‘question’, ‘answer’, ‘header’ and ‘other’), for semi-structured tabular information we define additional classes that define hierarchical elements, like column-header, table-footer, cells, merged-cell, table-summary etc. For these additional entities, we define hierarchical relations like a tuple of entities {table-header entity, column-header entity, row-header entity} that point to the unique entity referred as a value-entity. We treat tuple-entity and value-entity as two sentences and formulate the task of finding how likely is the value-entity to follow the tuple-entity. Empirically, we show that the proposed method, called as, Tuple-Value Identification (TVI), can exhaustively identify all the information in the hierarchical structures. Additionally, TVI also opens up for the potential use for Table Structure Recognition (TSR) for scanned documents in bank statements or medical bills, where the narration columns span multi-lines and is challenging for existing TSR systems.