OpenPSS: An Open Page Stream Segmentation Benchmark

In recent years, an increasing number of companies and institutions have begun the process of digitizing their physical records to promote digital access and searchability of their collections. For cost-efficiency, documents are often scanned in consecutively, resulting in large PDF files consisting of many documents. Although cost-effective, this practice can be harmful for searchability when these concatenated documents are used to build a search engine. The task of Page Stream Segmentation is concerned with recovering the original document boundaries through the analysis of the text and/or images of these PDF files. Currently, many of the approaches to solving this problem make use of machine learning techniques that require significant amounts of training data. However, due to the sometimes sensitive nature of the data, few large datasets exist, and there is a lack of agreed-upon metrics to measure system performance. In an effort to resolve these issues and provide a comprehensive overview of the state of the field, we constructed the OpenPSS benchmark, consisting of two large public datasets and a comprehensive study of various types of approaches, evaluated using multiple evaluation metrics. The datasets originated from several Dutch government institutions, cover a heterogeneous set of topics, and total roughly 141 thousand pages from around 32 thousand documents. The experimental results show that ensemble methods using both the text and image representations of pages are superior to uni-modal methods, and that image-based neural methods are not as robust as text models when evaluated on out-of-distribution data.