Multimodal Classification Algorithm for Turkish Document Archiving: Improving Digital Document Storage by Unifying Image and Text-Based Classifiers

Common usage and widespread availability of image-taking devices have made digital archiving of government and private sector documents a cheap and feasible alternative to conventional document storage, despite its own disadvantages. Indeed, storing digital documents either directly as images or converting them to texts via some OCR libraries [17] has its own problems, including but not limited to image noises, brightness problems, incomplete sentences and punctuations, as well as non-standard image resolutions. The combination of any of these problems makes it difficult to classify such digital documents without human intervention. To overcome this problem, this paper proposes a multimodal classification algorithm for classifying such documents by unifying features from image-based and text-based classifiers. While the proposed model utilizes a pre-trained network of VGG-16 [37] with transfer learning and 1.6M trainable parameters for the image classification part, it uses the EasyOCR library [17] with word embedding obtained from FastText [18] for text classification. By unifying the features from both models with simple concatenation and reclassifying with XGBoost [8], the algorithm achieves a classification performance of over 96%.