Lumped Approach to Recognize Types of Construction Defect from Text with Hand-Drawn Circles

This study aims to improve the performance of optical character recognition (OCR), particularly in identifying printed Korean text marked by hand-drawn circles from images of construction defect tags. Despite advancements in mobile technologies, marking text on paper remains a prevalent practice. The typical approach for recognition in this context is to first detect the circles from the images and then identify the text entity within the region using OCR. Numerous OCR models have been developed to automatically identify various text types, but even a competition-winning multilingual model by Baek et al. does not perform well in recognizing circled Korean text, yielding a weighted F1 score of just 69%. The core idea of the lumped approach proposed in this study is to recognize circles and named entities as one instance. For this purpose, the YOLOv5 is fine tuned to detect 65 types of named entity overlapped with hand-drawn circles and yields a weighted F1 score of 94%, 25% higher than a typical approach using YOLOv5 for circle detection and a model by Baek et al. for subsequent OCR. This work thereby introduces a novel approach for developing advanced text information extraction methods and processing paper-based marked text in the construction industry.