Image Segmentation on Concrete Damage for Augmented Reality Supported Inspection Tasks

The building inspection process is an important step in the maintenance phase of a building. However, human resources are limited, and the inspection process is still predominantly a manual process: Damage is documented on paper, recorded with cameras, and manually entered into databases. Digital tools could improve this process, making it more time and work efficient. Continuous advancements in Machine Learning (ML) and Augmented Reality (AR) can support inspectors during damage documentation tasks, allowing for combined capture and documentation of damage. This paper presents an approach to damage documentation with the Microsoft HoloLens 2 (HL2), a head-mounted optical see-through augmented reality device. To this end, we train and review image segmentation models based on over 5,500 images and deploy the best-performing model on the HL2. The segmentation model is trained to distinguish four concrete damage types. Model inference time is compared between the deployment of the ML model on the HL2 and a compute server. The application is tested on-site for a bridge inspection task, investigating the feasibility of the developed AR-ML-based sub-concept for damage documentation.