From Screenshots to Process Models: Improving Activity Identification Through Screen Text

The analysis for executing Robotic Process Automation (RPA) projects increasingly relies on monitoring user activities through Robotic Process Mining (RPM) techniques. Traditional approaches capture direct information using loggers that capture UI logs, i.e., sequences of events that include data from (1) the keyboard, (2) the mouse, and (3) the application elements, such as its name, the Excel cell, the clicked button, etc. Although the latter is highly relevant for identifying the activity that is being performed, this information is not accessible in virtualized environments; only screenshot data is available. This limitation necessitates activity identification based on screenshots alone. A significant challenge with this method is its sensitivity to minor interface changes, such as different zoom levels or notifications, which can cause detection failures. To address this, we propose a novel approach that, first, integrates embeddings from both screenshots and screen text obtained through OCR and, second, clusters the UI log events using these combined features to identify the activity. Our results show that this method enhances activity identification, outperforming current state-of-the-art techniques, and demonstrates promising improvements in accuracy and reliability.