What’s Behind the Screen? Unveiling UI Hierarchies in Process-Related UI Logs

Robotic Process Mining (RPM) leverages User Interface (UI) logs as a source of information to analyze the processes to be automated. These UI logs record user interactions with the graphical user interface of an information system during the execution of a process, encapsulating a large amount of data. However, despite the abundance of information captured in these logs, a considerable portion remains on the screen, whose structure is often inaccessible programmatically. This unstructured data poses a significant challenge, particularly the gap in the automatic extraction of UI hierarchies from desktop screenshots, which hinders the application of mining techniques. To address this challenge, this paper presents (1) an approach that employs a multi-model detection system designed specifically for different levels of UI abstraction and (2) an algorithm to reconstruct the UI hierarchy in a tree data structure. The results of this approach demonstrate higher performance in terms of accuracy, recall, and efficiency compared to other existing detection techniques. These findings represent a substantial contribution to the field of automated process analysis, providing a solid foundation for future advances in RPA analysis.