Segmentation of Graphical User Interface Elements Based on Topological Decomposition for GUI Testing Tasks

The paper addresses the issue of automating the testing process for graphical interfaces. It is shown that one of the main tasks in this area is segmentation of screen elements with further construction its internal structure. Emphasis is placed on the stability of the proposed method, regardless of changes in interface layout or the operating system employed. Our approach is based on decomposing a window screenshot into specific components that correspond to the elements of the original window and their hierarchy. We demonstrate the method’s resilience to the resizing of objects within windows. The research was conducted using interface element segmentation for the QGIS geographic information system on both Windows and Ubuntu operating systems. Experimental results revealed high levels of accuracy, ranging from 94 to 100 percent, in extracting segmented areas within QGIS windows.