UI-DETR: GUI Component Detection from the System Screen with Transformers

In the development of system screens, rebuilding consistent graphical user interface (GUI) screens is a costly and time-consuming process. The system screen consists of several parts, which we call GUI components in this study. Especially in the redevelopment of large-scale systems, it requires a great deal of effort to manually identify reusable GUI components from a large number of system screens. Therefore, a method to automatically detect GUI components from the system screen is needed to reduce the workload of operators and to speed up their work. In this study, we propose UI-DETR, an improved model of Detection Transformer (DETR), for automatic detection of GUI components. In particular, in UI-DETR, we improve the object query introduced in Anchor DETR. We attempt to improve the detection accuracy by inputting the positional features of GUI components into the object query. Then, evaluation experiments are conducted on the VINS dataset to validate the effectiveness of UI-DETR.