In recent years, detecting visual components in graphical user interfaces (GUIs) has become increasingly relevant for modern applications that rely on automatic UI understanding. Unlike natural images, UI layouts present unique challenges, such as small, densely packed elements and high visual variability, which can affect standard object detectors. To explore this problem, we conduct a comparative analysis of six recent YOLO (You Only Look Once) versions, ranging from YOLOv5 to YOLOv12, for the task of detecting graphical components in user interfaces. We target three major platforms (mobile, web, desktop) and use representative datasets: VINS (mobile), GENGUI (web), and UICVD (desktop). All annotations were standardized to include the same three core UI classes: Text, Button, and Icon. Each YOLO model was trained and evaluated separately on each dataset, using the lightweight nano (n), tiny (t), and small (s) variants to analyze the trade-off between accuracy and efficiency. The results show noticeable performance differences across versions and interface types. YOLOv9s and YOLO11s performed best in complex scenarios, while YOLOv5s excelled on mobile data. Button was the easiest class to detect, Text remained stable, and Icon proved the most challenging. Several models, including YOLOv9s and YOLO11s, achieved mean Average Precisions (mAP) above 93% across desktop, web, and mobile interfaces, confirming that detection quality depends strongly on both the architecture and the interface type.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

UI Component Detection in Graphical Interfaces: A YOLO-Based Comparative Analysis

  • Mădălina Dicu

摘要

In recent years, detecting visual components in graphical user interfaces (GUIs) has become increasingly relevant for modern applications that rely on automatic UI understanding. Unlike natural images, UI layouts present unique challenges, such as small, densely packed elements and high visual variability, which can affect standard object detectors. To explore this problem, we conduct a comparative analysis of six recent YOLO (You Only Look Once) versions, ranging from YOLOv5 to YOLOv12, for the task of detecting graphical components in user interfaces. We target three major platforms (mobile, web, desktop) and use representative datasets: VINS (mobile), GENGUI (web), and UICVD (desktop). All annotations were standardized to include the same three core UI classes: Text, Button, and Icon. Each YOLO model was trained and evaluated separately on each dataset, using the lightweight nano (n), tiny (t), and small (s) variants to analyze the trade-off between accuracy and efficiency. The results show noticeable performance differences across versions and interface types. YOLOv9s and YOLO11s performed best in complex scenarios, while YOLOv5s excelled on mobile data. Button was the easiest class to detect, Text remained stable, and Icon proved the most challenging. Several models, including YOLOv9s and YOLO11s, achieved mean Average Precisions (mAP) above 93% across desktop, web, and mobile interfaces, confirming that detection quality depends strongly on both the architecture and the interface type.