A real-time object detection method for electronic screen GUI test systems

Automated testing of GUI elements on electronic screens by machine vision is widely used in smart manufacturing production lines. However, in the complex environment of factories, interference factors such as light reflection, platform vibration, and placement angle make it difficult for testing robots to recognize and locate GUI elements on the equipment’s screen to be tested. In addition, the recognition algorithms based on screen GUI elements currently have problems such as slow detection speed and large model size, which limit the deployment of detection models on test robots. To address these problems, this paper proposes a lightweight model that can overcome the interference of the factory environment based on the YOLOv5 algorithm for recognizing GUI elements. First, a reinforced position and channel attention module (PCA) is proposed to enhance the extraction and fusion ability of weak feature elements affected by interference; then, a three-way feature pyramid network (TWFPN) module is constructed at the intermediate feature layer of the neck structure to provide more accurate localization information. Meanwhile, the loss function of the original algorithm is replaced with SIoU to improve the convergence speed. Finally, a lightweight structure combining the Ghost module and PConv is introduced into the model to reduce the model parameters and improve the detection speed. The method was evaluated on the constructed feature-enhanced GUI (FE_GUI) dataset. The final model shows a 2.7% improvement in average accuracy (mAP@0.5), a 30% reduction in model weight, and an increase in detection speed to 85 FPS compared to the original YOLOv5. The results demonstrate that the method solves the issues of element recognition and model deployment for a vision test system and surpasses existing models in terms of recognition accuracy and detection speed.