Recent innovations in large-scale multimodal language models (MLLM) have significantly improved the anchoring of graphical interfaces, a process that involves linking natural language instructions to specific interface elements. Performance tests such as ScreenSpot Pro have demonstrated the level of difficulty of anchoring in high-resolution professional scenarios. In such situations, small elements and complex layouts make it difficult for the model to function properly. However, currently, only the evaluation of a single image is taken into account, and the ability of MLLM to reason from multiple visual inputs is ignored. In this work, we propose the dual image prompting for GUI grounding approach. The model receives a template image that designates the target area and a full screen image that represents the candidate region. It must then find the template in the screen. To do this, it combines conventional template matching with semantic reasoning and multi-image alignment.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Dual-Image-Guided Grounding of User Interfaces with MLLMs

  • Mahaman Sanoussi Yahaya Alassan,
  • El Hassane Ettifouri,
  • Walid Dahhane

摘要

Recent innovations in large-scale multimodal language models (MLLM) have significantly improved the anchoring of graphical interfaces, a process that involves linking natural language instructions to specific interface elements. Performance tests such as ScreenSpot Pro have demonstrated the level of difficulty of anchoring in high-resolution professional scenarios. In such situations, small elements and complex layouts make it difficult for the model to function properly. However, currently, only the evaluation of a single image is taken into account, and the ability of MLLM to reason from multiple visual inputs is ignored. In this work, we propose the dual image prompting for GUI grounding approach. The model receives a template image that designates the target area and a full screen image that represents the candidate region. It must then find the template in the screen. To do this, it combines conventional template matching with semantic reasoning and multi-image alignment.