ReGA: Reasoning and Grounding Decoupled GUI Navigation Agents
摘要
The rise of multimodal large language models (LLMs) has led to a surge of GUI navigation agents, which aim to automate GUI operations following human instructions. Implementing such an agent implies being able to reason action intents at each step from instructions and to ground them on the GUI environment. Previous work entangle the reasoning and grounding processes by direct coordinate prediction, ignoring their distinct requirements of information at different abstraction levels. In this paper, we propose Reasoning and Grounding GUI Automation agent (ReGA), an multimodal LLM-based agent that decouples these two processes. We incorporate additional tokens into the existing vocabulary of the LLM as the representation of intended location, whose hidden embeddings are further decoded into coordinates. In this way, we transfer the grounding function from language model to the location decoder. Upon this design, we further enhance the grounding capability of location decoder by fusing low-level vision features, and improve the reasoning performance by high-level task-generic plans named Generic Interaction Plans. Experimental results shows that ReGA achieves state-of-the-art performance on representative GUI navigation benchmarks.