Agent Can Say No: Robot Task Planning by Natural Language Feedback Between Planner and Executor

Recent work has demonstrated the feasibility of employing Large Language Models (LLMs) as robot task planners. However, LLM planners often lack awareness of the executability of actions in the physical world, leading to impractical task instructions. Previous solutions rely on fine-tuning the planner for specific tasks and robot functions, which lack transferability across tasks and incur extra costs. We propose a framework that writes task details and robot operational methods into the system prompts of the LLM planner and the LLM executor, respectively, allowing the executor to provide feedback when the instructions given by the planner are deemed unreasonable, to aid the planner in replanning. This achieves zero-shot generalization across tasks and reduces the need for task-specific learning of the planner, like fine-tuning a policy head. Experimental results demonstrate the effectiveness of our approach in task completion, and the generalization ability to adapt different tasks.