The Large Language Model (LLM) has been proven to be capable of performing advanced planning for long-term robot tasks, but existing methods still have shortcomings in solving the illusion problem that is answered by large models. Therefore, how to provide task related knowledge to large models and help them make correct decisions has become a hot research topic. Based on the above issues, we propose a large model planning framework called MRLLM (Mutilmodal and Reflection LLM) based on multimodal knowledge and reflective self refinement. This framework provides knowledge sources and enhances reasoning for LLM through multimodal knowledge regularization and error feedback self refinement iteration, and constrains LLM output through template prompts, which to some extent alleviates the illusion problem of LLM. The experiment shows that our method has better performance compared to traditional methods in the classical planning domain of Blockworld and Housework.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

MRLLM: Multimodal Knowledge and Feedback Based Refinement Assist for Robotic Arm Operations Using Large Language Model Reasoning

  • Zhibin Yang,
  • Zhi Zheng

摘要

The Large Language Model (LLM) has been proven to be capable of performing advanced planning for long-term robot tasks, but existing methods still have shortcomings in solving the illusion problem that is answered by large models. Therefore, how to provide task related knowledge to large models and help them make correct decisions has become a hot research topic. Based on the above issues, we propose a large model planning framework called MRLLM (Mutilmodal and Reflection LLM) based on multimodal knowledge and reflective self refinement. This framework provides knowledge sources and enhances reasoning for LLM through multimodal knowledge regularization and error feedback self refinement iteration, and constrains LLM output through template prompts, which to some extent alleviates the illusion problem of LLM. The experiment shows that our method has better performance compared to traditional methods in the classical planning domain of Blockworld and Housework.