Large language model based collaborative robot system for daily task assistance

The recent advancements in natural language processing models have spurred extensive research into robots that interact with and execute tasks based on human natural language commands. This study introduces a robot system that interprets human commands using a large language model (LLM) to understand the intent and relay it to a robot for task execution. The robot system comprises three key components: natural language command inference, vision, and learning-based control systems. The command inference system utilizes the GPT-4 model to interpret natural language commands and issue task execution orders to the robot. Based on the transformer model OWL-ViT, the vision system recognizes objects related to user commands and communicates the findings to the inference system. Upon understanding the user’s commands, a UR3 robot executes the tasks using the Diffusion Policy. The robot system is integrated using ROS (robot operating system) and validated through experiments. Through the experiment, a success rate of 65% was achieved.