Traditional Human-Swarm Interaction (HSI) methods often lack intuitive real-time adaptive interfaces, making decision making slower and increasing cognitive load while limiting command flexibility. To solve this, we present SwarmChat, a context-aware, multimodal interaction system powered by Large Language Models (LLMs). SwarmChat enables users to issue natural language commands to robotic swarms using multiple modalities, such as text, voice, or teleoperation. The system integrates four LLM-based modules: Context Generator, Intent Recognition, Task Planner, and Modality Selector. These modules collaboratively generate context from keywords, detect user intent, adapt commands based on real-time robot state, and suggest optimal communication modalities. Its three-layer architecture offers a dynamic interface with both fixed and customisable command options, supporting flexible control while optimising cognitive effort. The preliminary evaluation also shows that the SwarmChat’s LLM modules provide accurate context interpretation, relevant intent recognition, and effective command delivery, achieving high user satisfaction.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

SwarmChat: An LLM-Based, Context-Aware Multimodal Interaction System for Robotic Swarms

  • Ettilla Mohiuddin Eumi,
  • Hussein Abbass,
  • Nadine Marcus

摘要

Traditional Human-Swarm Interaction (HSI) methods often lack intuitive real-time adaptive interfaces, making decision making slower and increasing cognitive load while limiting command flexibility. To solve this, we present SwarmChat, a context-aware, multimodal interaction system powered by Large Language Models (LLMs). SwarmChat enables users to issue natural language commands to robotic swarms using multiple modalities, such as text, voice, or teleoperation. The system integrates four LLM-based modules: Context Generator, Intent Recognition, Task Planner, and Modality Selector. These modules collaboratively generate context from keywords, detect user intent, adapt commands based on real-time robot state, and suggest optimal communication modalities. Its three-layer architecture offers a dynamic interface with both fixed and customisable command options, supporting flexible control while optimising cognitive effort. The preliminary evaluation also shows that the SwarmChat’s LLM modules provide accurate context interpretation, relevant intent recognition, and effective command delivery, achieving high user satisfaction.