Scholarlink

<p>Code generation has emerged as a valuable technique for accelerating robotic software development. However, unlike general-purpose programming, generating code for Cyber-Physical Systems (CPS) requires navigating complex, domain-specific frameworks like the Robot Operating System (ROS). Although Large Language Models (LLMs) excel in general-purpose code generation, their capability to synthesize executable, middleware-compliant ROS code remains poorly understood. This task demands that LLMs not only solve robotic tasks but also generate code that seamlessly integrates within the distributed ROS framework. To address this gap, we construct ROSDevEval, a specialized benchmark comprising 240 real-world ROS programming tasks, and conduct a comprehensive empirical study to evaluate four state-of-the-art LLMs alongside a specialized code assistant (GitHub Copilot). Our results reveal a severe domain capability gap: while state-of-the-art LLMs achieve up to 88.4% Pass@1 on general-purpose C++, their performance suffers a precipitous relative decline ranging from 78.8% to 85.4% in robotic contexts, and even the specialized assistant struggles to exceed a 32.5% success rate. Through a rigorous open-coding analysis, we systematically categorize LLMs’ generation failures into two distinct dimensions. In <i>robotic task resolution</i>, LLMs struggle to apply physical constraints and produce functional logic. In <i>ROS framework integration</i>, they systematically violate strict middleware contracts required for system orchestration. Furthermore, our root cause analysis explicitly decouples these surface-level failures from their underlying etiology, revealing a profound dual cognitive deficit: LLMs lack the long-horizon procedural decomposition capability to translate abstract tasks into executable sequences, and they fail to grasp the rigid architectural topology required for distributed system integration. These findings delineate current limitations and provide actionable, architecture-aware implications for advancing the next generation of robotic foundation models. Code and datasets are available at <a href="https://github.com/zyx065/rosdeveval">https://github.com/zyx065/rosdeveval</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Evaluating LLMs in ROS robotic software code generation

  • 期刊论文

  • 发表日期：

    2026年8月5日

查看全文
  • Yuxin Zhao,
  • Xinjun Mao,
  • Tanghaoran Zhang,
  • Tun Li,
  • Zhiqun Xiao

摘要

Code generation has emerged as a valuable technique for accelerating robotic software development. However, unlike general-purpose programming, generating code for Cyber-Physical Systems (CPS) requires navigating complex, domain-specific frameworks like the Robot Operating System (ROS). Although Large Language Models (LLMs) excel in general-purpose code generation, their capability to synthesize executable, middleware-compliant ROS code remains poorly understood. This task demands that LLMs not only solve robotic tasks but also generate code that seamlessly integrates within the distributed ROS framework. To address this gap, we construct ROSDevEval, a specialized benchmark comprising 240 real-world ROS programming tasks, and conduct a comprehensive empirical study to evaluate four state-of-the-art LLMs alongside a specialized code assistant (GitHub Copilot). Our results reveal a severe domain capability gap: while state-of-the-art LLMs achieve up to 88.4% Pass@1 on general-purpose C++, their performance suffers a precipitous relative decline ranging from 78.8% to 85.4% in robotic contexts, and even the specialized assistant struggles to exceed a 32.5% success rate. Through a rigorous open-coding analysis, we systematically categorize LLMs’ generation failures into two distinct dimensions. In robotic task resolution, LLMs struggle to apply physical constraints and produce functional logic. In ROS framework integration, they systematically violate strict middleware contracts required for system orchestration. Furthermore, our root cause analysis explicitly decouples these surface-level failures from their underlying etiology, revealing a profound dual cognitive deficit: LLMs lack the long-horizon procedural decomposition capability to translate abstract tasks into executable sequences, and they fail to grasp the rigid architectural topology required for distributed system integration. These findings delineate current limitations and provide actionable, architecture-aware implications for advancing the next generation of robotic foundation models. Code and datasets are available at https://github.com/zyx065/rosdeveval.