Scholarlink

<p>Large Language Models (LLMs) have shown strong potential in code generation for mainstream languages, yet their effectiveness in logic programming remains underexplored. This paper presents a configurable multi-agent framework for benchmarking LLMs in Prolog code generation. The system combines a Code Generator Agent, a deterministic execution layer using SWI-Prolog, and an evaluator based on the LLM-as-a-Judge paradigm. It supports model-agnostic experimentation and evaluates outputs across functional correctness, efficiency, structural properties and evaluator reliability. Experimental results show that leading models achieve high correctness on standard tasks but exhibit significant variability in efficiency, token consumption and code structure. More complex tasks expose clear performance gaps. Additionally, the LLM-as-a-Judge approach shows limited reliability, with low inter-rater agreement. These findings highlight both the capabilities and limitations of LLMs in logic programming and demonstrate the importance of combining execution with LLM-based evaluation.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A Multi-Agent Benchmarking Framework for Evaluating the Performance of Large Language Models in Logic Programming

  • 期刊论文

  • 发表日期：

    2026年7月30日

查看全文
  • Nikolaos Karamousalidis,
  • Petros Kefalas

摘要

Large Language Models (LLMs) have shown strong potential in code generation for mainstream languages, yet their effectiveness in logic programming remains underexplored. This paper presents a configurable multi-agent framework for benchmarking LLMs in Prolog code generation. The system combines a Code Generator Agent, a deterministic execution layer using SWI-Prolog, and an evaluator based on the LLM-as-a-Judge paradigm. It supports model-agnostic experimentation and evaluates outputs across functional correctness, efficiency, structural properties and evaluator reliability. Experimental results show that leading models achieve high correctness on standard tasks but exhibit significant variability in efficiency, token consumption and code structure. More complex tasks expose clear performance gaps. Additionally, the LLM-as-a-Judge approach shows limited reliability, with low inter-rater agreement. These findings highlight both the capabilities and limitations of LLMs in logic programming and demonstrate the importance of combining execution with LLM-based evaluation.