Scholarlink

<p>Despite growing enthusiasm for large language models (LLMs) as coding assistants, there remains limited empirical evidence of their effectiveness in domain-specific contexts such as data science. Existing benchmarks primarily focus on general-purpose programming and do not fully capture the challenges of data science tasks, which require data manipulation, statistical reasoning, algorithmic problem solving, and visualization. They also rarely assess practical dimensions such as first-attempt reliability, output consistency, error recovery, and cost efficiency. To address this gap, we introduce the <i>LLM4DS-Benchmark</i> and conduct a multidimensional empirical evaluation of seven LLMs—Gemini 2.5 Pro, Claude Sonnet 4.5, o3-mini, GPT-4.1, GPT-4o, Qwen3-Coder, and Perplexity Sonar—on 814 Python data science coding problems from StrataScratch platform, spanning Analytical, Algorithm, and Visualization tasks across three difficulty levels. Each problem received up to three attempts under a branching protocol that separates independent attempts from feedback-guided retries, enabling analysis of correctness, Pass@1, retry recovery, output consistency, execution behavior, visualization quality, code similarity, token usage, and cost per solved problem. Results show that Gemini 2.5 Pro achieved the highest overall success rate (81.3%) and Pass@1 (62.2%), but at a median cost of \$0.10740 per solved problem—316 times higher than Qwen3-Coder (\$0.00034). Across models, retries improved performance by 19–24 percentage points, with feedback resolving 20–31% of initial failures. Output consistency varied significantly across identical prompts, particularly for Analytical tasks. Model rankings also shifted by task type and evaluation dimension, with no single model dominating across all categories. Instead, a Pareto-optimal set—Qwen3-Coder, GPT-4.1, o3-mini, Claude Sonnet 4.5, and Gemini 2.5 Pro—emerged, reflecting trade-offs among accuracy, cost, and reliability. These findings highlight the need for multidimensional, task-aware benchmarking and suggest that model selection for data science coding should be guided by task characteristics and practical constraints rather than aggregate success rate alone.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Empirical benchmarking of large language models for data science coding: a multidimensional evaluation

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年7月13日

查看全文
  • Santhosh Anitha Boominathan,
  • Sai Sanjna Chintakunta,
  • Everton Guimaraes,
  • Kailasam Satyamurthy,
  • Nathalia Nascimento

摘要

Despite growing enthusiasm for large language models (LLMs) as coding assistants, there remains limited empirical evidence of their effectiveness in domain-specific contexts such as data science. Existing benchmarks primarily focus on general-purpose programming and do not fully capture the challenges of data science tasks, which require data manipulation, statistical reasoning, algorithmic problem solving, and visualization. They also rarely assess practical dimensions such as first-attempt reliability, output consistency, error recovery, and cost efficiency. To address this gap, we introduce the LLM4DS-Benchmark and conduct a multidimensional empirical evaluation of seven LLMs—Gemini 2.5 Pro, Claude Sonnet 4.5, o3-mini, GPT-4.1, GPT-4o, Qwen3-Coder, and Perplexity Sonar—on 814 Python data science coding problems from StrataScratch platform, spanning Analytical, Algorithm, and Visualization tasks across three difficulty levels. Each problem received up to three attempts under a branching protocol that separates independent attempts from feedback-guided retries, enabling analysis of correctness, Pass@1, retry recovery, output consistency, execution behavior, visualization quality, code similarity, token usage, and cost per solved problem. Results show that Gemini 2.5 Pro achieved the highest overall success rate (81.3%) and Pass@1 (62.2%), but at a median cost of \$0.10740 per solved problem—316 times higher than Qwen3-Coder (\$0.00034). Across models, retries improved performance by 19–24 percentage points, with feedback resolving 20–31% of initial failures. Output consistency varied significantly across identical prompts, particularly for Analytical tasks. Model rankings also shifted by task type and evaluation dimension, with no single model dominating across all categories. Instead, a Pareto-optimal set—Qwen3-Coder, GPT-4.1, o3-mini, Claude Sonnet 4.5, and Gemini 2.5 Pro—emerged, reflecting trade-offs among accuracy, cost, and reliability. These findings highlight the need for multidimensional, task-aware benchmarking and suggest that model selection for data science coding should be guided by task characteristics and practical constraints rather than aggregate success rate alone.