Scholarlink

<p>Large language models (LLMs) can generate impressive data visualizations from simple requests, yet their accuracy remains underexplored. Here we present a benchmark of 293 coding tasks derived from 39 studies across 7 biomedical research areas, including biomarkers, integrative analysis, genomic profiling, molecular characterization, therapeutic response, translational research and pan-cancer analysis. Benchmarking eight proprietary and eight open-source LLMs under various prompting strategies reveals an overall accuracy below 40%. This low accuracy raises serious concerns about the risk of propagating incorrect scientific findings when blindly relying on AI-generated analyses. Therefore, we develop an AI agent that begins with and iteratively refines an analysis plan before generating code, achieving 74% accuracy. We embody this insight in a platform that enables users to codevelop analysis plans with LLMs and execute them within an integrated environment. In a user study with five medical researchers, the platform enabled users to complete over 80% of the analysis code for three studies.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Making large language models reliable data science programming copilots for biomedical research

  • 期刊论文

  • 发表日期：

    2026年1月22日

查看全文
  • Zifeng Wang,
  • Benjamin Danek,
  • Ziwei Yang,
  • Zheng Chen,
  • Jimeng Sun

摘要

Large language models (LLMs) can generate impressive data visualizations from simple requests, yet their accuracy remains underexplored. Here we present a benchmark of 293 coding tasks derived from 39 studies across 7 biomedical research areas, including biomarkers, integrative analysis, genomic profiling, molecular characterization, therapeutic response, translational research and pan-cancer analysis. Benchmarking eight proprietary and eight open-source LLMs under various prompting strategies reveals an overall accuracy below 40%. This low accuracy raises serious concerns about the risk of propagating incorrect scientific findings when blindly relying on AI-generated analyses. Therefore, we develop an AI agent that begins with and iteratively refines an analysis plan before generating code, achieving 74% accuracy. We embody this insight in a platform that enables users to codevelop analysis plans with LLMs and execute them within an integrated environment. In a user study with five medical researchers, the platform enabled users to complete over 80% of the analysis code for three studies.