Scholarlink

Purpose <p>The American Society of Breast Surgeons (ASBrS) receives an increasing number of abstracts for its annual meeting, resulting in increasing time and effort for peer-review. We aimed to compare artificial intelligence (AI) large language models (LLMs) to human-generated scores to explore if AI-review is a plausible abstract screening process.</p> Methods <p>Abstracts published from the 2025 ASBrS meeting were assessed with three LLMs: OpenAI’s GPT4-o (GPT), Meta’s Llama-3.1-405b (Llama) and DeepSeek’s DeepSeek-V3 (DeepSeek). Abstracts were analyzed using zero shot (ZS), which included the 2025 ASBrS scoring rubric and few shot (FS) with examples of high- and low-scoring abstracts. Large language model scores were compared with the committee’s scores. Model inference was evaluated by using Spearman rank correlation, quartile accuracy, and mean absolute error; 95% CIs were calculated by using bootstrap resampling.</p> Results <p>A total of 378 published accepted abstracts were included. Human-generated median abstract score was 21.2 (rubric ranged 0–35). Median scores were higher amongst all LLMs for both the ZS and FS approaches (27.0–29.7, <i>p</i> &lt; 0.001). Spearman values with the ZS approach for GPT, DeepSeek, and Llama were 0.24, 0.30, and 0.31, respectively, and for the FS approach were 0.37, 0.35, and 0.27, respectively (<i>p</i> &lt; 0.001 for all). DeepSeek ranked the first quartile correctly with 53.5% accuracy while Llama ranked the fourth quartile correctly with 42.9% accuracy. The FS approach <i>modestly improved</i> correlation and first quartile accuracy, with greatest improvement seen in GPT.</p> Conclusions <p>Large language models showed statistically significant rank correlation with human ASBrS abstract ranking. Large language models may be useful as a preliminary screening tool to prioritize reviewer effort toward higher-quality abstracts.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Streamlining the Abstract Selection Process for the American Society of Breast Surgeons’ Annual Meeting by Utilizing Large Language Models

  • 期刊论文

  • 发表日期：

    2026年7月6日

查看全文
  • Lauren N. Cohen,
  • Drake Seibert,
  • Thomas Kosten,
  • Nathan Wolfrath,
  • Zahraa Al-Hilli,
  • Olga Kantor,
  • Puneet Singh,
  • Mediget Teshome,
  • Sarah Blair,
  • Tina J. Hieken,
  • Lee G. Wilke,
  • Anai Kothari,
  • Chandler S. Cortina

摘要

Purpose

The American Society of Breast Surgeons (ASBrS) receives an increasing number of abstracts for its annual meeting, resulting in increasing time and effort for peer-review. We aimed to compare artificial intelligence (AI) large language models (LLMs) to human-generated scores to explore if AI-review is a plausible abstract screening process.

Methods

Abstracts published from the 2025 ASBrS meeting were assessed with three LLMs: OpenAI’s GPT4-o (GPT), Meta’s Llama-3.1-405b (Llama) and DeepSeek’s DeepSeek-V3 (DeepSeek). Abstracts were analyzed using zero shot (ZS), which included the 2025 ASBrS scoring rubric and few shot (FS) with examples of high- and low-scoring abstracts. Large language model scores were compared with the committee’s scores. Model inference was evaluated by using Spearman rank correlation, quartile accuracy, and mean absolute error; 95% CIs were calculated by using bootstrap resampling.

Results

A total of 378 published accepted abstracts were included. Human-generated median abstract score was 21.2 (rubric ranged 0–35). Median scores were higher amongst all LLMs for both the ZS and FS approaches (27.0–29.7, p < 0.001). Spearman values with the ZS approach for GPT, DeepSeek, and Llama were 0.24, 0.30, and 0.31, respectively, and for the FS approach were 0.37, 0.35, and 0.27, respectively (p < 0.001 for all). DeepSeek ranked the first quartile correctly with 53.5% accuracy while Llama ranked the fourth quartile correctly with 42.9% accuracy. The FS approach modestly improved correlation and first quartile accuracy, with greatest improvement seen in GPT.

Conclusions

Large language models showed statistically significant rank correlation with human ASBrS abstract ranking. Large language models may be useful as a preliminary screening tool to prioritize reviewer effort toward higher-quality abstracts.