Scholarlink

<p>Large language models (LLMs) have gained widespread interest owing to their ability to process human language and perform tasks on which they have not been explicitly trained. However, we possess only a limited systematic understanding of the chemical capabilities of LLMs, which would be required to improve models and mitigate potential harm. Here we introduce ChemBench, an automated framework for evaluating the chemical knowledge and reasoning abilities of state-of-the-art LLMs against the expertise of chemists. We curated more than 2,700 question–answer pairs, evaluated leading open- and closed-source LLMs and found that the best models, on average, outperformed the best human chemists in our study. However, the models struggle with some basic tasks and provide overconfident predictions. These findings reveal LLMs’ impressive chemical capabilities while emphasizing the need for further research to improve their safety and usefulness. They also suggest adapting chemistry education and show the value of benchmarking frameworks for evaluating LLMs in specific domains.</p><p></p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年5月20日

查看全文
  • Adrian Mirza,
  • Nawaf Alampara,
  • Sreekanth Kunchapu,
  • Martiño Ríos-García,
  • Benedict Emoekabu,
  • Aswanth Krishnan,
  • Tanya Gupta,
  • Mara Schilling-Wilhelmi,
  • Macjonathan Okereke,
  • Anagha Aneesh,
  • Mehrdad Asgari,
  • Juliane Eberhardt,
  • Amir Mohammad Elahi,
  • Hani M. Elbeheiry,
  • María Victoria Gil,
  • Christina Glaubitz,
  • Maximilian Greiner,
  • Caroline T. Holick,
  • Tim Hoffmann,
  • Abdelrahman Ibrahim,
  • Lea C. Klepsch,
  • Yannik Köster,
  • Fabian Alexander Kreth,
  • Jakob Meyer,
  • Santiago Miret,
  • Jan Matthias Peschel,
  • Michael Ringleb,
  • Nicole C. Roesner,
  • Johanna Schreiber,
  • Ulrich S. Schubert,
  • Leanne M. Stafast,
  • A. D. Dinga Wonanke,
  • Michael Pieler,
  • Philippe Schwaller,
  • Kevin Maik Jablonka

摘要

Large language models (LLMs) have gained widespread interest owing to their ability to process human language and perform tasks on which they have not been explicitly trained. However, we possess only a limited systematic understanding of the chemical capabilities of LLMs, which would be required to improve models and mitigate potential harm. Here we introduce ChemBench, an automated framework for evaluating the chemical knowledge and reasoning abilities of state-of-the-art LLMs against the expertise of chemists. We curated more than 2,700 question–answer pairs, evaluated leading open- and closed-source LLMs and found that the best models, on average, outperformed the best human chemists in our study. However, the models struggle with some basic tasks and provide overconfident predictions. These findings reveal LLMs’ impressive chemical capabilities while emphasizing the need for further research to improve their safety and usefulness. They also suggest adapting chemistry education and show the value of benchmarking frameworks for evaluating LLMs in specific domains.