Scholarlink

Abstract <p>This study examines the applicability of large language models for automated data extraction from soil-science research papers, using a corpus of Russian<i>-</i>language articles from the journal <i>Pochvovedenie</i>/Eurasian Soil Science (2019–2020). Extraction quality was evaluated using GPT-4 Turbo’s answers to a set of 58 questions covering characteristics of research objects and conditions, soil types, measurement methods, and measurement results. Automated and expert-based approaches to assessing the correctness of the model’s answers were compared. The aim was to evaluate how well large language models (LLMs) can extract and verify data from scientific articles. The baseline model’s outputs were compared with expert gold-standard annotations, deterministic metrics based on lexical overlap, and the scores of several alternative language models acting as independent “judges.” The integrated F1 score for individual questions was at least 0.73, while the mean values across thematic question groups reached 0.91–0.99. The correlation between the most consistent judge model Gemini 2.5 Flash and expert ratings reached 0.8 and explained about 60% of the variance in expert assessments, whereas classical <i>n-</i>gram metrics were significantly less informative. These results indicate that LLMs are promising as the basis of a scalable pipeline for the automated construction of soil property databases and reducing manual effort in the analysis of scientific texts.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Soil Data Extraction from Scientific Publications Using Large Language Models

  • 期刊论文

  • 发表日期：

    2026年4月24日

查看全文
  • M. A. Soldatkina,
  • V. V. Klyueva,
  • M. V. Timofeeva,
  • M. A. Kochneva,
  • D. S. Fomin

摘要

Abstract

This study examines the applicability of large language models for automated data extraction from soil-science research papers, using a corpus of Russian-language articles from the journal Pochvovedenie/Eurasian Soil Science (2019–2020). Extraction quality was evaluated using GPT-4 Turbo’s answers to a set of 58 questions covering characteristics of research objects and conditions, soil types, measurement methods, and measurement results. Automated and expert-based approaches to assessing the correctness of the model’s answers were compared. The aim was to evaluate how well large language models (LLMs) can extract and verify data from scientific articles. The baseline model’s outputs were compared with expert gold-standard annotations, deterministic metrics based on lexical overlap, and the scores of several alternative language models acting as independent “judges.” The integrated F1 score for individual questions was at least 0.73, while the mean values across thematic question groups reached 0.91–0.99. The correlation between the most consistent judge model Gemini 2.5 Flash and expert ratings reached 0.8 and explained about 60% of the variance in expert assessments, whereas classical n-gram metrics were significantly less informative. These results indicate that LLMs are promising as the basis of a scalable pipeline for the automated construction of soil property databases and reducing manual effort in the analysis of scientific texts.