Scholarlink

Evaluating open-ended responses using Large Language Models (LLMs) remains challenging in natural language processing due to their variability and subjectivity. Traditional rubric-based assessments aim to provide structure but struggle with inconsistencies in scoring, particularly in subjective criteria like creativity and argumentation. This study builds on existing rubric-based approaches by introducing two new evaluation methods for LLM-based scoring. The first method refines scoring by assessing each level of performance individually and utilizing a Deep Neural Network (DNN) for calibration. The second one enforces a structured scoring shape with a single-peak distribution while keeping the DNN as a calibration method. A systematic evaluation using the 70-billion-parameter versions of Llama 3.1 and Llama 3.3 reveals that the first method significantly outperforms the classical approach, achieving a kappa score of 0.634 compared to 0.2233. However, the second method did not improve performance as expected.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

On Using Large Language Models for Rubric-Based Open Question Evaluation

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Mark Tič,
  • Miguel Arevalillo-Herráez,
  • Yuyan Wu,
  • Dejan Lavbič

摘要

Evaluating open-ended responses using Large Language Models (LLMs) remains challenging in natural language processing due to their variability and subjectivity. Traditional rubric-based assessments aim to provide structure but struggle with inconsistencies in scoring, particularly in subjective criteria like creativity and argumentation. This study builds on existing rubric-based approaches by introducing two new evaluation methods for LLM-based scoring. The first method refines scoring by assessing each level of performance individually and utilizing a Deep Neural Network (DNN) for calibration. The second one enforces a structured scoring shape with a single-peak distribution while keeping the DNN as a calibration method. A systematic evaluation using the 70-billion-parameter versions of Llama 3.1 and Llama 3.3 reveals that the first method significantly outperforms the classical approach, achieving a kappa score of 0.634 compared to 0.2233. However, the second method did not improve performance as expected.