Scholarlink

<p>Evaluating semantic consistency between text and images remains a critical challenge in text-to-image generation. While human evaluation is regarded as the gold standard, existing practices lack consensus, which undermines their reliability. To address this, we introduce <i>SCoRE</i>, a standardized human evaluation method that integrates psychometric theories and methodologies to provide reliable measurements of semantic consistency. Using <i>SCoRE</i>, we uncover systematic differences between standardized and ad hoc human evaluations, highlighting the advantages of standardized evaluation in methodological reliability and result interpretability. We further show that, despite advances in recent automatic evaluation methods—such as simulating human cognition and enabling multi-dimensional ratings—they remain constrained by the absence of accurate data aligned with human judgments. Overall, <i>SCoRE</i> establishes standardized human evaluation as a benchmark, strengthens the reliability of user study methodologies, and supports the development of more robust automatic evaluations. Standardized evaluation also contributes to closer AI-human cognition alignment and the advancement of practical AI applications. The dataset of 43,540 high-quality human annotations and the code for this work are available at <a href="https://github.com/9907lqlq/StanHumEval">https://github.com/9907lqlq/StanHumEval</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

SCoRE: Standardized Human Evaluation Provides a Reliable Measure for Semantic Consistency of Text-to-Image Generation

  • 期刊论文

  • 发表日期：

    2026年4月17日

查看全文
  • Zejian Li,
  • Qi Liu,
  • Jiaman Pan,
  • Rui Huang,
  • Lefan Hou,
  • Xiangfei Hu,
  • Jiarui Ma,
  • Shengyuan Zhang,
  • Jiesi Zhang,
  • Xuetao Tian,
  • Yun Lu,
  • Hao Jiang,
  • Ling Yang,
  • Xiaoming Deng

摘要

Evaluating semantic consistency between text and images remains a critical challenge in text-to-image generation. While human evaluation is regarded as the gold standard, existing practices lack consensus, which undermines their reliability. To address this, we introduce SCoRE, a standardized human evaluation method that integrates psychometric theories and methodologies to provide reliable measurements of semantic consistency. Using SCoRE, we uncover systematic differences between standardized and ad hoc human evaluations, highlighting the advantages of standardized evaluation in methodological reliability and result interpretability. We further show that, despite advances in recent automatic evaluation methods—such as simulating human cognition and enabling multi-dimensional ratings—they remain constrained by the absence of accurate data aligned with human judgments. Overall, SCoRE establishes standardized human evaluation as a benchmark, strengthens the reliability of user study methodologies, and supports the development of more robust automatic evaluations. Standardized evaluation also contributes to closer AI-human cognition alignment and the advancement of practical AI applications. The dataset of 43,540 high-quality human annotations and the code for this work are available at https://github.com/9907lqlq/StanHumEval.