Scholarlink

The ELOQUENT lab for evaluation of generative language model quality and usefulness addresses high-level quality criteria for generative language models through a set of open-ended shared tasks implemented, where possible, to minimise human effort in assessment, and with an objective to study how much the languages that the foundation model has been trained on make a difference in its responses. In this third ELOQUENT edition, the three planned tasks investigate how human-like text generated by language models can be (the Voight-Kampff task), how reliably a language model handles varied but equivalent input across languages (the Robustness and Consistency task), and if a generative language model can be used productively to generate and score topical quizzes without diverging into general knowledge acquired in foundational training (the PISA task). All tasks are continued evolved versions of previous editions’ tasks.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

ELOQUENT Lab at CLEF 2026: Evaluation of Generative Language Model Quality

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Jussi Karlgren,
  • Maria Barrett,
  • Ondřej Bojar,
  • Marie Isabel Engels,
  • Diandra Fabre,
  • Lorraine Goeuriot,
  • Josiane Mothe,
  • Philippe Mulhem,
  • Mario Piacentini,
  • Luis Francisco Vargas Madriz,
  • Didier Schwab,
  • Pavel Šindelář,
  • Georgios Stampoulidis,
  • Katherina Thomas,
  • Markarit Vartampetian

摘要

The ELOQUENT lab for evaluation of generative language model quality and usefulness addresses high-level quality criteria for generative language models through a set of open-ended shared tasks implemented, where possible, to minimise human effort in assessment, and with an objective to study how much the languages that the foundation model has been trained on make a difference in its responses. In this third ELOQUENT edition, the three planned tasks investigate how human-like text generated by language models can be (the Voight-Kampff task), how reliably a language model handles varied but equivalent input across languages (the Robustness and Consistency task), and if a generative language model can be used productively to generate and score topical quizzes without diverging into general knowledge acquired in foundational training (the PISA task). All tasks are continued evolved versions of previous editions’ tasks.