Scholarlink

<p>Words of estimative probability (WEPs), such as “maybe” or “probably not” are ubiquitous in natural language for communicating estimative uncertainty. In linguistics, WEPs are hypothesized to have special (probabilistic) semantics, and their calibration with numerical estimates has long been an area of study. Motivated by increasing usage of large language models (LLMs) in applications requiring robust communication of uncertainty, this article studies how divergences in interpreting WEP between humans and LLMs reveal the limits of statistical language models in reproducing the subtleties of communication under uncertainty. Through a detailed empirical study, we show that established LLMs align with human estimates from an established (Fagen–Ulmschneider) survey only for some WEPs presented in English. Divergence is also observed for prompts using gendered and Chinese contexts. Upon further investigating the ability of GPT-4 to consistently map statistical expressions of uncertainty to appropriate WEPs, we observe significant performance gaps. The results contribute to a growing body of research on using LLMs to study complex communicative phenomena under diverse experimental conditions.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

An evaluation of estimative uncertainty in large language models

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年2月2日

查看全文
  • Zhisheng Tang,
  • Ke Shen,
  • Mayank Kejriwal

摘要

Words of estimative probability (WEPs), such as “maybe” or “probably not” are ubiquitous in natural language for communicating estimative uncertainty. In linguistics, WEPs are hypothesized to have special (probabilistic) semantics, and their calibration with numerical estimates has long been an area of study. Motivated by increasing usage of large language models (LLMs) in applications requiring robust communication of uncertainty, this article studies how divergences in interpreting WEP between humans and LLMs reveal the limits of statistical language models in reproducing the subtleties of communication under uncertainty. Through a detailed empirical study, we show that established LLMs align with human estimates from an established (Fagen–Ulmschneider) survey only for some WEPs presented in English. Divergence is also observed for prompts using gendered and Chinese contexts. Upon further investigating the ability of GPT-4 to consistently map statistical expressions of uncertainty to appropriate WEPs, we observe significant performance gaps. The results contribute to a growing body of research on using LLMs to study complex communicative phenomena under diverse experimental conditions.