Large language models (LLMs) power many NLP applications; yet, they can produce fluent but incorrect content (hallucinations), which threatens reliability and user trust. This tutorial introduces uncertainty quantification (UQ) for text generation: methods that attach an explicit reliability signal to model outputs and enable practical safeguards such as hallucination detection and selective generation. We begin with core uncertainty concepts and explain why techniques that work well for classification do not directly transfer to autoregressive generation. We then survey representative white-box and black-box approaches, from entropy- and probability-based scores to learned probes that leverage internal representations. Retrieval-augmented generation (RAG) has become a core design pattern for LLM applications. Incorporating retrieved evidence introduces both new challenges and valuable structures for uncertainty estimation. In the ECIR edition of the tutorial, we focus on UQ techniques tailored to RAG pipelines and briefly discuss how uncertainty can guide agentic workflows. Practical demonstrations are done using LM-Polygraph ( https://github.com/IINemo/lm-polygraph ), an open-source toolkit that consolidates more than forty recent UQ and calibration methods and provides a large-scale benchmark, making it easy to reproduce results and integrate UQ into applications with minimal code. Overall, the tutorial is intended to lower the barrier to entry for researchers and developers who want to evaluate existing UQ methods, design improved ones, and deploy uncertainty-aware LLM systems.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Uncertainty Quantification for Large Language Models

  • Maxim Panov,
  • Artem Shelmanov,
  • Roman Vashurin,
  • Artem Vazhentsev,
  • Ekaterina Fadeeva,
  • Lyudmila Rvanova,
  • Timothy Baldwin

摘要

Large language models (LLMs) power many NLP applications; yet, they can produce fluent but incorrect content (hallucinations), which threatens reliability and user trust. This tutorial introduces uncertainty quantification (UQ) for text generation: methods that attach an explicit reliability signal to model outputs and enable practical safeguards such as hallucination detection and selective generation. We begin with core uncertainty concepts and explain why techniques that work well for classification do not directly transfer to autoregressive generation. We then survey representative white-box and black-box approaches, from entropy- and probability-based scores to learned probes that leverage internal representations. Retrieval-augmented generation (RAG) has become a core design pattern for LLM applications. Incorporating retrieved evidence introduces both new challenges and valuable structures for uncertainty estimation. In the ECIR edition of the tutorial, we focus on UQ techniques tailored to RAG pipelines and briefly discuss how uncertainty can guide agentic workflows. Practical demonstrations are done using LM-Polygraph ( https://github.com/IINemo/lm-polygraph ), an open-source toolkit that consolidates more than forty recent UQ and calibration methods and provides a large-scale benchmark, making it easy to reproduce results and integrate UQ into applications with minimal code. Overall, the tutorial is intended to lower the barrier to entry for researchers and developers who want to evaluate existing UQ methods, design improved ones, and deploy uncertainty-aware LLM systems.