Scholarlink

<p>Key Information Extraction (KIE) systems based on Deep Learning achieve strong token-level performance but offer no formal guarantees on prediction reliability, limiting their adoption in business-critical document workflows. In this work, we introduce a post hoc Uncertainty Quantification framework for KIE using Split Conformal Prediction (CP). After fine-tuning multimodal transformer models on a challenging receipt dataset, we reserve a held-out calibration set to derive nonconformity scores and construct entity-level prediction sets that satisfy a user-specified error rate. On unseen receipts, CP achieves tight marginal coverage (98.3% for <InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(\alpha =0.02\)</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mi>α</mi> <mo>=</mo> <mn>0.02</mn> </mrow> </math></EquationSource> </InlineEquation>), with 70% of predictions being high-confidence singletons. A detailed analysis shows that highly structured fields such as dates and prices yield small, singleton sets with near–perfect reliability, whereas rare or semantically ambiguous fields such as tips or generic keywords produce larger sets and lower coverage. By exposing positional biases and common label confusions that standard F1-scores and document-accuracy metrics overlook, CP reveals critical risk areas for downstream automation. Finally, we demonstrate how calibrated prediction-set sizes can drive risk-aware workflows by automatically processing high-confidence extractions and flagging uncertain cases for human review, thereby enhancing the efficiency, trustworthiness and operational feasibility of real-world document-processing systems.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Beyond Accuracy: Understanding Model Confidence in Key Information Extraction with Conformal Prediction

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年3月10日

查看全文
  • Alexander Rombach,
  • Nijat Mehdiyev

摘要

Key Information Extraction (KIE) systems based on Deep Learning achieve strong token-level performance but offer no formal guarantees on prediction reliability, limiting their adoption in business-critical document workflows. In this work, we introduce a post hoc Uncertainty Quantification framework for KIE using Split Conformal Prediction (CP). After fine-tuning multimodal transformer models on a challenging receipt dataset, we reserve a held-out calibration set to derive nonconformity scores and construct entity-level prediction sets that satisfy a user-specified error rate. On unseen receipts, CP achieves tight marginal coverage (98.3% for \(\alpha =0.02\) α = 0.02 ), with 70% of predictions being high-confidence singletons. A detailed analysis shows that highly structured fields such as dates and prices yield small, singleton sets with near–perfect reliability, whereas rare or semantically ambiguous fields such as tips or generic keywords produce larger sets and lower coverage. By exposing positional biases and common label confusions that standard F1-scores and document-accuracy metrics overlook, CP reveals critical risk areas for downstream automation. Finally, we demonstrate how calibrated prediction-set sizes can drive risk-aware workflows by automatically processing high-confidence extractions and flagging uncertain cases for human review, thereby enhancing the efficiency, trustworthiness and operational feasibility of real-world document-processing systems.