Scholarlink

For a given text corpus, model evaluation primarily includes model prediction performance estimation and model comparison. Existing methods are always directly based on point estimations of performance evaluation metrics, such as precision (p), recall (r), and \(f_1\) measure, to compare model performance. However, these methods may easily result in low replicable or even incorrect model comparison conclusions owing to the lack of probability evaluation. Thus, this study proposes voting aggregation estimators of p, r, and \(f_1\) measure based on block-regularized \(m\times 2\) cross validation. Then, a new Bayes test is constructed based on the proposed estimators to compare model performance. The proposed Bayes test can provide a probability estimation of the superiority of one model over another, giving more reliable model comparison conclusion. Several theoretical properties and extensive experiments on several natural language processing (NLP) tasks demonstrate the effectiveness of the proposed estimators and Bayes test. Our Appendix is available at https://github.com/chengjing-fighting/PRICAI2025-Paper624 .

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Model Evaluation with Precision, Recall, and F1 Measure Based on Block-regularized m×2 Cross Validation for Text Corpus

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Yan Xue,
  • Yu Wang,
  • Xuefei Cao,
  • Xingli Yang,
  • Jihong Li

摘要

For a given text corpus, model evaluation primarily includes model prediction performance estimation and model comparison. Existing methods are always directly based on point estimations of performance evaluation metrics, such as precision (p), recall (r), and \(f_1\) measure, to compare model performance. However, these methods may easily result in low replicable or even incorrect model comparison conclusions owing to the lack of probability evaluation. Thus, this study proposes voting aggregation estimators of p, r, and \(f_1\) measure based on block-regularized \(m\times 2\) cross validation. Then, a new Bayes test is constructed based on the proposed estimators to compare model performance. The proposed Bayes test can provide a probability estimation of the superiority of one model over another, giving more reliable model comparison conclusion. Several theoretical properties and extensive experiments on several natural language processing (NLP) tasks demonstrate the effectiveness of the proposed estimators and Bayes test. Our Appendix is available at https://github.com/chengjing-fighting/PRICAI2025-Paper624 .