Scholarlink

<p>Researchers often report the area under the curve (AUC/c-statistic), Brier-index (BI), and explained variation (R<sup>2</sup>) to assess risk prediction model performance. This may be inappropriate for scores to estimate individual patient risks because population characteristics like risk score distributions may influence these performance metrics. This study assessed performance values for a fully accurate risk score under different distributions. A risk score ranging 0-100% was simulated for 100,000 individuals with 1000 bootstraps under six distributions: normal, uniform, bimodal, extreme bimodal, uniform ascending and uniform descending. Outcomes were simulated with full accuracy: 1% of individuals with a score of 0.01 had an outcome, 2% with 0.02 etc. Values for the AUC, BI, BI scaled (BS), and R<sup>2</sup> were calculated, and calibration plots made. Sensitivity analyses included narrowing risk score range and reducing score granularity. For a normally distributed score, performance indices did not exceed values generally considered poor (AUC = 0.67, BI = 0.23, BS = 0.08, R<sup>2</sup> = 8%). Values were best for an extreme bimodal distribution (AUC = 0.93, BI = 0.10, BS = 0.61, R<sup>2</sup> = 61%). Performance indices for the other distributions were moderate (AUC = 0.80–0.83, BI = 0.17–0.18, BS = 0.25–0.33, R<sup>2</sup> = 25–33%). Calibration plots were perfect for all distributions. Values worsened with narrowing score ranges. Reducing granularity had marginal effects. Risk prediction model performance values depend strongly on risk distributions. They may not exceed values generally considered poor/moderate in many common risk distributions. When assessing patient risk prediction models, investigators and clinicians interested in the accuracy of risk estimations may prioritize calibration and net benefits over commonly used metrics like the AUC and BI.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Risk prediction for individual patients and the pitfalls of selecting an optimal prediction model: do not judge a model by its c-statistic

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年5月27日

查看全文
  • Jan Willem van Dalen

摘要

Researchers often report the area under the curve (AUC/c-statistic), Brier-index (BI), and explained variation (R2) to assess risk prediction model performance. This may be inappropriate for scores to estimate individual patient risks because population characteristics like risk score distributions may influence these performance metrics. This study assessed performance values for a fully accurate risk score under different distributions. A risk score ranging 0-100% was simulated for 100,000 individuals with 1000 bootstraps under six distributions: normal, uniform, bimodal, extreme bimodal, uniform ascending and uniform descending. Outcomes were simulated with full accuracy: 1% of individuals with a score of 0.01 had an outcome, 2% with 0.02 etc. Values for the AUC, BI, BI scaled (BS), and R2 were calculated, and calibration plots made. Sensitivity analyses included narrowing risk score range and reducing score granularity. For a normally distributed score, performance indices did not exceed values generally considered poor (AUC = 0.67, BI = 0.23, BS = 0.08, R2 = 8%). Values were best for an extreme bimodal distribution (AUC = 0.93, BI = 0.10, BS = 0.61, R2 = 61%). Performance indices for the other distributions were moderate (AUC = 0.80–0.83, BI = 0.17–0.18, BS = 0.25–0.33, R2 = 25–33%). Calibration plots were perfect for all distributions. Values worsened with narrowing score ranges. Reducing granularity had marginal effects. Risk prediction model performance values depend strongly on risk distributions. They may not exceed values generally considered poor/moderate in many common risk distributions. When assessing patient risk prediction models, investigators and clinicians interested in the accuracy of risk estimations may prioritize calibration and net benefits over commonly used metrics like the AUC and BI.