Scholarlink

<p>There is growing interest in integrating explanatory and predictive research practices in psychological research. For this integration to be successful, the psychologist’s toolkit must incorporate standard procedures that enable a direct estimation of the prediction error, such as cross-validation (CV). Despite their apparent simplicity, CV methods are intricate, and thus it is crucial to adapt them to specific contexts and predictive metrics. This study delves into the performance of different CV methods in estimating the prediction error in the <InlineEquation ID="IEq1"> <InlineMediaObject> <ImageObject Color="BlackWhite" FileRef="13428_2024_2588_Article_IEq1.gif" Format="GIF" Height="16" Rendition="HTML" Resolution="72" Type="Linedraw" Width="21" /> </InlineMediaObject> <EquationSource Format="TEX">\({R}^{2}\)</EquationSource> <EquationSource Format="MATHML"><math> <msup> <mrow> <mi>R</mi> </mrow> <mn>2</mn> </msup> </math></EquationSource> </InlineEquation> and <InlineEquation ID="IEq2"> <InlineMediaObject> <ImageObject Color="BlackWhite" FileRef="13428_2024_2588_Article_IEq2.gif" Format="GIF" Height="14" Rendition="HTML" Resolution="72" Type="Linedraw" Width="37" /> </InlineMediaObject> <EquationSource Format="TEX">\(\text{MSE}\)</EquationSource> <EquationSource Format="MATHML"><math> <mtext>MSE</mtext> </math></EquationSource> </InlineEquation> metrics in regression analysis, ubiquitous in psychological research. Current approaches, which rely on the 5- or 10-fold rule of thumb or on the squared correlation between predicted and observed values, present limitations when computing the prediction error in the <InlineEquation ID="IEq3"> <InlineMediaObject> <ImageObject Color="BlackWhite" FileRef="13428_2024_2588_Article_IEq3.gif" Format="GIF" Height="16" Rendition="HTML" Resolution="72" Type="Linedraw" Width="21" /> </InlineMediaObject> <EquationSource Format="TEX">\({R}^{2}\)</EquationSource> <EquationSource Format="MATHML"><math> <msup> <mrow> <mi>R</mi> </mrow> <mn>2</mn> </msup> </math></EquationSource> </InlineEquation> metric, a widely used statistic in the behavioral sciences. We propose the use of an alternative method that overcomes these limitations and enables the computation of the leave-one-out (LOO) in the <InlineEquation ID="IEq4"> <InlineMediaObject> <ImageObject Color="BlackWhite" FileRef="13428_2024_2588_Article_IEq4.gif" Format="GIF" Height="16" Rendition="HTML" Resolution="72" Type="Linedraw" Width="21" /> </InlineMediaObject> <EquationSource Format="TEX">\({R}^{2}\)</EquationSource> <EquationSource Format="MATHML"><math> <msup> <mrow> <mi>R</mi> </mrow> <mn>2</mn> </msup> </math></EquationSource> </InlineEquation> metric. Through two Monte Carlo simulation studies and the application of CV to the data from the Many Labs Replication Project, we show that the LOO consistently has the best performance. The CV methods discussed in the present study have been implemented in the R package OutR2.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Cross-validation and predictive metrics in psychological research: Do not leave out the leave-one-out

  • 期刊论文

  • 发表日期：

    2025年2月3日

查看全文
  • Diego Iglesias,
  • Miguel A. Sorrel,
  • Ricardo Olmos

摘要

There is growing interest in integrating explanatory and predictive research practices in psychological research. For this integration to be successful, the psychologist’s toolkit must incorporate standard procedures that enable a direct estimation of the prediction error, such as cross-validation (CV). Despite their apparent simplicity, CV methods are intricate, and thus it is crucial to adapt them to specific contexts and predictive metrics. This study delves into the performance of different CV methods in estimating the prediction error in the \({R}^{2}\) R 2 and \(\text{MSE}\) MSE metrics in regression analysis, ubiquitous in psychological research. Current approaches, which rely on the 5- or 10-fold rule of thumb or on the squared correlation between predicted and observed values, present limitations when computing the prediction error in the \({R}^{2}\) R 2 metric, a widely used statistic in the behavioral sciences. We propose the use of an alternative method that overcomes these limitations and enables the computation of the leave-one-out (LOO) in the \({R}^{2}\) R 2 metric. Through two Monte Carlo simulation studies and the application of CV to the data from the Many Labs Replication Project, we show that the LOO consistently has the best performance. The CV methods discussed in the present study have been implemented in the R package OutR2.