Scholarlink

<p>Researcher degrees of freedom in data analysis present significant challenges in social sciences, where different analytical decisions can lead to varying conclusions. In this work, we propose an example of an exploratory multiverse simulation to empirically compare various decision pathways to identify an effect’s sensitivity to different analytical choices. The approach is demonstrated on the congruency sequence effect (CSE), a well-studied phenomenon in cognitive control research. We reviewed existing literature to identify common non-theory-specific analytical decisions, such as outlier exclusion criteria and hypothesis testing methods, and incorporated these into our simulation framework. Using 20,000 simulated datasets, we compared the true positive rates (TPR) and false positive rates (FPR) across 50 different decision pathways, resulting in a total of 1 million analyses. Our results indicate substantial differences in power and type I error rates across the analytical pathways, with some posing a significant risk of producing high false positives. The findings underscore the importance of running extensive simulations to investigate different data handling and hypothesis testing approaches in certain research fields. This case study serves as an example for conducting similar simulation procedures in research fields characterized by high variability in analytical decisions when investigating an otherwise widely accepted effect.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Multiverse simulation to explore the impact of analytical choices on type I and type II errors in a reaction time study

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年9月18日

查看全文
  • Miklos Bognar,
  • Marton A. Varga,
  • Don van Ravenzwaaij,
  • Zoltan Kekecs,
  • James A. Grange,
  • Mate Gyurkovics,
  • Balazs Aczel

摘要

Researcher degrees of freedom in data analysis present significant challenges in social sciences, where different analytical decisions can lead to varying conclusions. In this work, we propose an example of an exploratory multiverse simulation to empirically compare various decision pathways to identify an effect’s sensitivity to different analytical choices. The approach is demonstrated on the congruency sequence effect (CSE), a well-studied phenomenon in cognitive control research. We reviewed existing literature to identify common non-theory-specific analytical decisions, such as outlier exclusion criteria and hypothesis testing methods, and incorporated these into our simulation framework. Using 20,000 simulated datasets, we compared the true positive rates (TPR) and false positive rates (FPR) across 50 different decision pathways, resulting in a total of 1 million analyses. Our results indicate substantial differences in power and type I error rates across the analytical pathways, with some posing a significant risk of producing high false positives. The findings underscore the importance of running extensive simulations to investigate different data handling and hypothesis testing approaches in certain research fields. This case study serves as an example for conducting similar simulation procedures in research fields characterized by high variability in analytical decisions when investigating an otherwise widely accepted effect.