What Can Comparisons of Randomised and Non-Randomised Studies Tell Us?

A strand of research in Evidence-Based Medicine (EBM) compares results of randomised and non-randomised studies to empirically assess the methodological superiority of randomised controlled trials (RCTs). This chapter describes how early comparisons of randomised and non-randomised studies found that non-randomised studies tend to overestimate positive effects, while more recent reviews suggest this is not the case. These comparisons have influenced debates about RCTs for decades, initially supporting RCTs as the preferred methodology in EBM and later bolstering critiques of RCTs’ “gold standard” status. However, learning from these comparisons requires careful exploration of auxiliary hypotheses about how the results of RCTs and non-randomised studies would be expected to differ, on the assumption that RCTs are in fact superior. Two such auxiliary hypotheses, labelled directional bias and random bias, are discussed. While more recent reviews call directional bias into question, random bias, though testable, has yet to be thoroughly investigated.