Data Too Good, Data Too Strange: Piltdown Hoax and the Reproducibility Crisis in Psychology

Science knows mischief. It doesn’t relish the knowledge and likely wishes that the mischief would disappear, buried beneath piles of meticulously conducted experiments. A hopeful thought, but unrealistic. For better or worse, what was earlier said of Freud’s shortcomings must now be said about science in sum: Scientists ply their trade in an impure world and sometimes the unexpected happens. The unexpected takes several forms. One such form is familiar: the too-good-to-be-true finding, the one that can be uncovered at times rather straightforwardly because the data are too smooth, too much of a perfect fit to theory, and lack the nasty bits of disconfirming evidence that seem required to accompany any new scientific findings, perhaps especially those in the social sciences. On the other hand, such smooth and uncomplicated data are often accepted without dissent. The data can be in a sense protected from critical scrutiny because of their match to current culture or to accepted scientific theory. This protection constitutes an unusual kind of choosing soft.