Science is in the midst of a crisis: A surprising fraction of published studies fail to replicate when the procedures are repeated.<\/p>\n
For example, take the study, published in 2007, that claimed that tricky math problems requiring careful thought\u00a0are easier to solve when presented in a fuzzy font<\/a>. When researchers found in a small study that using a fuzzy font improved performance accuracy, it supported a claim that encountering perceptual challenges could induce people to reflect more carefully.<\/p>\n However,\u00a016 attempts to replicate the result failed<\/a>, definitively demonstrating that the original claim was erroneous. Plotted together on a graph, the studies formed a perfect bell curve centered around zero effect. As is frequently the case with failures to replicate, of the 17 total attempts, the original had both the smallest sample size and the most extreme result.<\/p>\n The Reproducibility Project, a collaboration of 270 psychologists, has\u00a0attempted to replicate 100 psychology studies<\/a>, while\u00a0a 2018 report <\/a>examined studies published in the prestigious scholarly journals Nature and Science between 2010 and 2015. These efforts find that about two-thirds of studies do replicate to some degree, but that the strength of the findings is often weaker than originally claimed.<\/p>\n Is this bad for science? It\u2019s certainly uncomfortable for many scientists whose work gets undercut, and the rate of failures may currently be unacceptably high. But, as a psychologist and a statistician, I believe confronting the replication crisis is good for science as a whole.<\/p>\n Practicing good science<\/strong><\/p>\n First, these replication attempts are examples of good science operating as it should. They are focused applications of the scientific method, careful experimentation and observation in the pursuit of reproducible results.<\/p>\n Many people incorrectly assume that, due to the \u201cp<.05\u201d threshold for statistical significance, only 5 percent of discoveries will prove to be errors. However, 15 years ago, physician John Ioannidis pointed to some fallacies in that assumption, arguing that false discoveries made up the majority of the published literature<\/a>. Replication efforts are confirming that the false discovery rate is much higher than 5 percent.<\/p>\n Awareness about the replication crisis appears to be promoting better behavior among scientists. Twenty years ago, the cycle for publication was basically complete after a scientist convinced three reviewers and an editor that the work was sound. Yes, the published research would become part of the literature, and therefore open to review \u2013 but that was a slow-moving process.<\/p>\n Today, the stakes have been raised for researchers. They know that there\u2019s the possibility that their study might be reviewed by thousands of opinionated commenters on the internet or by a high-profile group like the Reproducibility Project. Some journals now require scientists to make their data and computer code available, which makes it likelier that others will catch errors in their work. What\u2019s more, some scientists can now \u201cpreregister\u201d their hypotheses before starting their study \u2013 the equivalent of calling your shot before you take it.<\/p>\n Combined with open sharing of materials and data, preregistration improves the transparency and reproducibility of science, hopefully ensuring that a smaller fraction of future studies will fail to replicate.<\/p>\n While there are signs\u00a0that scientists are indeed reforming their ways<\/a>, there is still a long way to go. Out of the 1,500 accepted presentations\u00a0at the annual meeting for the Society for Behavioral Medicine in March<\/a>, only 1 in 4 of the authors reported using these open science techniques in the work they presented.<\/p>\n Improving statistical intuition<\/strong><\/p>\n Finally, the replication crisis is helping improve scientists\u2019 intuitions about statistical inference.<\/p>\n Researchers now better understand how weak designs with high uncertainty \u2013 in combination with choosing to publish only when results are statistically significant \u2013 produce exaggerated results. In fact, it is one of the reasons more than 800 scientists recently argued in favor of abandoning\u00a0statistical significance testing<\/a>.<\/p>\n