Reproducibility Project

The Reproducibility Project: Psychology was a collaboration of 270 contributing authors to repeat 100 published experimental and correlational psychological studies. This project was led by the Center for Open Science and its co-founder, Brian Nosek, who started the project in November 2011. The results of this collaboration were published in August 2015. Reproducibility is the ability to produce a copy or duplicate, in this case it is the ability to replicate the results of the original studies. The project has illustrated the growing problem of failed reproducibility in social science. This project has started a movement that has spread through the science world with the expanded testing of the reproducibility of published works.^[1]

Reproducibility Project

Brian Nosek of University of Virginia and colleagues sought out to replicate 100 different studies that all were published in 2008.^[2] The project pulled these studies from three different journals, Psychological Science, the Journal of Personality and Social Psychology, and the Journal of Experimental Psychology: Learning, Memory, and Cognition, published in 2008 to see if they could get the same results as the initial findings. In their initial publications 97 of these 100 studies claimed to have significant results. To stay as true as they could the group went through extensive measures to remain true to the original studies, to the extent of consulting the original authors. Even with all the extra steps taken to ensure the same conditions of the original studies only 36.1% of the studies replicated, and if they did replicate their effects were smaller than the initial studies effects. The authors emphasized that the findings reflect a problem that affects all of science not just psychology, and that there is room to improve reproducibility in psychology.

Statistical relevance

Failure to replicate can be caused by a few different reasons. The first is a type II error, which is when you accept the null hypothesis when it is false.^[3] This can be classified as a false negative. A type I error is the rejection of a null hypothesis even if it is true, so this is considered a false positive.^[4]

Center for Open Science

The Center for Open Science was founded by Brian Nosek and Jeff Spies in 2013 with a $5.25 million grant from the Laura and John Arnold Foundation.^[5] They have built a team that today has about 50 individuals on it. The goal of the group is to help increase the openness, integrity and reproducibility of scientific research. The COS which is a rather small number of individuals oversee much larger groups that are helping with the COS's mission.^[6] The group is made up of multiple different kinds of scientists which include astronomers, biologists, chemists, computer scientists, education researchers, engineers, neuroscientists, and psychologists.^[7] By 2017 the Foundation had provided an additional $10 million in funding.^[5]

Outcome and importance

There have been multiple implications of the Reproducibility Project. People all over have started to question the legitimacy of scientific studies that have published in esteemed journals. Journals typically only publish articles with big effect sizes that fail to reject the null hypothesis. Leading into the huge issue of people re-doing studies that have already found to fail, but not knowing because there is no record of the failed studies, which will lead to more false positives to be published. It is unknown if any of the original study authors committed fraud in publishing their projects, but some of the authors of the original studies are part of the 270 contributors of this project.

One earlier study found that around $28 billion worth of research per year in medical fields is non-reproducible.^[8]

See also

• Invalid science
• John Ioannidis
• Meta-analysis
• Proteus phenomenon
• Publication bias
• Replication crisis
• Scientific method

References