Professors Criticize Landmark Study Undermining Psychology Research

Psychology professor Daniel Gilbert and University professor Gary King criticized a 2015 study claiming that more than half of all psychology studies cannot be replicated, finding that the study itself contains replication flaws.

More than 250 authors contributed to the original “Science” magazine study, which was named by the magazine as one of the top 10 most important scientific breakthroughs of 2015 and garnered significant media attention, King said.

“I do think the original article had such incredibly far reach, having been published in the world’s premier scientific journal, and having leveled a charge against the entire field that really shook everyone up, so I think it really did quite a lot of damage to the reputation of psychology,” Gilbert said.

Gilbert said that the original article had three overarching problems. The authors failed to draw a random sample of psychology studies, they replicated the experiments in the studies that they chose under different conditions than the original experiments, and the data contained statistical errors.

“The most important mistake the authors of the original paper made was that they viewed their work as separate from, and above, the rules of science,” King wrote in an email.

Stephen Pettigrew, a graduate student in the Government Department, was brought into the project by King to work with the data from the original authors. All of this data was available online. Pettigrew, Gilbert, and King were also in communication with the lead author of the article.

“It was a really interesting case of collaboration,” Pettigrew said. “It was the scientific process at work for sure, where everybody’s trying to work toward the truth.”

Since publication, the team has received many responses to their critique, some positive and some negative. According to Pettigrew, the lead author of the original article, Brian A. Nosek, has been retweeting comments on both sides of the debate.

“My general response is that I am delighted that they took the time to explore the data and generate a comment,” Nosek wrote in an email. “The purpose of doing the project openly - sharing the process, materials and data - was so that many people would do exactly this.”

Gilbert said he hopes people will consider both sides of the argument.

“I’m only concerned with people who really carefully read all the material and drew their own conclusions,” Gilbert said.

He said he is confident that any scientist who does indeed read carefully will formulate his same conclusion.

Gilbert emphasized that the critique was not an attack against the original article, but rather part of the process of science.

“The beauty of science is that we keep correcting each other and that helps us move forward.” Gilbert said. “So we’re glad that they published the article, and we’re glad that we were able to publish the critique and engage in a productive conversation with our critics.”

—Staff writer Mia C. Karr can be reached at mia.karr@thecrimson.com. Follow her on Twitter at @miackarr