The Paradox of Peer Review

Peer review, which can be traced back to the 18th century, originated as a publicity stunt. The Royal Society of London began a tradition of having well-known scientists write reports on new scientific manuscripts to increase the visibility of science.

It was only in the 20th century that peer reviewers alongside editors were imagined as scientific gatekeepers “to ensure the integrity of the scientific literature as a whole,” as Harvard History of Science professor Alex Csiszar wrote in a commentary in Nature.

Peer review became a more standardized, formalized journal practice after World War II. Since then, most journals follow a similar practice: When a researcher submits a paper for publication, journal editors bring in peer reviewers to evaluate the paper’s quality, suitability for their journal, and importance.

As Jack Szostak, Harvard professor of Chemistry and Chemical Biology and Genetics and winner of the 2009 Nobel Prize in Physiology or Medicine, told me in an interview, some reviewers do give constructive suggestions.

“There've definitely been times when reviewers pointed out a mistake that we made in interpreting the data or things like that,” Szostak said. “That's really the ideal.”

But oftentimes, peer review, a process that can take months, doesn’t align with the ideal of improving conclusions — or misses it altogether.

Problematically, peer review assumes a veil of objectivity and expertise when it is an incredibly subjective process. Reviewers inevitably have overt or unconscious biases towards the institution, methods, or even individual researchers of the publication they’re reviewing. Additionally, because reviewers must have a certain level of authority in the subject, their work is often in direct competition with what’s presented in these potential publications. In some specialized fields, only a handful of researchers — and by extension, reviewers — are available, most of whom are familiar with each others’ work.

Even more insidiously, peer review actually holds back scientific progress in two crucial ways. Firstly, it enables faulty studies to parade an illusion of authority, like the now-retracted fraudulent claim published in The Lancet in 1998 that linked the measles, mumps, and rubella vaccine with autism. Secondly, peer review can actually suppress conclusions that shift scientific paradigms, discouraging those who question or bring up evidence against prevailing theories.

“The hardest things to get published are things that are really new and that go against the accepted wisdom in a field, and if you're contradicting what everybody thinks they know,” Szostak said.

It is hard to imagine most classic earth-shaking theories, for example, passing the threshold of peer review. Galileo’s and Einstein’s contemporaries all mounted harsh resistance to their novel theories, and would have likely prevented publication of heliocentric or relativistic theories, respectively. Maxwell’s famous kinetic theory of gases were anticipated and submitted for publication by another scientist over a decade before James Maxwell popularized them in a paper rejected by a “referee” in 1845. Such ideas would all be lost in today’s “publish or perish” culture where not publishing in top journals can severely jeopardize scientific careers.

“I mean, how many great ideas do we not know about because of a reviewer that was super nasty to someone, or how many scientists have decided to quit science because of a review that was super nasty?” Demba Ba, associate professor of electrical engineering and bioengineering at Harvard, said in an interview

While there is no perfect solution that can allow dominant existing ideologies to accurately judge new ones, some adjustments to peer review can ensure it’s more productive, such as more defined assessment metrics and training for reviewers. In a survey by Elsevier and Sense about Science in 2019, 66 percent of surveyed scientists “felt reviewers would benefit from clearer guidance on reviewing criteria.”Greater transparency is also needed, such as through public reviews with reviewers’ names. Additionally, many journals are moving towards “double-blind review,” which preserves the anonymity of the reviewer and the author. It’s not foolproof, however, because authors are recognizable from their body of work in the subject matter, citations, or writing style.

Ba is on the Board of Reviewing Editors for the journal eLife, where reviewers and editors collaborate to write their reviews, which are then published alongside accepted papers. eLife and other journals are experimenting with varying methods of peer review in an attempt to move away from the hypercompetitive yet ineffective publication process.

There’s been growing use of preprint servers like arXiv and bioRxiv, where scientists can upload papers that are not yet published or have already been rejected by peer review. At such forums, scientists can share conclusions and debate results without the gatekeeping of peer reviewers. It’s evolution in real-time.

“People can see it, people can write comments. They can write criticisms that can give you ideas for how to make a stronger case for whatever you're proposing,” Szostak said about preprint servers. “So even if a paper doesn't get accepted in some fancy journal because it's a really groundbreaking new finding, at least it's out there, and people can see it and they can judge for themselves.”

When introduced in the early 1800s, some claimed peer reviewers “used the cover of anonymity to advance their personal interests — perhaps through undetectable acts of piracy — at the expense of helpless authors,” as Csiszar puts it. Since then, peer review has changed, both in practice and in principle. While worthwhile adjustments can and should continue to be made, peer review ultimately continues to be a paradox, a seemingly necessary evil. In this way, peer review, like science, is a work in progress.

Julie Heng ’24 is a Crimson Editorial editor. Her column runs on alternate Tuesdays.

Have a suggestion, question, or concern for The Crimson Editorial Board? Click here.