Over the past 300 years, the peer review process has played an integral role in scientific publishing. If you don’t happen to spend your time injecting laboratory mice with toxic chemicals and submitting your results to the New England Journal of Medicine, here’s how it typically works: A manuscript is submitted to a journal using an online manager, after which editors screen the paper and decide whether it merits further consideration. At some journals, as many as 50% of submissions are declined at this stage. If the paper receives a “yay” instead of a “nay,” the journal sends it to at least two peer reviewers who are asked to scrutinize the paper and provide criticisms and critiques, ultimately recommending that the editors either accept it “as is,” accept it with revisions, or reject it.

Studies have shown that about 50% of journal editors rely almost exclusively on these peer evaluations, which raises an important question: How do journals select competent reviewers?

Common sense says that more experience and higher rank translate to better reviewing skills. However, a PLOS Medicine study in 2007 showed no such relationship. The authors examined 2,856 reviews by 308 reviewers for Annals of Emergency Medicine, a revered journal that for over 15 years has rated the quality of each review using a numerical scoring system. The results showed that experience, academic rank, and formal training in epidemiology or statistics did not significantly predict subsequent performance of higher-quality reviews. It also suggested that, in general, younger reviewers submitted stronger reviews.

So what? When presented the opportunity, any physician can and would produce a scrupulous review of a manuscript — right?

Wrong.

Flash back to 1998, when Annals of Emergency Medicine cleverly put together a fictitious manuscript riddled with errors and distributed it to 203 reviewers for evaluation. The errors were divided into major and minor categories. The major errors included such blunders as faulty or plainly unscientific methods, as well as blatantly erroneous data analyses. Minor errors consisted of failure to observe or report negative effects on study participants, incorrect statistical analysis, and fabricated references — just to mention a few. According to the authors, the majority of peer reviewers failed to identify two-thirds of the major errors in the manuscript. Forty-one percent of reviewers indicated that the manuscript should be accepted for publication.

What about consistency? In 1982, two authors took twelve papers that had been published by prestigious psychology journals within the previous three years and resubmitted them to the respective journals. The names of the authors for the resubmitted papers, and the names of their affiliations, were all changed to fictitious ones. Three manuscripts were recognized as being duplicates. Of the nine remaining papers, eight were rejected for “methodological inconsistencies,” not for lack of originality. One paper was accepted again.

Last week, I received an email from a well-respected medical journal. The editor wanted my help reviewing a manuscript that was being considered for publication. Noticing the request was addressed to “Dr. Spencer,” I shot back a quick reply saying there’d been a mistake. I’m not a doctor; I’m a medical student.

Hours later, I got this response:

Thank you for your email. We would very much appreciate your review of this manuscript. Your degree information has been corrected.

The peer review process clearly has flaws. It’s no wonder so many publications are retracted every year, or that each issue of a journal includes several corrections of previously published articles. Without universal standards, manuscript reviews remain subjective, imperfect, and inconsistent at best. The time has come to re-examine the quality of this system.

Meanwhile, those who rely on medical journals for practice guidelines should be educated on the flawed nature of this system and should be encouraged to judge publications on their merit rather than the apparent prestige of a journal.

Now, if you’ll please excuse me, I have a manuscript to review.

Robert Spencer is a medical student.