Who's Afraid of Peer Review?

Letharia vulpina, one of the species of lichen credited with having a cancer-inhibiting molecule in a fake manuscript.

"Who's Afraid of Peer Review?" is an article written by Science correspondent John Bohannon that describes his investigation of peer review among fee-charging open-access journals. Between January and August 2013, Bohannon submitted fake scientific papers to 304 journals owned by fee-charging open access publishers. The papers, writes Bohannon, "were designed with such grave and obvious scientific flaws that they should have been rejected immediately by editors and peer reviewers", but 60% of the journals accepted them. The article and associated data were published in the 4 October 2013 issue of Science as open access.

Background

The first fee-charging open access scientific journals began appearing in 2000 with the creation of BioMed Central and then the Public Library of Science. Rather than deriving at least some of their revenue from subscription fees, fee-charging open access journals only charge the authors (or their funders) a publication fee. The published papers are then freely available on the internet. This business model, gold open access, is one of several solutions devised to make open access publishing sustainable. The number of articles published open access, or made freely available after some time behind a paywall (delayed open access), has grown rapidly. A 2013 study showed more than half of the scientific papers published in 2011 were available for free.

In part because of the low barrier to entry into this market, as well as the fast and potentially large return on investment, many so-called "predatory publishers" have created low-quality journals that provide little to no peer review or editorial control, essentially publishing every submitted article as long as the publication fee is paid. Some of these publishers additionally deceive authors about publication fees, use the names of scientists as editors and reviewers without their knowledge, and/or obfuscate the true location and identity of the publishers. The prevalence of these deceptive publishers, and what the scientific community should do about them, has been debated.

Methods

Fake papers

Bohannon used Python to create a "scientific version of Mad Libs". The paper's template is "Molecule X from lichen species Y inhibits the growth of cancer cell Z". He created a database of molecules, lichens, and cancer cells to substitute for X, Y, and Z. The data and conclusions were identical in every paper. The authors and their affiliations were also unique and fake. The papers all described the discovery of a new cancer drug extracted from a lichen, but the data did not support that conclusion and the papers had intentionally obvious flaws.

Publisher targets

To build a comprehensive list of fee-charging open access publishers, Bohannon relied on two sources: Beall's List of predatory publishers and the Directory of Open Access Journals (DOAJ). After filtering both lists for open access journals published in English, that charge authors a publication fee, and that have at least one medical, biological, or chemical journal, the list of targets included 304 publishers: 167 from the DOAJ, 121 from Beall's list, and 16 that were listed by both. The investigation focused entirely on fee-charging open access journals. Bohannon did not include other types of open access journals or subscription journals for comparison because the turnaround time for reviews in traditional journals is too long. The study consequently makes no claim about the relative quality of the different types of journals.

Results

Acceptance versus rejection

In total, 157 of the journals accepted the paper and 98 rejected it, with the other 49 not having completed their evaluation by the time Bohannon wrote his article. Of the 255 papers that underwent the entire peer review process to acceptance or rejection, about 60% of the final decisions occurred with no sign of actual peer review. For rejections, that may possibly have reflected filtering at the editorial level, but for acceptance can only reflect a flawed process. Only 36 submissions generated review comments recognizing any of the paper's scientific problems. 16 of those 36 papers were nonetheless accepted, in spite of poor to damning reviews. Many of the journals that accepted the paper are published by prestigious institutions and publishing companies, including Elsevier, SAGE, Wolters Kluwer - through its subsidiary Medknow (Journal of Natural Pharmaceuticals), and several universities.

Among those that rejected the paper are journals published by PLOS (PLOS ONE), MDPI (Cancer) and Hindawi (Chemotherapy Research and Practice, ISRN Oncology). The peer review provided by PLOS ONE was reported to be the most rigorous of all, and it was the only journal that identified the paper's ethical problems, for example the lack of documentation of how animals were treated in the creation of the cancer cell lines.

DOAJ versus Beall's list

Among the publishers on Beall's list that completed the review process, 82% accepted the paper. Bohannon stated "the results show that Beall is good at spotting publishers with poor quality control." According to Jeffrey Beall, who created the list, this supports his claim to be identifying "predatory" publishers. However, the remaining 18% of publishers identified by Beall as predatory rejected the fake paper, causing science communicator Phil Davis to state, "That means that Beall is falsely accusing nearly one in five."

Among the DOAJ publishers that completed the review process, 45% accepted the paper. According to a statement published on the DOAJ website, new criteria for inclusion in the DOAJ are being implemented.

Global map of journal fraud

Along with the report, Science published a map that shows the location of publishers, editors, and their bank accounts, color-coded by acceptance or rejection of the paper. The locations were derived from IP address traces within the raw headers of e-mails, WHOIS registrations, and bank invoices for publication fees. India emerged as the world's largest base for fee-charging open-access publishing, with 64 accepting the fatally flawed papers and only 15 rejecting it. The United States is the next largest base, with 29 publishers accepting the paper and 26 rejecting it. In Africa, Nigeria has the largest number, of which 100% accepted the paper.

Responses

Responses from open-access academic publishers

Since the story was released, publishers of three journals have stated that they are shutting them down. The DOAJ is reviewing its list and instituting tighter criteria for inclusion. The Open Access Scholarly Publishers Association (OASPA) formed a committee to investigate the circumstances that led to the acceptance of the fake paper by three of its members. On 11 November 2013, OASPA terminated the membership of two publishers (Dove Medical Press and Hikari Ltd.) who accepted the fake paper. SAGE Publications, which also accepted a fake paper, was put "under review" for 6 months. Sage announced in a statement that it was reviewing the journal that accepted the fake paper, but that it would not shut it down. Sage's membership was reinstated at the end of the review period following changes to the journal's editorial processes.

Responses from the scientific community

Within hours of its publication, the Science investigation came under intense criticism by some supporters of the open-access movement.

The first substantial critique was posted by PLOS cofounder Michael Eisen on his blog. "To suggest – as Science (though not Bohannon) are trying to do – that the problem with scientific publishing is that open access enables internet scamming is like saying that the problem with the international finance system is that it enables Nigerian wire transfer scams. There are deep problems with science publishing. But the way to fix this is not to curtail open-access publishing. It is to fix peer review." Eisen pointed out the irony of a subscription-based journal like Science publishing this report when its own peer review has failed so badly before, as in the 2010 publication of the arsenic DNA paper.

In an exchange between Eisen and Bohannon in a discussion hosted by Peter Suber, director of the Harvard Open Access Project, Eisen criticized the investigation for the bad publicity it generated for the open-access movement. "Your study exclusively targeted open access journals – [which] strongly suggested, whether you meant to suggest this or not, that open access journals are more likely to engage in shoddy peer review and therefore more deserving of scrutiny." Bohannon responded that this critique was equivalent to "shoot[ing] the messenger".

There have also been many statements of support for the investigation, and statements of concern about the publishing fraud that it revealed. The Committee on Publication Ethics has responded that "There is no doubt that this 'sting' raises a number of issues ... though I'd argue they are not necessarily the ones that Science thinks are top priorities."

Implications

Some scientists have discussed a number of options for making peer review more transparent. Doing so would make it harder to maintain a predatory journal that does no peer review, because the record of peer review would be lacking or would need to be faked. Another option is to more rigorously vet journals, for example by further empowering DOAJ or OASPA. DOAJ has recently tightened up its inclusion criteria, with the purpose of serving as a whitelist, very much like Beall's List has been a blacklist.

See also

• International Journal of Advanced Computer Technology
• List of scholarly publishing hoaxes
• Metascience
• Predatory publishing
• Sting operation
• Sokal affair
• SCIgen
• Beall's List

External links

• Who's Afraid of Peer Review?
• Full Data Set
• Interactive Map