Who's Afraid of Peer Review?

Bohannon used Python to create a "scientific version of Mad Libs".[2][8] The paper's template is "Molecule X from lichen species Y inhibits the growth of cancer cell Z". He created a database of molecules, lichens, and cancer cells to substitute for X, Y, and Z. The data and conclusions were identical in every paper. The authors and their affiliations were also unique and fake. The papers all described the discovery of a new cancer drug extracted from a lichen, but the data did not support that conclusion and the papers had intentionally obvious flaws.[9][10]

To build a comprehensive list of fee-charging open access publishers, Bohannon relied on two sources: Beall's List of predatory publishers and the Directory of Open Access Journals (DOAJ). After filtering both lists for open access journals published in English, that charge authors a publication fee, and that have at least one medical, biological, or chemical journal, the list of targets included 304 publishers: 167 from the DOAJ, 121 from Beall's list, and 16 that were listed by both. The investigation focused entirely on fee-charging open access journals. Bohannon did not include other types of open access journals or subscription journals for comparison because the turnaround time for reviews in traditional journals is too long. The study consequently makes no claim about the relative quality of the different types of journals.[11]

In total, 157 of the journals accepted the paper and 98 rejected it, with the other 49 not having completed their evaluation by the time Bohannon wrote his article.[2] Of the 255 papers that underwent the entire peer review process to acceptance or rejection, about 60% of the final decisions occurred with no sign of actual peer review. For rejections, that may possibly have reflected filtering at the editorial level, but for acceptance can only reflect a flawed process. Only 36 submissions generated review comments recognizing any of the paper's scientific problems. 16 of those 36 papers were nonetheless accepted, in spite of poor to damning reviews. Many of the journals that accepted the paper are published by prestigious institutions and publishing companies, including Elsevier, SAGE, Wolters Kluwer - through its subsidiary Medknow (Journal of Natural Pharmaceuticals), and several universities.[2]

Among those that rejected the paper are journals published by PLOS (PLOS ONE), MDPI (Cancer) and Hindawi (Chemotherapy Research and Practice, ISRN Oncology). The peer review provided by PLOS ONE was reported to be the most rigorous of all, and it was the only journal that identified the paper's ethical problems, for example the lack of documentation of how animals were treated in the creation of the cancer cell lines.[2]

Among the publishers on Beall's list that completed the review process, 82% accepted the paper. Bohannon stated "the results show that Beall is good at spotting publishers with poor quality control." According to Jeffrey Beall, who created the list, this supports his claim to be identifying "predatory" publishers.[12] However, the remaining 18% of publishers identified by Beall as predatory rejected the fake paper, causing science communicator Phil Davis to state, "That means that Beall is falsely accusing nearly one in five."[13]

Among the DOAJ publishers that completed the review process, 45% accepted the paper.[13] According to a statement published on the DOAJ website, new criteria for inclusion in the DOAJ are being implemented.[14]

Along with the report, Science published a map that shows the location of publishers, editors, and their bank accounts, color-coded by acceptance or rejection of the paper. The locations were derived from IP address traces within the raw headers of e-mails, WHOIS registrations, and bank invoices for publication fees. India emerged as the world's largest base for fee-charging open-access publishing, with 64 accepting the fatally flawed papers and only 15 rejecting it.[2] The United States is the next largest base, with 29 publishers accepting the paper and 26 rejecting it. In Africa, Nigeria has the largest number, of which 100% accepted the paper.[15]

Since the story was released, publishers of three journals have stated that they are shutting them down.[16] The DOAJ is reviewing its list and instituting tighter criteria for inclusion.[17] The Open Access Scholarly Publishers Association (OASPA) formed a committee to investigate the circumstances that led to the acceptance of the fake paper by three of its members.[18] On 11 November 2013, OASPA terminated the membership of two publishers (Dove Medical Press and Hikari Ltd.) who accepted the fake paper. SAGE Publications, which also accepted a fake paper, was put "under review" for 6 months.[19] Sage announced in a statement that it was reviewing the journal that accepted the fake paper, but that it would not shut it down.[20] Sage's membership was reinstated at the end of the review period following changes to the journal's editorial processes.[21]

Within hours of its publication, the Science investigation came under intense criticism by some supporters of the open-access movement.[22][23]

The first substantial critique was posted by PLOS cofounder Michael Eisen on his blog. "To suggest – as Science (though not Bohannon) are trying to do – that the problem with scientific publishing is that open access enables internet scamming is like saying that the problem with the international finance system is that it enables Nigerian wire transfer scams. There are deep problems with science publishing. But the way to fix this is not to curtail open-access publishing. It is to fix peer review."[24] Eisen pointed out the irony of a subscription-based journal like Science publishing this report when its own peer review has failed so badly before, as in the 2010 publication of the arsenic DNA paper.

In an exchange between Eisen and Bohannon in a discussion hosted by Peter Suber, director of the Harvard Open Access Project, Eisen criticized the investigation for the bad publicity it generated for the open-access movement.[25] "Your study exclusively targeted open access journals – [which] strongly suggested, whether you meant to suggest this or not, that open access journals are more likely to engage in shoddy peer review and therefore more deserving of scrutiny." Bohannon responded that this critique was equivalent to "shooting the messenger".

There have also been many statements of support for the investigation,[26][13] and statements of concern about the publishing fraud that it revealed.[27][28][29] The Committee on Publication Ethics has responded that "There is no doubt that this 'sting' raises a number of issues ... though I'd argue they are not necessarily the ones that Science thinks are top priorities."[30]