The Scientific Alliance
9th April 2010
The perils of peer review
Scientific publication has mushroomed in recent years: according to Bjork and co-authors in Information Research, about 1,350,000 papers appeared in journals in 2006. For academic scientists, such publication is vital, as they are judged on the basis of the number of papers they write. Many may regret the passing of an era when brilliant individuals could spend their time in creative thought and publish only rarely. Others may point out that that such an attitude could encourage indolence in the less gifted. Whatever the arguments, scientific publishing is nowadays a vast business.
As scientific research has progressed, so have the individual branches become more specialised, and most academic scientists these days inhabit one of a myriad of individual silos. C P Snow famously criticised the inability of the two cultures of science and humanities to communicate. But now, at least at the level of university education and beyond, the single ‘science’ culture has become a large collection of sub-cultures which do not – and in many cases cannot – communicate with each other. Physicists do not share a vocabulary with biologists, but beyond that the increasing specialisation of science gives rise to small international communities of experts, busy dealing with each other, but with little interest in having contact with the inhabitants of related silos.
Against this background, the main guarantor of scientific quality in papers, which may be read by relatively few colleagues, is the system of peer review which has become established since the mid-20th Century. Papers submitted for publication are passed by the editorial staff to a group of suitably qualified anonymous scientists who can judge the merit of the work and recommend publication, modification or rejection.
This seems an entirely rational and commonsense approach, likely to ensure that only worthwhile work is published. The body of peer-reviewed literature is treated with respect by the scientific community; anything not peer-reviewed is ignored or treated with suspicion. But, despite there being a clear need for proper checks on the quality of publications in learned journals, the present system is not perfect, nor is it realistic to expect it to become so.
Scientists are human and make choices rooted in their own prejudices and beliefs. Take, for example, the highly contentious research done by Ewen and Pusztai which, on the basis of experiments in which rats were fed various types of raw potato, was used to call into question the safety of GM foods. Following the appearance of Pusztai on the ITV World in Actionprogramme which started the whole furore, the medical journal The Lancet published a paper by the two authors. Five out of six reviewers recommended this. On the other hand, six other reviewers appointed by the Royal Society concluded that the work was flawed and should not have been published. A dozen highly-trained scientists could not agree on the merits of a relatively straightforward paper. It is inconceivable that all made an entirely objective decision based entirely on the paper itself.
Richard Horton, then editor of The Lancet, contributed a guest editorial for the Medical Journal of Australia (Genetically modified food: consternation, confusion and crack-up; MJA 2000; 172: 148-149) in which he wrote:
“The mistake, of course, is to have thought that peer review was any more than a crude means of discovering the acceptability – not the validity – of a new finding. Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong.”
From a journal editor, that sounds pretty damning, but perhaps it is just an acceptance of reality. The system may mean that – quite rightly – a paper has been subject to independent review before publication, but it does not make it irrefutable. And, of course, science itself is not a set of absolute truths but at best a collection of the best knowledge and interpretation at that particular time, open to review, validation or falsification. Human nature being what it is, peer review can get in the way of this process, since reviewers are usually the ones whose treasured beliefs might be questioned. It is disingenuous for editors to talk about how eager they are to publish groundbreaking new work which upsets existing orthodoxies; it takes a degree of courage to break away from the herd.
With no peer review or editorial control, pretty much anything could be published. The argument goes that post-publication criticism would sort the wheat from the chaff, and any journal regularly publishing rubbish would quickly lose its credibility. But, in practice, scientists busy doing their own work cannot subject all papers to line-by-line scrutiny and criticism, so the proportion of bad papers would probably rise.
When publication is not in a learned journal, then authors can say pretty much what they like, and this is the case with, for example, Jeffrey Smith, an American writer who has made a name for himself crusading against GM foods in two books, Seeds of Deception and Genetic Roulette. Although the egregious nature of his attacks of crop biotechnology have already been widely criticised, Prof Bruce Chassy (professor of food microbiology and nutritional science at the University of Illinois) and Dr David Tribe (senior lecturer in food science, food safety, biotechnology and microbiology at the University of Melbourne) have recently set up a website (Academics Review) with the tagline ‘Testing popular claims against peer-reviewed science’.
While wishing them good luck, I think that the power of peer-reviewed science to sway an argument has been much over-hyped. In similar fashion, climate change activists have claimed that their pressure for radical policies is backed purely by peer-reviewed science. Such an approach cuts no ice with sceptics. It is time to acknowledge that the current review process is necessary, but also flawed. Whatever the source of information, we should always approach it with an appropriate degree of proper scientific scepticism. The conclusions of others should never be taken on trust, whether peer-reviewed or not; we should not be afraid to interpret the evidence for ourselves.
The Scientific Alliance
St John’s Innovation Centre, Cowley Road, Cambridge CB4 0WS
Tel: +44 1223 421242
St John’s Innovation Centre, Cowley Road, Cambridge CB4 0WS
Tel: +44 1223 421242
