Peer review is dead, long live blog review



The title above is meant to be sarcastic but in the course of getting my many papers published in the academic journals, I got a close-up look at peer review and what I saw was not pretty. I often wondered if the reviewer had in fact read my paper, for instance. So I actually agree that review via the many specialized blogs can be much more searching and valuable, whether or not it was in the case under discussion.

Rather amusingly, however, because peer review is so poor many papers published in my field were glaringly bad -- showing no knowledge of the prior literature on the subject, for instance. It was therefore a doddle for me to write a critique tearing the negligent paper to shreds. Journal editors hate publishing critiques but my critiques were often so devastating that they just had to be published. So I got an extra opportunity to put my message across because peer review was so incompetent -- JR


In January 2009, Nature splashed its front cover with the results of a new study titled 'Warming of the Antarctic ice-sheet surface since the 1957 International Geophysical Year'. The article was accompanied by a glowing editorial from Nature and was widely reported on in the media.

A very short time after the paper was published, a number of factual errors were found in the paper, along with significant issues with the methodology used to obtain the surprising results. The errors and the methodological problems were reported and discussed by climate change blogs Watts Up With That, The Air Vent, Climate Audit and Real Climate.

Imagine if at this stage Nature's editor in chief looked at the reported blog commentary and decided the journal had published a paper, which while it had gone through the normal peer review processes, based on some of the blog commentary, was basically fundamentally flawed and should not have been published.

Furthermore, the original reviewers may have shared some of the climate alarm notions of the authors, bringing the veracity of the original review into question. Media coverage also sensationalised aspects of the results. The editor in chief is so embarrassed by the publication of the erroneous paper, he decides to resign.

Sounds farcical? In fact Nature's editor did not resign. Indeed there was no need to resign, there was no expectation on the part of the scientific community that a resignation was called for, regardless of the issues with the paper.

Subsequently Nature published a correction by the authors that dealt with some of the factual errors. And later, the blog commentary dealing with the methodological problems, ended up being published as a peer reviewed paper, by Ryan O'Donnell, Nicolas Lewis, Steve McIntyre and Jeff Condon, in the Journal of Climate.

Unlike the original paper however, this received very little media attention. Perhaps the long time the paper spent in peer review (10 months) and the less sensational results dulled the media's interest.

This is just one example of how the peer review system works. Papers are written, reviewed, rejected accepted, acclaimed, criticised, corrected, refuted and debunked. When they are significantly in error they may even be retracted. The process of science, and the reason why it works so well, is because it is one of continual correction and revision. Theory stands until a better theory comes along to replace it. Peer review acts as a general screening tool, but it is by no means perfect, and it is ridiculous to expect it to work perfectly every time.

A better system involving a combination of anonymous and online reviews is emerging to replace traditional review and is represented by journals such as The Cryosphere. Ideally we would see independent auditing of the results and conclusions of important papers, but as yet there is no funding for such a system, nor much enthusiasm for this among the scientific community. However, blogs such as Climate Audit are picking up the slack in this area with mixed reactions from the scientific community.

In terms of ground-breaking papers, the rate of misses is much greater than the hits. Indeed, for one branch of science, epidemiology, recent research suggests that most published findings are proven false within five years of publication. There is no reason to suggest that other disciplines have better score sheets. If an editor resigned every time a problem was found with a published paper, scientific publishing would quickly grind to a halt.

To some astonishment the scenario outlined above, in which a journal editor resigns over the publication of a controversial paper, has recently occurred. It involves a paper by Roy Spencer and William Braswell published just last month in the journal, Remote Sensing titled 'On the Misdiagnosis of Climate Feedbacks from Variations in Earth's Radiant Energy Balance'. Like the Antarctic paper in Nature, the paper by Spencer and Braswell went through the normal peer review process. It was promoted by the authors' university through a press release and received a few mentions in the media.

Like the Antarctic paper, some of the media coverage sensationalised the results. The paper also came in for favourable and harsh criticism on the internet, and it appears the paper is not free from error, or methodological issues.

However, rather than allow the peer reviewed system to take care of the issues in the normal manner, the journal's editor, Wolfgang Wagner, took the unprecedented step of resigning over it. In his editorial comment in Remote Sensing, Professor Wagner explains how he, remarkably, relied not on the peer reviewed literature to back his decision, but on comments on an internet blog. He states:
Peer-reviewed journals are a pillar of modern science. Their aim is to achieve highest scientific standards by carrying out a rigorous peer review that is, as a minimum requirement, supposed to be able to identify fundamental methodological errors or false claims. Unfortunately, as many climate researchers and engaged observers of the climate change debate pointed out in various internet discussion fora, the paper by Spencer and Braswell that was recently published in Remote Sensing is most likely problematic in both aspects and should therefore not have been published. After having become aware of the situation, and studying the various pro and contra arguments, I agree with the critics of the paper. Therefore, I would like to take the responsibility for this editorial decision and, as a result, step down as Editor-in-Chief of the journal Remote Sensing.

In the process the editor has also broke the trust of the reviewers, deriding them in the process by stating in his editorial:
The editorial team unintentionally selected three reviewers who probably share some climate sceptic notions of the authors.

God forbid that a paper ever be reviewed by scientists who may have similar opinions to the author. On this basis will we now see "sceptical" reviewers invited to provide critical input into the next IPCC report?

Professor Wagner, who has no expertise in the relevant area of climate science and hence cannot judge the value of the paper on his own the merits (that's why expert reviewers are used), relied on the commentary of a non-peer reviewed climate blog in order to justify his resignation. The resignation was not expected, or required and is highly unusual. The paper by Spencer and Braswell has not been retracted, however a number of formal critiques of the paper have very rapidly appeared. One in the journal Geophysical Research Letters and one in Remote Sensing. Commentary about both has also been quick on the internet (eg, at Climate Audit - here and here), but no resignations are expected from the issues uncovered.

The commentary in Remote Sensing was received and accepted at warp speed, all in the space of 24 hours; and oddly, against convention, it does not appear with a reply by the original authors. It is anticipated that this will be published in the near future.

In comparison, criticism of Nature's Antarctic warming paper sat in peer review at the Journal of Climate for an amazing 10 months. It seems an expressway exists through the peer review system for papers and comments that support a particular view, while others are considered and reviewed at a snail's pace. Nevertheless ,the system works in slowly advancing knowledge, even if sometimes we go back a few steps in order to progress further in the future.

Wagner's resignation has unnecessarily prejudiced opinion about Spencer and Braswell's paper, regardless of whether, in the long run, the work gets refuted, or it ends up being confirmed. The peer review system, even when abused, is more than capable of dealing with the scientific debate and makes a mockery of Wagner's immature decision. Based on Wagner's actions, peer review is dead; long live blog review.

SOURCE

No comments:

Post a Comment

All comments containing Chinese characters will not be published as I do not understand them