Peer review 0

The indispens... uh, hang on, let me check my thesaurus ... the necessitous RealClimate has a stellar essay up on the subject of scientific peer review, a topic that anyone who ever talks about climate change ought to know a little something about. They agree with the general sentiment that non-peer reviewed scientific papers shouldn't be taken seriously, but go on to say that peer review is not a magic bullet. It's an important process, but doesn't ensure scientific validity.

The best part is a discussion of some of the many recent peer-reviewed papers that have been hyped as overturning the scientific consensus on anthropocentric global warming. They show how the peer review process breaks down, and more importantly, how even after the scientific community has refuted some of these papers, they go on being hyped by climate change skeptics. Specific examples abound.  Here's the money passage:

The current thinking of scientists on climate change is based on thousands of studies (Google Scholar gives 19,000 scientific articles for the full search phrase "global climate change"). Any new study will be one small grain of evidence that adds to this big pile, and it will shift the thinking of scientists slightly. Science proceeds like this in a slow, incremental way. It is extremely unlikely that any new study will immediately overthrow all the past knowledge. So even if the conclusions of the Shaviv and Veizer (2003) study discussed earlier, for instance, had been correct, this would be one small piece of evidence pitted against hundreds of others which contradict it. Scientists would find the apparent contradiction interesting and worthy of further investigation, and would devote further study to isolating the source of the contradiction.  They would not suddenly throw out all previous results. Yet, one often gets the impression that scientific progress consists of a series of revolutions where scientists discard all their past thinking each time a new result gets published. This is often because only a small handful of high-profile studies in a given field are known by the wider public and media, and thus unrealistic weight is attached to those studies. New results are often over-emphasised (sometimes by the authors, sometimes by lobby groups) to make them sound important enough to have news value. Thus "bombshells" usually end up being duds.

However, as demonstrated above, even when it initially breaks down, the process of peer-review does usually work in the end. But sometimes it can take a while. Observers would thus be well advised to be extremely skeptical of any claims in the media or elsewhere of some new "bombshell" or "revolution" that has not yet been fully vetted by the scientific community.

Also of interest is a pointer by one of the commenters to Robert L. Park's Seven Signs of Bogus Science. Excellent reading.

• Comments