Peer review: Is it all it's cracked up to be?



Sometimes, it seems as though every new day brings a new groundbreaking finding in the quest to understand autism. New genes discovered. New bits of brain found to be a different shape or size; to be over-activated, underactivated, or not properly connected. New tasks that people with autism are either exceptionally good or exceptionally bad at. New environmental factors linked to an ever so slightly increased chance of having an autistic kid.

With all this progress, it's a wonder we haven't, well, made more progress.

In truth, advances in scientific understanding are a little more gradual than the blizzard of press releases would have you believe. Science is also a fairly haphazard process. Less the sleek and indomitable machine of popular imagination, more like a drunkard trying to find the light switch. Sometimes, science gets it completely wrong and, like my best friend on a caravan holiday, ends up weeing in the oven. True story.

Keeping things as much as possible on the straight and narrow is the process of peer review. Before they can be published as journal articles, scientific papers must be vetted by other researchers in the field. These reviewers will report back to the editor of the journal who ultimately gives the thumbs up or the thumbs down. However, the fact that a paper has successfully run the peer review gauntlet is no guarantee that it's actually any good, that it's free from errors, or that the conclusions reached are justified.

Some journals, let's be frank, will publish any old rubbish. And even well-respected journals sometimes let things slip through the net.

An example I covered a while back. In 2009, the prestigious journal, Biological Psychiatry, published a paper claiming that people with autism have extraordinary visual acuity. To cut a long story short, it’s now clear that there were major problems with the astudy and, as several subsequent studies have shown, people with autism seem to have visual acuity that is distinctly uneaglelike - no better or worse than your average non-autistic person in the street. The reviewers didn’t spot the technical problem. Neither, I should confess, did I initially. But still a peer review #fail.

A more recent example, also coincidentally from Biological Psychiatry, involved a study trying to use MRI brain scans to diagnose autism. Remarkably, one of the measures derived from this process was able to predict the severity of communication difficulties – information that it hadn't been trained on. However, a quick look at Figure 3 in the paper showed a pretty glaring mistake.

Figure 3, Uddin et al, 2011, Biological Psychiatry

The severity scores for three of the autistic kids had been lost but, instead of excluding them from the analysis, the authors had accidentally included them, giving each person a score of zero, as if they didn’t have any communication difficulties. As it happens, the authors have confirmed that the result just about holds up when the analyses are done correctly, although the effect is somewhat diminished. [Update: A correction has now been published to this effect]. The point, nonetheless, is that the paper made it past the reviewers (and eight authors and an editor) without this problem being noticed.

Mistakes such as these are easy to make – and they’re not always so easy to spot. In the case of the brain scan paper, it was only because the actual data points were plotted in a figure in the paper that the error was even visible. Errors buried deep in the analyses may never be discovered. Peer review can't help.

Even when there are no technical problems and the statistical analysis is flawless, there's still no guarantee that the results mean what the authors think they mean. The conclusions drawn depend on assumptions about what the tests are actually measuring. If those assumptions are wrong then so too might be the conclusions.

A classic example, again from the autism literature. In 1991, Josef Perner and Sue Leekam reported a neat dissociation - kids with autism failed to comprehend that a person could believe something that wasn't true any more, but those same kids were perfectly able to understand that a photograph could show a scene that had since changed.

The false photograph task by Axel Sheffler


The conclusion at the time was that kids with autism must have a very specific problem with understanding other people's mental states, otherwise they would have found both tasks equally difficult. However, this story has gradually unravelled. As Perner and Leekam have latterly argued, the two tasks aren't really equivalent at all. In particular, a photograph isn't a false representation of the present (in the way that a false belief can be), but a true representation of the past. As such, the conclusions of a specific problem with mental states were not warranted.

In hindsight it all seems quite obvious. Indeed, it is really only with hindsight that we can see which ideas, studies, and methods were the ones worth following.

This, in essence, is why scientific progress is slow and haphazard. And while peer review does serve a function, at best it's a crude spam filter for weeding out those papers that are most obviously problematic. It isn't a stamp of scientific truth, because there is no such thing as scientific truth. We shouldn't be shocked or surprised when results don't hold up. Even good science can be wrong - and it frequently is.

What is absolutely critical is that any statement presented as being "science-based" is backed up with a clear report of how the data were collected and analysed. Then the whole world can see how you reached those conclusions; where the problems in your method, your analysis, your conclusions might lie.


Further reading:
The post that got me writing this:
And a response: