|Margaret and David's journal club Source|
In a recent article in the Observer newspaper, veteran film critic, Mark Kermode, discussed his changing role in the internet age - a time when anyone with a blog can be an amateur critic, and when "recommendations" used in film advertising often come, not from established critics like himself, but from anonymous and potentially untraceable twitter accounts. Kermode's musings led him to conclude that what matters above all is reputation. The critic must have something at stake if his or her critique is to be of any worth:
"criticism without risk to the critic has no value whatsoever – [...] an opinion is only worth as much as its author has to lose: their good name; their reputation; their audience; their job."
Reading Kermode's article, I couldn't help but think of the parallels with professional criticism in science - otherwise known as peer review.
Before a paper can be published in a scientific journal it's first sent to a small number of experts in the field, who report back to the journal's editor. He or she then decides whether to publish the paper, ask the authors to change it to address the reviewers' comments, or reject the paper entirely.
Typically the reviewers are anonymous, uncredited as well as unpaid. The only reward is the good karma associated with contributing towards the broader endeavour of scientific progress. The main rationale for anonymity is that it allows you as the reviewer to say what you might be reluctant to say if everyone knew who you were. The authors of the paper you've just criticized might soon be reviewing your next paper or grant proposal or sitting on your hiring committee. Perhaps you'd be tempted to mince your words for fear of retribution or repercussions down the line.
Of course, this works both ways. Hiding behind the veil of anonymity, you can be as mean and unprofessional as you like. If you so desired, you could sabotage the paper of a competitor or someone you dislike personally. You could give the paper a cursory glance, pick out something that doesn't sit with your preconceptions and, rather than consider the authors' careful justification of their approach, dismiss the paper out of hand for being "fundamentally flawed". Alternatively, the same cursory glance could lead you to write a positive review that misses the crucial details that really do make the paper problematic. A good, constructive review takes time, thought, and effort. But again, apart from the karma, there's not a whole lot of motivation or incentive to do a good job.
Last week, Science magazine published an article highlighting further problems with the current peer review system. In an elaborate sting operation, investigative journalist, John Bohannon sent out variations on a fake microbiology paper to 304 journals, more than half of which accepted the paper, despite its obvious flaws.
As it happened, all of the journals targeted were open access - anyone can read them without having to pay a subscription. In some cases, this means the authors pay the journal a fee to cover the costs associated with publication. The implication of Bohannon's article was that, if publishers have a financial motivation to accept a paper (they only get paid if that happens) then they will inevitably let through a lot of rubbish. This is a genuine concern. But, as many commenters were quick to point out, we have no idea how many closed access journals would have also accepted the paper because none of them were targeted in the sting.
If there's a lesson here, it's that peer review itself is becoming a devalued product. When anyone can set up a "peer reviewed" journal; but nobody other than the journal editor knows who the reviewers are; and when only they and the authors know what the reviews say (or if they even exist), we eventually arrive at a situation where the fact of publication means next to nothing at all.
As Kermode's discussion suggests, reputation is what matters. Established journals have their reputations and dare not lose them. New journals, open access or otherwise, have reputations to establish. And so too should we, as reviewers have reputations to build. Clearly, this is only possible if, like film critics, our reviews are attributable:
"if no one knows (or cares) who you are or what you have done, then what have you invested in your review? What do you have to lose?"
A reviewer who consistently fails to spot glaring problems in a paper will soon develop a reputation as a sloppy scientist. On the other hand, a reviewer who is always constructively critical will become known as someone whose opinion matters. A review with their signature will come to mean more - just as a review from Mark Kermode or his like should carry more weight than one from Julie, 23, on twitter.
The analogy is imperfect. For a start, film critics generally sit outside the film-making process. As reviewers of scientific articles, we are taking a break from our normal role in the "production" side of the science industry to become a temporary critic. This does introduce some conflicts of interest, but it's not clear that anonymity solves them.
And this closer involvement - knowing that the shoe will soon be on the other foot - should lead to a more constructive approach to review. As Kermode points out, it's relatively easy as a critic to do a hatchet job on a film, to produce a pithy negative review that translates into a succinct headline. Saying what's good about a film makes you vulnerable:
On some level, saying you love a film is a bit like admitting you have a crush on someone – it opens you up to accusations of foolishness, setting you up for inevitable heartbreak.Anyone who's ever been in a journal club and ventured the opinion that they "actually quite liked the paper" will empathise with this view. Picking holes is easy. Finding the true value in a paper is the real difficult art. But this after all is what we want from peer review. We want to know that someone whose opinion we value has studied the paper in depth, has thought about it, and has concluded that it's worth our while taking a closer look.