Sphinx

I seem to have been writing quite a lot of confidential reviews lately, of article submissions, grant applications and cases for promotion (way to make almost everyone a little bit paranoid…), and so I was intrigued by the news that a journal in the sciences, eLife, has decided to abandon the tradition of acceptance/rejection on the basis of peer review (see announcement here). If an editor decides that a submission is worth sending out to reviewers, then it will be published, together with the reviews.

To be honest, I’m not sure what problem this is supposed to be solving; if one were really cynical, one might think that it’s all about the financial interests of a journal with substantial APCs wanting to maximise the number of articles published – ker-ching! – while maintaining the quality claim that everything is peer reviewed. At the same time, while Reviewer 2 is thus robbed of the power to put the boot in anonymously, there is still an unaccountable gate-keeping stage, vested in the editor. Reviewers are still going to have to be found, and persuaded to offer free labour, which is one of the more obvious problems with the present system; is there some idea that published reviews will count as publications and hence are more worthwhile? This does seem to be the assumption – the work put into peer review should be public and acknowledged, which will make reviewers feel better about it.

Hmm. I’ve had some bruising experiences with apparently ignorant, stupid or downright hostile reviewers over the years, so part of me completely understands the wish to force the bastards to own up to their wrecking activities or tone them down. But thinking as a reviewer – not usually hostile, though I suppose my ignorance and stupidity are always up for debate – I can see problems.

Most obviously, I have benefitted from the constructive comments and suggestions of reviewers as well as ground my teeth over their criticisms, being able to improve the final product as a result, and certainly that’s how I try to write reviews. Is the idea now that there would no longer be a revision stage, but submissions would simply be published as submitted together with the reviewers’ critical comments that the author might really have wished to incorporate (or even, conceivably, withdraw the paper)? If the submission can be revised, do the reviews get published as submitted (so that they appear to miss the point by criticising things that are no longer there) or does the reviewer have to write a new review (more work, clear disincentive to take on task in the first place)?

(This may all be very science-focused in practice, given the emphasis in the announcement on pre-prints; in other words, authors are publishing their papers anyway, it’s just a matter of the imprimatur of a journal. But given how far everything else, not least issues around Open Access, gets driven by developments on the science side of things, that doesn’t mean it’s irrelevant to us).

There’s also a question of audience, and hence appropriate content and style – not of the article, but of the review. I address reviews to the editor or funding body that’s asked me to do them, but with a powerful sense that the author is just as important if not more so; they’re the one who might find the comments most useful, after all, and the one who’s going to be hurt and infuriated – and I suppose that one of my aims is not just to be fair but to convey this in such a way that in six months’ time they might grudgingly admit that Reviewer 2 did actually have a point and the work is better as a result.

One consequence is that, I suspect, my reviews might seem to a less involved reader to be both too pedantic (because they have to deal with all the detail as well as the overall argument) and too vague (because I’m writing for people, author and editor or funding panel, who also know the submission very well). A review written to be published would be significantly different; needing to spell more out, needing to leave out the detail (or at least putting it at the end in tiny font in the way some Classical Review reviews do).

And, worse, it becomes a performance. The great thing about the anonymous review is that it is not about me, but about putting my expertise to the service of the journal, the funders and the author. The moment that ego intrudes, that I feel any inclination to explain how the author should have framed their topic, or any irritation that they haven’t cited Morley 2013, I know I need to be very careful – those might be valid criticisms, but they shouldn’t be made just because that’s how I feel. Put another way: at least for the author, the review needs to be credible on its own merits, not on the basis of the authority behind it. On reflection, my reviews are probably easily recognisable because if I suggest that the author should have consulted Morley 2013, I’ll include a paragraph’s worth of explanation of why this really is a source worth considering…

An open, published review means I have to stand by my criticisms, true; but it also means my comments will be interpreted by others in relation to me as their author, not just in their own right, regardless of how they were intended. I can easily imagine then not writing certain things because of the possibility or likelihood that these would be seen as personally motivated, because the review will reflect on me – and maybe some of those things really shouldn’t be said, but I don’t think that is universally true. As I commented on the Twitter, my concern about removing accept/reject and publishing the reviews is not that we’ll be overwhelmed with a flood of rubbish, but that we’ll be faced with unceasing flame wars and social media feuds.

I don’t think I have ever written anything in a review that I would not have offered as feedback to a graduate student or colleague – okay, maybe with some slight adjustments in tone. It would certainly affect my approach to a review if I knew the author would know my identity, but, I hope, not too much. The issue with the open review is that it’s not just deanonymised but public; not me having a private exchange or chat with student or colleague to say, look, I really don’t think your argument makes any sense, but me standing up in a research seminar to say it. (Yes, I know there are people who do that too – I was a student ar Cambridge under Keith Hopkins…).

“If you can’t say anything nice, don’t say anything at all” is a faulty principle for scholarship in general. Academics may, as a rule, be hyper-self-critical, but not necessarily in the right ways; we do need people to suggest that we’re going wrong in ways we haven’t noticed – and quite possibly non-friends will be better at this than the sort of people we might ask for feedback. That’s obvious for graduate students, but I think it applies more generally, and that surely means there is a place for private if not necessarily anonymous reviews, rather than everything being about the public performance of reviewing, critical or otherwise.

I have written a few responses to articles, and it can be very rewarding – the emphasis on responses to pieces as much as on the pieces themselves is one of the great things about Epoiesen, for example – but these are completely different things from reviews. More responses, more publications focused on dialogue, would be great; but it really doesn’t look like a solution to problems of journal gate-keeping and peer review. The worst system, apart from all the others..?