Me as a peer reviwer

I find it painful to read a very critical peer review, even if it clearly intended to be constructive. This is somehow especially true when I am the reviewer. That I've just written a rejection-worthy critical review, and seem to do so fairly often (always accompanied by specific suggestions for improvements) is part of my job, but not one I relish.

How not to respond to unhelpful peer reviewers

For as long as I've been a scientist, and longer, there has been extensive discussion on the many ways that peer review is broken. Peer review is how, in theory, science gets evaluated and hopefully improved, before publication, and therefore hard to dispense with, despite being widely seen as inefficient, biased, and corrupt. It goes like this: Author submits manuscript to journal, journal sends it out to independent experts for feedback, these experts (the scientific peers of the author) decide whether they are expert and independent enough to give appropriate feedback carefully read it, think about it, identify its flaws, make constructive detailed suggestions, and finally recommend to the journal whether it should be published as is, fixed and then reevaluated, or just rejected. That is, at least ideally, how is supposed to work.

There are a great many ways in which that ideal can fail. I draw a great deal of schadenfreude from reading Retraction Watch, which is effectively a blog about cases where peer review failed in one of many ways, something was published, and mistakes or misdeed were later found out. I, like most scientists, know a few people whose work may show up all over Retraction Watch some day.

Which brings me to the fact that I am currently figuring out how to respond to a review that has failed with regards to independence, expertise, detail, fact, specificity and constructiveness. I would have suggested to the journal that this person could not be an independent reviewer, except that it never occurred to me that anyone would consider him to know anything about the topic of the paper. Explaining the long history of this interaction to the journal, we have now been assured that our re-submission would be sent out to different reviewers. Even so, in resubmitting, I have to respond to all the reviewer's comments, even those that are wildly counterfactual, have nothing to do with the current manuscript, or are just complaints about the reviewer's own work not being cited more extensively. And it has to be done politely and factually. So one must never include responses like these:

• This highlights a fundamental difference in approach to science. Reviewer's comment, and publications, suggest that scientific papers should be fishing expeditions in which everything that can be gotten out of a data set is analyzed and those results that test significant published breathlessly. We started with one, a priori original question, gathered all of the available data to address it, and got a clear result, which we state concisely. While some authors would stretch the results section out to numerous exploratory paragraphs, expounding upon questions that were tailored to fit the results of the numerous analyses, that would surely be a disservice to science.
• It is not clear what this means. Perhaps the reviewer did not find our Methods section. It is in there between the Introduction and the Results.
• It does not seem that the Reviewer has any idea what kind of data we are using, despite the six paragraphs on the topic.
• Furthermore, a reading of the manuscript would have revealed that no matrix models are employed. Reviewer's comments would seem to be hastily copied and pasted from review of an unrelated paper.
• The Reviewer's publications are not relevant or useful here. Perhaps they were relevant to the paper for which most of this review was written?
• This is counterfactual and the Reviewer has excellent reason to know that.
• These quotes of the journal's rules are from an entirely different journal that the Reviewer often reviews for.
• Not only can we find no mention of this statistical rule anywhere, we note that Reviewer's own papers don't follow it. We asked an expert in these methods about this 'rule.' She called it, "hilariously made up." 

I need some empenadas.

Choosing peer reviewers

Several years back, I was unbearably excited to be, for the first time, submitting my own manuscript to a real scientific journal. I'd spent months polishing it, and was totally confident it should be published. Working through the various steps of submission (input author names and contact info, input title, input abstract, keywords, ect.) I was surprised to be asked to input names and contact info for three recommended reviewers. Defendants in courts don't get to recommend specific peers to serve on their juries, how could science be served by asking me to recommend peers to evaluate my work? Confused, I emailed one of my advisers, who happens to be an outspoken crank when it comes to all of officialdom, and came away with the impression that this was not to be taken too seriously. I hastily plugged my keywords into Google and chose three prominent names I had never heard of before, who went on to tell the journal my paper was technically sound but not of the greatest interest. I was extremely lucky in that the editors of the journal found it very interesting, and published it anyway. Failing to have learned my lesson, a few papers I submitted since then were sent to unfavorable reviewers that I had recommended.

Now, journal editors can decide to ignore these recommendations, and those invited to review can say no, but in a large portion of cases (I have no data, this is simply a strong impression), some or all of those recommended end up writing reviews. The reason journals ask for such recommendations is to help the journal editors quickly find people who are highly qualified to review. I already know who is very knowledgeable about my specific topic, while the journal editors may not. Many journals have editors who are not paid anything for their service, and all have editors whose time is limited. Asking for and using these recommendations saves time, and probably helps avoid unqualified reviewers.

Doing so, however, has some pretty clear corrupting influence. Those who are good at this game pay great attention to whom they recommend, not only carefully considering the knowledge, viewpoints and interests of those they will recommend, but crafting the paper to raise as few objections as possible from these individuals. Papers that defy much of what is well established in one field regularly are published based on the recommendations of reviewers whose knowledge comes from another branch, and this fact is not ignored in making these recommendations. If the paper is very likely to be sent to a particular reviewer, that person's terminology and definitions will be used and his or her papers referenced. At scientific meetings, people will say, "I like this idea, list me as a reviewer." At its worst, peer review is reduced to a popularity contest, with well established authors having their work (not only journal articles, but also grant applications) evaluated mostly by their friends and allies.

Mixing moral distaste, political naivity, and hurry, I have generally spent no more than a few minutes on the question of who my recommended reviewers will be. The latest iteration of this came with a paper on which two of my undergraduate students are lead authors. We submitted it to one journal with a recommended list of reviewers that was hastily thrown together, experts on the organism we were working on but not necessarily interested in our particular topic. They trashed it solely on the basis that no one cared. We submitted the same manuscript, with very few edits, to another journal, listing reviewers with some knowledge of the organism, but a strong interest in questions related to our own. We got back three extremely positive reviews, praising our highly original and relevant work, recommending several very minor changes and urging the editors to publish this paper. Yesterday morning we submitted these revisions, and yesterday evening this journal officially accepted our paper.

So I've not only learned my lesson, but decided to heed it. Whom I ask to review a paper is (depressingly) almost as important as the quality of the work. I see truly terrible papers come out in excellent journals, presumably approved by carefully chosen reviewers, and some very good papers get rejected by less selective journals, in part because of poorly considered recommendations. From now on, I will put my qualms aside and think carefully, early in the process, about whom I will recommend as reviewers. After all, everybody else is doing it.

One way to avoid writing negative reviews is to decline to act as reviewer

"Please tell us why you have declined to review this article"

I am afraid I am unable to decipher the meaning of even the title of this paper, and the abstract implies a communicative style that could charitably be called idiomatic. I know all the words being used, (and am familiar with the probable subject matter) but couldn't tell you what most of them mean in this context. Were I to review a paper written in this manner I would spend an excessive amount of time on these linguistic issues, and might or might not ever get around to figuring out the embedded science.

Lines 147-152 show great potential.

I hate writing negative reviews of people's manuscripts. I really hate writing negative reviews when I know the authors. But one has to be honest in these things, and honestly, this one needs to be tossed out and maybe even not started again. I put it nicer than that in the review. I really worked to find nice things to say about this manuscript. I'm not sure I entirely succeeded.

And while I'm on the subject...

In contrast, last week I turned down a request to review.

I replied honestly, "Judging from the abstract, this article is on topics that I know nothing about, and both it and your journal have nothing to do with the subjects of any of my publications. I am surprised you would send it to me. Please don't waste people's time with fake peer-review."

They replied, "Could you suggest a replacement reviewer?"

I replied, "No."

Coming home now

-->

I have been asked to review another paper. It is for a very good journal. I have published with them and thought their editorial office did a good job. It is on a topic I am very interested in and know the literature on quite well. It is a professional responsibility.

I have no time. I have a million projects and papers I am behind on. I have a bunch of papers I could get out relatively quickly if only I had the time, and I really need to get papers out. When I was a kid and my father came home late every night, I swore I would never "accept a bunch of extra work responsibilities just because my colleagues relied on me." My daughter is so damn cute and changing so fast. Every time I see her she says, "I need you Daddy, I need you!" (actually more like, "I ne-Jew Daddy, I ne-Jew!!!"). How can I ignore an adorable two year-old who not only needs me, but tells me so? I can't.

I said yes to the review, because that is what I am supposed to say. I will do it, and do it right. Sigh. But not tonight.

Accepted, but...

We have set ourselves a difficult task. Some months ago, my friends and I submitted a review article, clarifying some common (within the scientific literature) misconceptions, to a very good anthropology journal. Today we heard back from them. The editor explained that it took longer than usual because he had sent it out to "several" reviewers, and then "several more" and was waiting to get comments from all of them.

The good news is that all the reviewers seemed to like it, and the editor knows which issue of the journal he wants to put it in, which we take as the paper being accepted. The bad news (or at least time-consuming news) is that all of the several and several reviewers made long lists of things they would like to see changed, added, clarified or reorganized. I have not yet finished reading all these comments, but I don't see a lot of repetition, meaning that we have hundreds of distinct comments and criticisms to deal with. In the months we were waiting, we also showed the draft to a couple of colleagues, who gave us still different but also very useful comments. The good news is that the editor has given us permission to go well beyond the usual page limit for this journal in order to deal with all the reviewer comments. The bad news is that we now can't use space limitations as an excuse for not dealing with relevant points or citing relevant literature. So we have a great deal of rewriting to do.

Part of the problem with writing an article pointing out places where other people's thinking or language has been unclear is that one has to live up to very high standards for clarity in one's thinking and language. We've already extensively rewritten this paper a few times, and each time it has gotten clearer. Nevertheless, a large portion of the reviewers' comments are right on, and further clarification is needed. By the time this thing comes out it will either be brilliant or a total muddle. I'm not clear which.

A reviewer's lot is not an'appy one

Suppose you are a scientist, and suppose you get an email asking you to peer-review a paper. This email contains the abstract of the paper, so that you can assess whether you are qualified to review the paper. Reading the abstract, you find that you are qualified, as the topic is one you know well. You also notice that you are deeply skeptical of the argument being made, and that you are very likely to recommend that the article not be published. What is your professional duty? Should you try to read the article with an open mind, despite your misgivings, or should you simply decline to review it out of fear of being biased?

In this situation, I did the former, reasoning that if we only review article we are sympathetic to, many terrible articles will be published simply because skeptical reviewers eliminated themselves, and some good articles with unpopular claims will be rejected for lack of qualified reviewers. I read the article, found it irreparably flawed in several major respects, and suggested that the journal reject it. I looked hard for nice things to say about it and didn't find much. While I'm confident my review was accurate, I'm glad these things are anonymous.

Constructive trashing

I very much wanted to like the first paper I peer-reviewed. Having gotten some negative peer reviews on my own papers, I didn't want to inflict the same upon whoever the authors might be. So when the journal sent me the manuscript, I was eager to see the good in it. I wasn't able to find much. The analysis was poorly designed, the questions not clearly framed and the writing somewhat loose and hard to follow. The topic was interesting, and the authors had the data to address it well, but failed to do so. I decided that if I couldn't write a positive review I could at least give detailed criticisms, so that if the authors do rewrite the paper they will have some guidance on what needs to be fixed. To the authors this will look like two pages of dumping, at least at first. I hope they do rewrite, as it has the potential to be a useful paper, and I hope that if they do, they find my comments helpful.

I had decided beforehand that whatever I wrote, I would keep the review anonymous. This means that the journal editors know who wrote it, but the authors do not. They give reviewers the option of signing their reviews, but I'm not sure why. The temptation of course is to sign the positive reviews only, but this is considered something akin to slimy. I prefer to be able to give my opinion without considering what the authors might think of me, or how they will review my papers in the future. Unlike online forums where people can be as abusive as they want because they are entirely anonymous, here the editors of the journal know who wrote what and can respond accordingly.

I'm happy to have been asked, by a well-respected journal, to be a reviewer. I've learned from the experience. Perhaps I should request that next time they send me a paper I'll like?