Asking impertinent questions: The art of manuscript peer review

Abstract

Prepublication peer review of scientific manuscripts is used by most legitimate scientific publications. Skillful peer reviews represent a valuable contribution to the field by improving the quality of scientific research communication and promoting scientific integrity.

Peer review of scientific manuscripts was introduced in 1733 by the Royal Society of Edinburgh. In 1752, a committee of the Royal Society of London began prepublication reviews for the society's Philosophical Transactions.¹ In 1893, the British Medical Journal began using outside referees for noneditorial articles.² However, prepublication peer review of medical and scientific manuscripts did not become standard until the mid-1900s, led by the introduction of manuscript review by JAMA and Science in the 1940s and Lancet in 1976.² The peer review process was likely facilitated by the advent of photocopy technology.³

The overlapping goals of peer review are to provide editors with an assessment of the veracity and potential significance of the submission and to help the author improve the quality of the manuscript. However, there are sometimes benefits to a reviewer as well, such as improved manuscript preparation skills, exposure to new concepts and ideas, and opportunities for improved professional standing.

A number of studies have analyzed the effectiveness of manuscript peer review, the potential effects of reviewer bias, and the authors' satisfaction with the review process.^4-7 Despite the acknowledged importance of manuscript peer review, information about how to do it effectively is scant, and formal training in manuscript reviewing is usually limited. There are many different ways to complete a manuscript review, but what follows is a primer on effective manuscript peer review by an editor with years of reviewing and editing experience.

Does the manuscript pass the “smell test”? As a reviewer, you are the content expert, and editors will appreciate your placing a study into context. Are earlier publications being ignored or trivialized? Well-crafted manuscripts offer a clear explanation of their purpose, ideally in the introduction. If the reviewer struggles to grasp the manuscript's importance, so will the readers.

Rare is not the same thing as novel. Manuscripts occasionally have important implications that are not recognized by the authors. More often, the authors suggest novelty when little exists. Not all novelty is equally meaningful. I am usually unimpressed with “geographic” manuscripts (“we describe the first patient with West Nile encephalitis in all of Wyoming”), “making people aware” manuscripts (describing something that is uncommon but already well-known), “masquerading as” manuscripts, incremental data reports (adding a few more patients to last year's published summary), and manuscripts whose sole novelty is a new variant of an already well-characterized gene.

Does the title convey an adequate sense of the study's contents? Just as the abstract should provide an overview of the manuscript's contents, a title should telegraph the story an article will tell. Cute catchy titles may be lost on individuals who are less familiar with English idioms, so it is best to limit such titles to commentaries and editorials. It is generally better to avoid abbreviations in titles unless they are so widely used that they are universally recognized.

Pay close attention to the abstract. The abstract is the only part of an article that many people will read, so it should provide enough details to be useful independent of the remaining article. Does it provide context for the article? Does it contain substantive data or gloss over the results? Are there meaningful conclusions, and are these adequately supported by the information presented? There are few things more disappointing than an abstract that concludes “the data will be discussed.”

Are the figures technically adequate and the tables well-organized? Overly complicated tables or figures are difficult to interpret and ineffective. A few carefully selected, high-quality illustrations are preferable to multiple illustrations depicting the same findings. Many journals allow online supplemental data files, and these are an excellent way to provide more detailed information while streamlining the main article. Do the figure legends provide enough information to allow a reader to interpret the figure without extensive reference to the other text?

Are the methods described in sufficient detail to allow one to understand the results and, potentially, to independently confirm the data? Do the statistical methods seem appropriate? If you have concerns about the statistical methods but feel uncomfortable trying to verify them, it is fine to suggest a separate statistical review to the editor.

Does the discussion adequately credit earlier work? Is there a clear delineation of what the current study adds to our understanding of the topic? Are the study's limitations mentioned? Are the manuscript's conclusions adequately supported by the information presented? It is not reasonable, for example, to suggest that all patients with celiac disease be tested for hearing loss in a report of two such patients.

Can the manuscript be shortened without omitting important information? The shift toward electronic publication has made strict manuscript word limits less important, but removing bloated text is still a good way to improve a manuscript's focus and readability.

Mercifully, reviewers are not expected to correct grammatical errors or identify plagiarism. However, it is useful to mention writing that is so bad that it impairs understanding of the manuscript or is likely to defy correction. It is rather poor form to criticize an author's spelling and grammar in review comments that are filled with misspellings and grammatical errors.

While editors do not expect a reviewer to personally verify every bibliography entry, it is sometimes helpful to spot-check a few of the citations for completeness, accuracy, and importantly, whether the cited article actually states what the authors suggest. Primary peer-reviewed sources are generally preferable to textbooks, abstracts, chapters, or websites. Are the citations reasonably accessible to the readers? Citing an obscure publication is sometimes necessary, but references that are not readily available to the readers are not very helpful.

Does the manuscript document approval or waiver of approval by the institution's research ethics committee? Does the study maintain equipoise? Dunking children with Dravet syndrome into a hot bath to see if increased body temperature triggers seizures is problematic (yes, I once rejected such a manuscript) even in the unlikely event of its being approved by an ethics committee. Are clinical trials properly registered?

Are potential conflicts of interest adequately disclosed? The most obvious conflicts are typically the economic ones, such as the corporate sponsorship of the study itself, employment of an author by the corporate sponsor, or financial incentives to the authors.⁸ Other conflicts are more subtle, and even the appearance of conflict can be problematic. If properly disclosed, conflicts of interest do not always preclude publication. A good approach is to err on the side of over-disclosure.

As a reviewer, be mindful of your own potential conflicts. Are you able to consider authors' work fairly and objectively? It is best not to review manuscripts written by relatives, close colleagues, trainees, mentors, or individuals from your own institution. Not all conflicts disqualify a reviewer. Having a direct financial interest in the outcome of the study under consideration is disqualifying, while receiving an occasional honorarium from the sponsor for work unrelated to the study at hand may be acceptable. At a minimum, the reviewer should disclose potential conflicts to the editor.

What is missing from the manuscript? It is reasonable to ask for text clarification, another table or figure, or additional data that will improve the manuscript. However, it is probably too late at this stage to demand fundamental study redesigns or the collection of extensive new data. Missing data that impair a study's analysis should at a minimum be listed as a study limitation.

Begin the review comments with a one- or two-sentence summary of the manuscript. This sets the stage for the critique that follows and assures the authors that the reviewer has at least read the manuscript.

Even if you recommend rejection, try to provide helpful, actionable suggestions for improving a manuscript. Assuming that the authors are sufficiently motivated to fully address the reviewers’ suggestions, the manuscript they submit to a different journal will often be vastly improved.

Organize your comments to the authors as a numbered list. This makes it much easier for the editors to grasp the essential points of the review and facilitates an author's creation of a point-by-point response to these comments when revising the manuscript. The authors' point-by-point response, in turn, makes it easier for you and the editors to assess the revised manuscript. Typically, some reviewer concerns are more serious than others, and grouping the major and the minor concerns is useful and takes little additional time.

Point out inconsistencies. Do the data differ in the text, figures, and tables? Are the conclusions more robust than the data support? Are elements of the discussion contradictory? Does the information in the abstract contradict the text comments? Do the study's results and conclusions contradict earlier studies, and are these differences adequately documented and explained?

Are the manuscript's limitations and potential biases adequately acknowledged? All studies have flaws. These do not necessarily invalidate the manuscript's conclusions or prevent its publication, but a clear summary of the study's limitations is necessary.

Avoid making comments to the authors about whether the manuscript should be published. There are many factors that go into a publication decision, such as the other reviewer's opinion, the editor's opinion, a statistical review, the quality of the writing, and the plagiarism analysis. Most journals receive more submissions than can be accepted, so some manuscripts must be declined only because they are perceived to be less important than other submissions. A recommendation in favor of publication in the comments to the author makes it more difficult for the editors to tactfully decline a manuscript. A clearly stated publication recommendation to the editor, in contrast, is quite useful.

You cannot always support an author's approach and conclusions, but you can always be kind. Remember that even a substandard manuscript represents a fellow human being's sincere effort and earnest hopes, however modest. Before harshly ridiculing their effort, imagine how you would feel if a reviewer characterized your work as “an utter waste of time and effort” or stated that they “wouldn't wrap fish in this paper.” Such hurtful comments serve no useful purpose, and many editors remove overly harsh remarks before sharing the comments with the authors. I also stop inviting these individuals to serve as reviewers.

Manuscript peer reviewers are the unsung heroes of science. They make an important contribution to the advancement of the field, anonymously donating their time and expertise to improve other people's work. As annoying as reviewer comments can be to the authors, most manuscripts benefit greatly from the review and revision process. Many people whose own work has benefited greatly from the insights and suggestions of anonymous peer reviewers cannot be bothered to pay it forward and help other authors. Finding qualified, willing peer reviewers is the single biggest challenge for most medical editors. Please consider the importance of the peer review process the next time you are asked to serve.

E. Steve Roach: Conceptualization; project administration; writing original draft; manuscript review and editing.

The author is the editor-in-chief of the Annals of the Child Neurology Society, but the expressed opinions do not reflect the official policy of the Child Neurology Society.