Author: Michael Kalichman, 2001
Contributors: P.D. Magnus, Dena Plemmons
The integrity of science depends on effective peer review
A published paper reflects not only on the authors of that paper, but also on the community of scientists. Without the judgment of knowledgeable peers as a standard for the quality of science, it would not be possible to differentiate what is and is not credible.
Effective peer review depends on competent and responsible reviewers
The privilege of being part of the research community implies a responsibility to share in the task of reviewing the work of peers.
Background
For much of the last century, peer review has been the principal mechanism by which the quality of research is judged. In general, the most respected research findings are those that are known to have faced peer review. Most funding decisions in science are based on peer review. Academic advancement is generally based on success in publishing peer-reviewed research and being awarded funding based on peer review; further, it involves direct peer review of the candidate's academic career. In short, research and researchers are judged primarily by peers.
The peer-review process is based on the notion that, because much of academic inquiry is relatively specialized, peers with similar expertise are in the best position to judge one another's work. This mechanism was largely designed to evaluate the relative quality of research. However, with appropriate feedback, it can also be a valuable tool to improve a manuscript, a grant application, or the focus of an academic career. Despite these advantages, the process of peer review is hampered by both perceived and real limitations.
Critics of peer review worry that reviewers may be biased in favor of well-known researchers, or researchers at prestigious institutions, that reviewers may review the work of their competitors unfairly, that reviewers may not be qualified to provide an authoritative review, and even that reviewers will take advantage of ideas in unpublished manuscripts and grant proposals that they review. Many attempts have been made to examine these assumptions about the peer review process. Most have found such problems to be, at worst, infrequent (e.g., Abby et al., 1994; Garfunkel et al., 1994; Godlee et al., 1998; Justice et al., 1998; van Rooyen et al., 1998; Ward and Donnelly, 1998). Nonetheless, problems do occur.
Because the process of peer review is highly subjective, it is possible that some people will abuse their privileged position and act based on unconscious bias. For example, reviewers may be less likely to criticize work that is consistent with their own perceptions (Ernst and Resch, 1994) or to award a fellowship to a woman rather than a man (Wennerds and Wold, 1997). It is also important to keep in mind that peer review does not do well either at detecting innovative research or filtering out fraudulent, plagiarized, or redundant publications (reviewed by Godlee, 2000).
Despite its flaws, peer review does work to improve the quality of research. Considering the possible failings of peer review, the potential for bias and abuse, how can the process be managed so as to minimize problems while maintaining the advantages?
Regulations and Guidelines
Most organizations reviewing research have specific guidelines regarding confidentiality and conflicts of interest. In addition, many organizations and institutions have guidelines dealing explicitly with the responsibilities of peer reviewers, such as those of the American Chemical Society (2006), the Society for Neuroscience (1998, and the Council of Biology Editors (CBE Peer Review Retreat Consensus Group, 1995).And, though currently suspended, there had been a federal requirement that made discussion of peer review part of instruction in the responsible conduct of research (Office of Research Integrity, 2000).
Peer review is governed by federal regulations in two respects. First, federal misconduct regulations can be invoked if a reviewer seriously abuses the review process, and second, peer review for the grant process prohibits review by individuals with conflicts of interest.
Despite these regulations, much of peer review is not directly regulated. It is governed instead by guidelines and custom.
Discussion
Case Studies
Discussion Questions
Based on your own experience, or on discussion with someone who is an experienced reviewer, which of the following are common practice? Which of the following should not be acceptable practice?
a. The reviewer is not competent to perform a review, but does so anyway.
b. Reviewer bias results in a negative review that is misleading or untruthful.
c. The reviewer delays the review or provides an unfairly critical review for the purpose of personal advantage.
d. The reviewer and his or her research group take advantage of privileged information to redirect research efforts.
e. The reviewer shares review material with others (for the purpose of training or scientific discussion) without notifying or obtaining approval from the editor or funding agency.
What are the advantages and disadvantages of having a reviewer blinded to the identity of manuscript authors, a grant applicant, or a candidate for academic advancement?
What are the advantages and disadvantages of having manuscript authors, a grant applicant, or a candidate for academic advancement blinded to the identity of a reviewer?
What are the ethical responsibilities of peer reviewers?
List and describe federal regulations relevant to peer review.
Should reviewers working in the same field of research be excluded from reviewing each others' work? How can the risks of bias and the advantages of expertise be reconciled in the selection of peer reviewers?
What are the responsibilities of a reviewer to preserve the confidentiality of work under review? What protections, if any, help to prevent the loss of confidentiality?
Additional Considerations
The purpose of peer review is not merely to evaluate the submitted work, but also to promote better work within the scientific community. As such, there are several essential responsibilities for peer reviews.
Provide a timely response
Reviewers should make every effort to complete a review in the time requested. If it is not possible to meet the conditions for the review, then the reviewer should promptly decline or see if some accommodation is possible. Research reports, grant applications, and academic files submitted for review all represent a significant investment of time and effort, and frequently the documents under review contain timely results that will suffer if delayed in the review process.
Ensure Competence
Reviewers who realize that their expertise is limited have a responsibility to make their degree of competence clear to the editor, funding agency, or academic institution asking for their expert opinion. A reviewer who does not have the requisite expertise is at risk of approving a submission that has substantial deficiencies or rejecting one that is meritorious. Such errors are a waste of resources and hamper the scientific enterprise.
Avoid Bias
Reviewers' comments and conclusions should be based on a consideration of the facts, exclusive of personal or professional bias. To the extent possible, the system of review should be designed to minimize actual or perceived bias on the reviewers' part. If reviewers have any interest that might interfere with an objective review, then they should either decline a role as reviewer or declare the conflict of interest to the editor, funding agency, or academic institution and ask how best to manage the conflict.
Maintain Confidentiality
Material submitted for peer review is a privileged communication that should be treated in confidence. Material under review should not be shared or discussed with anyone outside the designated review process unless approved by the editor, funding agency, or academic institution. Authors, grant applicants, and candidates for academic review have a right to expect that the review process will remain confidential. Reviewers unsure about policies for enlisting the help of others should ask.
Avoid unfair advantage
A reviewer should not take advantage of material available through the privileged communication of peer review. One exception is that if a reviewers becomes aware on the basis of work under review that a line of her or his own research is likely to be unprofitable or a waste of resources, then they may ethically discontinue that work (American Chemical Society, 2006; Society for Neuroscience, 1998. In such cases, the circumstances should be communicated to those who requested the review.
Beyond this exception, every effort should be made to avoid even the appearance of taking advantage of information obtained through the review process. Potential reviewers concerned that their participation would be a substantial conflict of interest should decline the request to review.
Offer constructive criticism
Reviewers' comments should acknowledge positive aspects of the material under review, assess negative aspects constructively, and indicate clearly the improvements needed. The purpose of peer review is not to demonstrate the reviewer's proficiency in identifying flaws, but to help the authors or candidates identify and resolve weaknesses in their work.
Resources
Abby M, Massey MD, Galandiuk S, Polk HC (1994): Peer review is an effective screening process to evaluate medical manuscripts. JAMA 272: 105-107.
CBE Peer Review Retreat Consensus Group (1995): Peer review guidelines - A working draft. CBE Views 18(5): 79-81.
Ernst E, Resch KL (1994): Reviewer bias: a blinded experimental study. Journal of Laboratory and Clinical Medicine 124(2): 178-82.
Garfunkel JM, Ulshen MH, Hamrick HJ, Lawson EE (1994): Effect of institutional prestige on reviewers' recommendations and editorial decisions. JAMA 272: 137-138.
Godlee F, Gale CR, Martyn CN (1998): Effect on the quality of peer review of blinding reviewers and asking them to sign their reports. JAMA 280: 237-240.
Godlee F (2000): The ethics of peer review. In (Jones AH, McLellan F, eds.): Ethical Issues in Biomedical Publication. Johns Hopkins University Press, Baltimore, MD, pp. 59-84.
Justice AC, Cho MK, Winker MA, Berlin JA, Rennie D, PEER Investigators (1998): Does masking author identity improve peer review quality? JAMA 280: 240-242.