Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. Effects of training on quality of peer review: randomised controlled trial. Comparison of review articles published in peer-reviewed and throwaway journals. Readers' evaluation of effect of peer review and editing on quality of articles in the Nederlands Tijdschrift voor Geneeskunde. The Medical Journal of Australia Internet peer-review study. Effect of open peer review on quality of reviews and on reviewers' recommendations: a randomised trial. Open peer review: a randomised controlled trial. Publication bias and meta-analyses: a practical example. Does masking author identity improve peer review quality? A randomized controlled trial. PEER Investigators.
Anxiety about bias, lack of accountability, and poor quality of peer review has led to questions about the imbalance in anonymity between reviewers and authors.
                To evaluate the effect on the quality of peer review of blinding reviewers to the authors' identities and requiring reviewers to sign their reports.
                Randomized controlled trial.
                A general medical journal.
                A total of 420 reviewers from the journal's database.
                We modified a paper accepted for publication introducing 8 areas of weakness. Reviewers were randomly allocated to 5 groups. Groups 1 and 2 received manuscripts from which the authors' names and affiliations had been removed, while groups 3 and 4 were aware of the authors' identities. Groups 1 and 3 were asked to sign their reports, while groups 2 and 4 were asked to return their reports unsigned. The fifth group was sent the paper in the usual manner of the journal, with authors' identities revealed and a request to comment anonymously. Group 5 differed from group 4 only in that its members were unaware that they were taking part in a study.
                The number of weaknesses in the paper that were commented on by the reviewers.
                Reports were received from 221 reviewers (53%). The mean number of weaknesses commented on was 2 (1.7, 2.1, 1.8, and 1.9 for groups 1, 2, 3, and 4 and 5 combined, respectively). There were no statistically significant differences between groups in their performance. Reviewers who were blinded to authors' dentities were less likely to recommend rejection than those who were aware of the authors' identities (odds ratio, 0.5; 95% confidence interval, 0.3-1.0).
                Neither blinding reviewers to the authors and origin of the paper nor requiring them to sign their reports had any effect on rate of detection of errors. Such measures are unlikely to improve the quality of peer review reports. To determine the effects of training on the quality of peer review.
                Single blind randomised controlled trial with two intervention groups receiving different types of training plus a control group.
                Reviewers at a general medical journal. Interventions Attendance at a training workshop or reception of a self taught training package focusing on what editors want from reviewers and how to critically appraise randomised controlled trials.
                Quality of reviews of three manuscripts sent to reviewers at four to six monthly intervals, evaluated using the validated review quality instrument; number of deliberate major errors identified; time taken to review the manuscripts; proportion recommending rejection of the manuscripts.
                Reviewers in the self taught group scored higher in review quality after training than did the control group (score 2.85 v 2.56; difference 0.29, 95% confidence interval 0.14 to 0.44; P = 0.001), but the difference was not of editorial significance and was not maintained in the long term. Both intervention groups identified significantly more major errors after training than did the control group (3.14 and 2.96 v 2.13; P < 0.001), and this remained significant after the reviewers' performance at baseline assessment was taken into account. The evidence for benefit of training was no longer apparent on further testing six months after the interventions. Training had no impact on the time taken to review the papers but was associated with an increased likelihood of recommending rejection (92% and 84% v 76%; P = 0.002).
                Short training packages have only a slight impact on the quality of peer review. The value of longer interventions needs to be assessed. To compare the quality, presentation, readability, and clinical relevance of review articles published in peer-reviewed and "throwaway" journals.
                We reviewed articles that focused on the diagnosis or treatment of a medical condition published between January 1 and December 31, 1998, in the 5 leading peer-reviewed general medical journals and high-circulation throwaway journals. Reviewers independently assessed the methodologic and reporting quality, and evaluated each article's presentation and readability. Clinical relevance was evaluated independently by 6 physicians.
                Of the 394 articles in our sample, 16 (4.1%) were peer-reviewed systematic reviews, 135 (34.3%) were peer-reviewed nonsystematic reviews, and 243 (61.7%) were nonsystematic reviews published in throwaway journals. The mean (SD) quality scores were highest for peer-reviewed articles (0.94 [0.09] for systematic reviews and 0.30 [0.19] for nonsystematic reviews) compared with throwaway journal articles (0.23 [0.03], F(2,391) = 280.8, P<.001). Throwaway journal articles used more tables (P =.02), figures (P =.01), photographs (P<.001), color (P<.001), and larger font sizes (P<.001) compared with peer-reviewed articles. Readability scores were more often in the college or higher range for peer-reviewed journals compared with the throwaway journal articles (104 [77.0%] vs 156 [64.2%]; P =.01). Peer-reviewed article titles were judged less relevant to clinical practice than throwaway journal article titles (P<.001).
                Although lower in methodologic and reporting quality, review articles published in throwaway journals have characteristics that appeal to physician readers. Academic biomedical journals use peer review and editing to help to select and improve the quality of articles. We have investigated whether articles accepted by the Nederlands Tijdschrift voor Geneeskunde, the Dutch Journal of Medicine, were improved after peer review and editing (post-acceptance scientific and copy editing).
                400 readers of the journal (100 each of medical students, recent medical graduates, general practitioners, and specialists) were invited to participate in a questionnaire survey. The first 25 from each group who agreed to participate were included. We posted a pack containing a set of identically appearing typescripts (ie, blinding) of the submitted, accepted, and published versions of 50 articles that had been published in Ned Tijdschr Geneeskd. Each evaluator received two of the sets of versions, and each set was evaluated by one person from each group. The package also included two questionnaires: the first was used to compare the submitted with the accepted version (25 questions), the second compared the accepted with the published version (17 questions). The questions were answered on five-point scales, and were about the quality of the articles or were general/overall scores. We analysed the data as scores of 3-5 (ie, improvement) versus 1-2.
                After peer review, the quality in 14 of 23 questions (61%) was significantly improved (p = 0.03 or smaller). In particular, the overall score and general medical value were significantly improved (p = 0.00001 for each). Editing led to significant improvement in 11 of 16 questions (69%, p = 0.017 or smaller), and especially in style and readability (p = 0.001 and p = 0.004). Generally, we found no differences between the scores of the four categories of evaluators. 72% of the evaluators correctly identified which version was which.
                Evaluations by readers of the Ned Tijdschr Geneeskd indicated significant improvement of published articles after both peer review and editing. We think that peer review and editing are worthwhile tasks. We also think that possible biases would have had a negligible effect on our results (including the fact that we selected the first 25 evaluators who responded, that some evaluators may have read the published version, and that one questionnaire may have looked more scientific than the other, more editorial one). Peer review of medical papers is a confidential consultancy between the reviewer and the journal editor, and has been criticised for its potential bias and inadequacy. We explored the potential of the internet for open peer review to see whether this approach improved the quality and outcome of peer review.
                Research and review articles that had been accepted for publication in The Medical Journal of Australia (MJA) were published together with the reviewers' reports on the worldwide web, with the consent of authors and referees. Selected readers' e-mailed comments were electronically published as additional commentary; authors could reply or revise their paper in response to readers' comments. Articles were edited and published in print after this open review.
                60 (81%) of 74 authors agreed to take part in the study, together with 150 (92%) of 162 reviewers. There was no significant difference in the performance of commissioned reviewers before and during the study. Four articles were not included because of insufficient time before print publication. Of the remaining 56 papers, 28 received 52 comments from 42 readers (2% of readers submitted comments). Most readers' comments were short and specific, and seven articles were changed by the authors in response.
                Open peer review is acceptable to most authors and reviewers. Postpublication review by readers on the internet is no substitute for commissioned prepublication review, but can provide editors with valuable input from individuals who would not otherwise be consulted. Readers also gain insight into the processes of peer review and publication. To examine the effect on peer review of asking reviewers to have their identity revealed to the authors of the paper.
                Randomised trial. Consecutive eligible papers were sent to two reviewers who were randomised to have their identity revealed to the authors or to remain anonymous. Editors and authors were blind to the intervention.
                The quality of the reviews was independently rated by two editors and the corresponding author using a validated instrument. Additional outcomes were the time taken to complete the review and the recommendation regarding publication. A questionnaire survey was undertaken of the authors of a cohort of manuscripts submitted for publication to find out their views on open peer review.
                Two editors' assessments were obtained for 113 out of 125 manuscripts, and the corresponding author's assessment was obtained for 105. Reviewers randomised to be asked to be identified were 12% (95% confidence interval 0.2% to 24%) more likely to decline to review than reviewers randomised to remain anonymous (35% v 23%). There was no significant difference in quality (scored on a scale of 1 to 5) between anonymous reviewers (3.06 (SD 0.72)) and identified reviewers (3.09 (0.68)) (P=0.68, 95% confidence interval for difference - 0.19 to 0.12), and no significant difference in the recommendation regarding publication or time taken to review the paper. The editors' quality score for reviews (3.05 (SD 0.70)) was significantly higher than that of authors (2.90 (0.87)) (P<0.005, 95%confidence interval for difference - 0.26 to - 0.03). Most authors were in favour of open peer review.
                Asking reviewers to consent to being identified to the author had no important effect on the quality of the review, the recommendation regarding publication, or the time taken to review, but it significantly increased the likelihood of reviewers declining to review. Most scientific journals practise anonymous peer review. There is no evidence, however, that this is any better than an open system.
                To evaluate the feasibility of an open peer review system.
                Reviewers for the British Journal of Psychiatry were asked whether they would agree to have their name revealed to the authors whose papers they review; 408 manuscripts assigned to reviewers who agreed were randomised to signed or unsigned groups. We measured review quality, tone, recommendation for publication and time taken to complete each review.
                A total of 245 reviewers (76%) agreed to sign. Signed reviews were of higher quality, were more courteous and took longer to complete than unsigned reviews. Reviewers who signed were more likely to recommend publication.
                This study supports the feasibility of an open peer review system and identifies such a system's potential drawbacks. Publication bias is widely appreciated, but considerable time and effort are needed to locate and obtain data from unpublished randomized controlled trials (RCTs), those published in non-English language journals or those reported in the gray literature; for this publication, we will call this collection of trials the "gray+literature." However, excluding such trials from systematic reviews could introduce bias and give rise to misleading conclusions.
                We aimed to explore and quantify the impact of inclusion of gray+ literature on the results of all completed individual patient data (IPD) reviews coordinated by our group (13 meta-analyses). For each IPD review, results were calculated for RCTs fully published in English language journals and RCTs fully published in English language journals and the gray+literature.
                The IPD meta-analyses based only on RCTs that were fully published in English language journals tended to give more favorable results than those that included RCTs from the gray+literature. Although in most cases the addition of gray+data gave less encouraging results, moving the estimated treatment effect toward a null result, the direction of effect was not always predictable.
                We recommend that all systematic reviews should at least attempt to identify trials reported in the gray+literature and, where possible, obtain data from them. All authors may not be equal in the eyes of reviewers. Specifically, well-known authors may receive less objective (poorer quality) reviews. One study at a single journal found a small improvement in review quality when reviewers were masked to author identity.
                To determine whether masking reviewers to author identity is generally associated with higher quality of review at biomedical journals, and to determine the success of routine masking techniques.
                A randomized controlled trial performed on external reviews of manuscripts submitted to Annals of Emergency Medicine, Annals of Internal Medicine, JAMA, Obstetrics & Gynecology, and Ophthalmology.
                Two peers reviewed each manuscript. In one study arm, both peer reviewers received the manuscript according to usual masking practice. In the other arm, one reviewer was randomized to receive a manuscript with author identity masked, and the other reviewer received an unmasked manuscript.
                Review quality on a 5-point Likert scale as judged by manuscript author and editor. A difference of 0.5 or greater was considered important.
                A total of 118 manuscripts were randomized, 26 to usual practice and 92 to intervention. In the intervention arm, editor quality assessment was complete for 77 (84%) of 92 manuscripts. Author quality assessment was complete on 40 (54%) of 74 manuscripts. Authors and editors perceived no significant difference in quality between masked (mean difference, 0.1; 95% confidence interval [CI], -0.2 to 0.4) and unmasked (mean difference, -0.1; 95% CI, -0.5 to 0.4) reviews. We also found no difference in the degree to which the review influenced the editorial decision (mean difference, -0.1; 95% CI,-0.3 to 0.3). Masking was often unsuccessful (overall, 68% successfully masked; 95% CI, 58%-77%), although 1 journal had significantly better masking success than others (90% successfully masked; 95% CI, 73%-98%). Manuscripts by generally known authors were less likely to be successfully masked (odds ratio, 0.3; 95% CI, 0.1-0.8). When analysis was restricted to manuscripts that were successfully masked, review quality as assessed by editors and authors still did not differ.
                Masking reviewers to author identity as commonly practiced does not improve quality of reviews. Since manuscripts of well-known authors are more difficult to mask, and those manuscripts may be more likely to benefit from masking, the inability to mask reviewers to the identity of well-known authors may have contributed to the lack of effect.
At present, little empirical evidence is available to support the use of editorial peer review as a mechanism to ensure quality of biomedical research. However, the methodological problems in studying peer review are many and complex. At present, the absence of evidence on efficacy and effectiveness cannot be interpreted as evidence of their absence. A large, well-funded programme of research on the effects of editorial peer review should be urgently launched.