Assessment of risk of bias among pediatric randomized controlled trials. Balneotherapy and quality assessment: interobserver reliability of the Maastricht criteria list and the need for blinded quality assessment. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. The effects of blinding on acceptance of research papers by peer review. Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. Publication bias and meta-analyses: a practical example. Effect of blinding and unmasking on the quality of peer review: a randomized trial.
The goal was to assess the risk of bias among pediatric, randomized, controlled trials (RCTs) reported in 8 high-impact journals.
                We searched PubMed for all pediatric RCTs reported between July 1, 2007, and June 30, 2008, in 8 journals with high impact factors. Using Cochrane Collaboration methods for risk assessment, we evaluated all reports for risk of bias according to domain (ie, randomized sequence generation, allocation concealment, masking, incomplete outcome data, selective outcome reporting, and other). We used multiple logistic regression to test for associations between the presence of a high risk of bias according to domain and funding source, intervention type, trial registration, and multicenter status.
                Industry-funded RCTs were more likely to show a high risk of bias for sequence generation, compared with government-funded RCTs (adjusted odds ratio [aOR]: 6.1 [95% confidence interval [CI]: 1.70- 21.89]), and behavioral/educational trials were more likely to show a high risk of bias for sequence generation (aOR: 2.8 [95% CI: 1.06-7.36]) and allocation concealment (aOR: 4.09 [95% CI: 1.69-9.90]), compared with drug trials. Registered trials were less likely to have a high risk of bias for sequence generation, compared with nonregistered trials (aOR: 0.33 [95% CI: 0.15-0.71]).
                Overall, we found a large proportion of pediatric RCT reports with a high risk of bias for sequence generation and allocation concealment. Factors associated with a high risk of bias included industry funding and assessment of behavioral/educational interventions, whereas trial registration was associated with a lower risk of bias. This study investigates aspects of the reliability of the Maastricht criteria list for quality assessment in systematic reviews, and whether blinded reviewing is necessary to prevent review bias. We used the data set of 12 articles from a systematic review concerning the efficacy of balneotherapy in patients with arthritis. Twenty reviewers participated of which two reviewers, who have been involved in developing the Maastricht criteria list, acted as reference standard. Half of all assessments were performed blindly. A high level of agreement was found between the reviewers and a high level of correlation with the reference standard. The quality scores between the blinded and unblinded assessment did not differ much. Based on the results we conclude that the Maastricht criteria list is a reliable instrument in quality assessment of clinical trials. Within the limits of this study we found no evidence that blinding is necessary to prevent review bias. It has been suggested that the quality of clinical trials should be assessed by blinded raters to limit the risk of introducing bias into meta-analyses and systematic reviews, and into the peer-review process. There is very little evidence in the literature to substantiate this. This study describes the development of an instrument to assess the quality of reports of randomized clinical trials (RCTs) in pain research and its use to determine the effect of rater blinding on the assessments of quality. A multidisciplinary panel of six judges produced an initial version of the instrument. Fourteen raters from three different backgrounds assessed the quality of 36 research reports in pain research, selected from three different samples. Seven were allocated randomly to perform the assessments under blind conditions. The final version of the instrument included three items. These items were scored consistently by all the raters regardless of background and could discriminate between reports from the different samples. Blind assessments produced significantly lower and more consistent scores than open assessments. The implications of this finding for systematic reviews, meta-analytic research and the peer-review process are discussed. To determine if inadequate approaches to randomized controlled trial design and execution are associated with evidence of bias in estimating treatment effects.
                An observational study in which we assessed the methodological quality of 250 controlled trials from 33 meta-analyses and then analyzed, using multiple logistic regression models, the associations between those assessments and estimated treatment effects.
                Meta-analyses from the Cochrane Pregnancy and Childbirth Database.
                The associations between estimates of treatment effects and inadequate allocation concealment, exclusions after randomization, and lack of double-blinding.
                Compared with trials in which authors reported adequately concealed treatment allocation, trials in which concealment was either inadequate or unclear (did not report or incompletely reported a concealment approach) yielded larger estimates of treatment effects (P < .001). Odds ratios were exaggerated by 41% for inadequately concealed trials and by 30% for unclearly concealed trials (adjusted for other aspects of quality). Trials in which participants had been excluded after randomization did not yield larger estimates of effects, but that lack of association may be due to incomplete reporting. Trials that were not double-blind also yielded larger estimates of effects (P = .01), with odds ratios being exaggerated by 17%.
                This study provides empirical evidence that inadequate methodological approaches in controlled trials, particularly those representing poor allocation concealment, are associated with bias. Readers of trial reports should be wary of these pitfalls, and investigators must improve their design, execution, and reporting of trials. To study whether reviewers aware of author identity are biased in favor of authors with more previous publications.
                Randomized controlled trial.
                Editorial office of the Journal of Developmental and Behavioral Pediatrics.
                Two "blinded" and two "nonblinded" reviewers assigned to 57 consecutive manuscripts submitted between September 1991 and March 1992.
                Spearman rank correlation coefficients were used to compare the sum of rating scores of 1 to 5 (1, accept; 5, reject) given by the two blinded reviewers, the two nonblinded reviewers, and the editors to the number of articles published previously by the first and senior authors (as determined from requested curricula vitae). Blinded reviewers were sent a questionnaire asking whether they could determine the identity of the authors, how they knew, and whether they thought binding changed the quality or difficulty of their review.
                The Wilcoxon Sign Rank Test disclosed no differences between blinded and nonblinded scores. The number of previous articles by the senior author was significantly correlated (P < .01) with blinded scores (r = -.45) and editors' decisions (r = -.45), but not with nonblinded scores; the number of articles by the first author was correlated (P < .05) with editors' decisions (r = -.35) but not with blinded or nonblinded scores. Fifty (46%) of 108 blinded reviewers correctly guessed the identity of the authors, mostly from self-references and knowledge of the work; 86% believed blinding did not change the quality of their review, and 73% believed it did not change the difficulty of performing a review.
                Blinded reviewers and editors in this study, but not nonblinded reviewers, gave better scores to authors with more previous articles. These results suggest that blinded reviewers may provide more unbiased reviews and that nonblinded reviewers may be affected by various types of bias. To explore whether reported methodologic quality affects estimated intervention effects in randomized trials and contributes to discrepancies between the results of large randomized trials and small randomized trials in meta-analyses.
                Meta-analyses of randomized trials that included at least one large trial (>/=1000 participants) were included, regardless of the therapeutic area. Eligible meta-analyses were identified through electronic searches and bibliographies of relevant articles.
                Full-length randomized trials.
                Methodologic quality was assessed according to reported randomization, double blinding, and follow-up as separate components and by using the Jadad composite scale.
                Fourteen meta-analyses involving 190 randomized trials from eight therapeutic areas were included. Compared with large trials, intervention effects were exaggerated in small trials with inadequate allocation sequence generation (ratio of odds ratios, 0.46 [95% CI, 0.25 to 0.83]; P = 0.011), inadequate allocation concealment (ratio of odds ratios, 0.49 [CI, 0.27 to 0.86]; P = 0.014), and no double blinding (ratio of odds ratios, 0.52 [CI, 0.28 to 0.96]; P = 0.01). Large trials did not differ significantly from small trials with adequate generation of the allocation sequence, adequate allocation concealment, or adequate double blinding. No association was seen between reported follow-up and intervention effects. The Jadad scale provided no additional information because the scale and the quality components overlapped substantially.
                Inadequate generation of the allocation sequence, allocation concealment, and double blinding lead to exaggerated estimates of intervention benefit and may contribute to discrepancies between the results of large randomized trials and small randomized trials in meta-analyses. Selective reporting of outcomes within published studies based on the nature or direction of their results has been widely suspected, but direct evidence of such bias is currently limited to case reports.
                To study empirically the extent and nature of outcome reporting bias in a cohort of randomized trials.
                Cohort study using protocols and published reports of randomized trials approved by the Scientific-Ethical Committees for Copenhagen and Frederiksberg, Denmark, in 1994-1995. The number and characteristics of reported and unreported trial outcomes were recorded from protocols, journal articles, and a survey of trialists. An outcome was considered incompletely reported if insufficient data were presented in the published articles for meta-analysis. Odds ratios relating the completeness of outcome reporting to statistical significance were calculated for each trial and then pooled to provide an overall estimate of bias. Protocols and published articles were also compared to identify discrepancies in primary outcomes.
                Completeness of reporting of efficacy and harm outcomes and of statistically significant vs nonsignificant outcomes; consistency between primary outcomes defined in the most recent protocols and those defined in published articles.
                One hundred two trials with 122 published journal articles and 3736 outcomes were identified. Overall, 50% of efficacy and 65% of harm outcomes per trial were incompletely reported. Statistically significant outcomes had a higher odds of being fully reported compared with nonsignificant outcomes for both efficacy (pooled odds ratio, 2.4; 95% confidence interval [CI], 1.4-4.0) and harm (pooled odds ratio, 4.7; 95% CI, 1.8-12.0) data. In comparing published articles with protocols, 62% of trials had at least 1 primary outcome that was changed, introduced, or omitted. Eighty-six percent of survey responders (42/49) denied the existence of unreported outcomes despite clear evidence to the contrary.
                The reporting of trial outcomes is not only frequently incomplete but also biased and inconsistent with protocols. Published articles, as well as reviews that incorporate them, may therefore be unreliable and overestimate the benefits of an intervention. To ensure transparency, planned trials should be registered and protocols should be made publicly available prior to trial completion. Publication bias is widely appreciated, but considerable time and effort are needed to locate and obtain data from unpublished randomized controlled trials (RCTs), those published in non-English language journals or those reported in the gray literature; for this publication, we will call this collection of trials the "gray+literature." However, excluding such trials from systematic reviews could introduce bias and give rise to misleading conclusions.
                We aimed to explore and quantify the impact of inclusion of gray+ literature on the results of all completed individual patient data (IPD) reviews coordinated by our group (13 meta-analyses). For each IPD review, results were calculated for RCTs fully published in English language journals and RCTs fully published in English language journals and the gray+literature.
                The IPD meta-analyses based only on RCTs that were fully published in English language journals tended to give more favorable results than those that included RCTs from the gray+literature. Although in most cases the addition of gray+data gave less encouraging results, moving the estimated treatment effect toward a null result, the direction of effect was not always predictable.
                We recommend that all systematic reviews should at least attempt to identify trials reported in the gray+literature and, where possible, obtain data from them. Little research has been conducted into the quality of peer review and, in particular, the effects of blinding peer reviewers to authors' identities or masking peer reviewers' identities.
                To determine whether concealing authors' identities from reviewers (blinding) and/or revealing the reviewer's identity to a coreviewer (unmasking) affects the quality of reviews, the time taken to carry out reviews, and the recommendation regarding publication.
                Randomized trial of 527 consecutive manuscripts submitted to BMJ, which were randomized and each sent to 2 peer reviewers.
                Manuscripts were randomized as to whether the reviewers were unmasked, masked, or uninformed that a study was taking place. Two reviewers for each manuscript were randomized to receive either a blinded or an unblinded version.
                Mean total quality score, time taken to carry out the review, and recommendation regarding publication.
                Of the 527 manuscripts entered into the study, 467 (89%) were successfully randomized and followed up. The mean total quality score was 2.87. There was little or no difference in review quality between the masked and unmasked groups (scores of 2.82 and 2.96, respectively) and between the blinded and unblinded groups (scores of 2.87 and 2.90, respectively). There was no apparent Hawthorne effect. There was also no significant difference between groups in the recommendations regarding publication or time taken to review.
                Blinding and unmasking made no editorially significant difference to review quality, reviewers' recommendations, or time taken to review. Other considerations should guide decisions as to the form of peer review adopted by a journal, and improvements in the quality of peer review should be sought via other means.
Our review highlights that discordance exists between studies examining blinded versus unblinded risk of bias assessments at the systematic review level. The best approach to risk of bias assessment remains unclear, however, given the increased time and resources required to conceal reports effectively, it may not be necessary for risk of bias assessments to be conducted under blinded conditions in a systematic review.