Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. Impact of study quality on outcome in placebo-controlled trials of homeopathy. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. The reporting of methodological factors in randomized controlled trials and the association with a journal policy to promote adherence to the Consolidated Standards of Reporting Trials (CONSORT) checklist. Quality of randomized controlled trials reporting in the primary treatment of brain tumors. Evidence of improving quality of reporting of randomized controlled trials in subfertility. A cohort study of summary reports of controlled trials.
To explore whether reported methodologic quality affects estimated intervention effects in randomized trials and contributes to discrepancies between the results of large randomized trials and small randomized trials in meta-analyses.
                Meta-analyses of randomized trials that included at least one large trial (>/=1000 participants) were included, regardless of the therapeutic area. Eligible meta-analyses were identified through electronic searches and bibliographies of relevant articles.
                Full-length randomized trials.
                Methodologic quality was assessed according to reported randomization, double blinding, and follow-up as separate components and by using the Jadad composite scale.
                Fourteen meta-analyses involving 190 randomized trials from eight therapeutic areas were included. Compared with large trials, intervention effects were exaggerated in small trials with inadequate allocation sequence generation (ratio of odds ratios, 0.46 [95% CI, 0.25 to 0.83]; P = 0.011), inadequate allocation concealment (ratio of odds ratios, 0.49 [CI, 0.27 to 0.86]; P = 0.014), and no double blinding (ratio of odds ratios, 0.52 [CI, 0.28 to 0.96]; P = 0.01). Large trials did not differ significantly from small trials with adequate generation of the allocation sequence, adequate allocation concealment, or adequate double blinding. No association was seen between reported follow-up and intervention effects. The Jadad scale provided no additional information because the scale and the quality components overlapped substantially.
                Inadequate generation of the allocation sequence, allocation concealment, and double blinding lead to exaggerated estimates of intervention benefit and may contribute to discrepancies between the results of large randomized trials and small randomized trials in meta-analyses. To determine if inadequate approaches to randomized controlled trial design and execution are associated with evidence of bias in estimating treatment effects.
                An observational study in which we assessed the methodological quality of 250 controlled trials from 33 meta-analyses and then analyzed, using multiple logistic regression models, the associations between those assessments and estimated treatment effects.
                Meta-analyses from the Cochrane Pregnancy and Childbirth Database.
                The associations between estimates of treatment effects and inadequate allocation concealment, exclusions after randomization, and lack of double-blinding.
                Compared with trials in which authors reported adequately concealed treatment allocation, trials in which concealment was either inadequate or unclear (did not report or incompletely reported a concealment approach) yielded larger estimates of treatment effects (P < .001). Odds ratios were exaggerated by 41% for inadequately concealed trials and by 30% for unclearly concealed trials (adjusted for other aspects of quality). Trials in which participants had been excluded after randomization did not yield larger estimates of effects, but that lack of association may be due to incomplete reporting. Trials that were not double-blind also yielded larger estimates of effects (P = .01), with odds ratios being exaggerated by 17%.
                This study provides empirical evidence that inadequate methodological approaches in controlled trials, particularly those representing poor allocation concealment, are associated with bias. Readers of trial reports should be wary of these pitfalls, and investigators must improve their design, execution, and reporting of trials. We investigated the influence of indicators of methodological quality on study outcome in a set of 89 placebo-controlled clinical trials of homoeopathy in three different ways: (1) The results of studies meeting single criteria (explicit statement of random allocation, allocation concealment, double-blinding, completeness of follow-up) of methodological quality were compared with those of studies not meeting the criteria in univariate and multivariate analyses; (2) The results of studies scoring above and below predefined scores in two quality assessment scales were compared; (3) Primary studies were consecutively entered into a cumulative meta-analysis according to the summary scores derived from the quality assessment scales. All analyses were performed using meta-regression methods. Studies that were explicitly randomized and were double-blind as well as studies scoring above the cut-points yielded significantly less positive results than studies not meeting the criteria. In the cumulative meta-analyses, there was a trend for increasing effect sizes when more studies with lower-quality scores were added. However, there was no linear relationship between quality scores and study outcome. We conclude that in the study set investigated, there was clear evidence that studies with better methodological quality tended to yield less positive results. Because summarizing disparate study features into a single score is problematic, meta-regression methods simultaneously investigating the influence of single study features seem the best method for investigating the impact of study quality on outcome. Few meta-analyses of randomised trials assess the quality of the studies included. Yet there is increasing evidence that trial quality can affect estimates of intervention efficacy. We investigated whether different methods of quality assessment provide different estimates of intervention efficacy evaluated in randomised controlled trials (RCTs).
                We randomly selected 11 meta-analyses that involved 127 RCTs on the efficacy of interventions used for circulatory and digestive diseases, mental health, and pregnancy and childbirth. We replicated all the meta-analyses using published data from the primary studies. The quality of reporting of all 127 clinical trials was assessed by means of component and scale approaches. To explore the effects of quality on the quantitative results, we examined the effects of different methods of incorporating quality scores (sensitivity analysis and quality weights) on the results of the meta-analyses.
                The quality of trials was low. Masked assessments provided significantly higher scores than unmasked assessments (mean 2.74 [SD 1.10] vs 2.55 [1.20]). Low-quality trials (score < or = 2), compared with high-quality trials (score > 2), were associated with an increased estimate of benefit of 34% (ratio of odds ratios [ROR] 0.66 [95% CI 0.52-0.83]). Trials that used inadequate allocation concealment, compared with those that used adequate methods, were also associated with an increased estimate of benefit (37%; ROR=0.63 [0.45-0.88]). The average treatment benefit was 39% (odds ratio [OR] 0.61 [0.57-0.65]) for all trials, 52% (OR 0.48 [0.43-0.54]) for low-quality trials, and 29% (OR 0.71 [0.65-0.77]) for high-quality trials. Use of all the trial scores as quality weights reduced the effects to 35% (OR 0.65 [0.59-0.71]) and resulted in the least statistical heterogeneity.
                Studies of low methodological quality in which the estimate of quality is incorporated into the meta-analyses can alter the interpretation of the benefit of intervention, whether a scale or component approach is used in the assessment of trial quality. Selective reporting of outcomes within published studies based on the nature or direction of their results has been widely suspected, but direct evidence of such bias is currently limited to case reports.
                To study empirically the extent and nature of outcome reporting bias in a cohort of randomized trials.
                Cohort study using protocols and published reports of randomized trials approved by the Scientific-Ethical Committees for Copenhagen and Frederiksberg, Denmark, in 1994-1995. The number and characteristics of reported and unreported trial outcomes were recorded from protocols, journal articles, and a survey of trialists. An outcome was considered incompletely reported if insufficient data were presented in the published articles for meta-analysis. Odds ratios relating the completeness of outcome reporting to statistical significance were calculated for each trial and then pooled to provide an overall estimate of bias. Protocols and published articles were also compared to identify discrepancies in primary outcomes.
                Completeness of reporting of efficacy and harm outcomes and of statistically significant vs nonsignificant outcomes; consistency between primary outcomes defined in the most recent protocols and those defined in published articles.
                One hundred two trials with 122 published journal articles and 3736 outcomes were identified. Overall, 50% of efficacy and 65% of harm outcomes per trial were incompletely reported. Statistically significant outcomes had a higher odds of being fully reported compared with nonsignificant outcomes for both efficacy (pooled odds ratio, 2.4; 95% confidence interval [CI], 1.4-4.0) and harm (pooled odds ratio, 4.7; 95% CI, 1.8-12.0) data. In comparing published articles with protocols, 62% of trials had at least 1 primary outcome that was changed, introduced, or omitted. Eighty-six percent of survey responders (42/49) denied the existence of unreported outcomes despite clear evidence to the contrary.
                The reporting of trial outcomes is not only frequently incomplete but also biased and inconsistent with protocols. Published articles, as well as reviews that incorporate them, may therefore be unreliable and overestimate the benefits of an intervention. To ensure transparency, planned trials should be registered and protocols should be made publicly available prior to trial completion. The "Consolidated Standards of Reporting Trials" (CONSORT) was developed to improve the suboptimal reporting of randomized controlled trials (RCTs). However, little is known about the quality of reporting since this publication. We undertook an observational study to determine the quality of reporting key methodological factors in RCTs since the publication of the CONSORT statement and if a journal policy to promote adherence to the CONSORT checklist was associated with superior reporting. We recorded the reporting of 11 key methodological factors in 105 RCTs from 29 medical journals published subsequent to the CONSORT statement. We examined the quality of reporting in relation to whether a journal was a "CONSORT promoter" as defined by inclusion of the CONSORT checklist in a journal's "information to authors" section or a requirement that authors, manuscript reviewers, or copy editors complete the CONSORT checklist. Multivariate analysis controlled for journal impact factor, study outcome, and time of publication. Six of the 11 methodological factors were reported <50% of the time. The number of methodological factors reported was greater in CONSORT promoters than in journals not promoting CONSORT in both unadjusted (6.0 and 5.1, respectively, p-value = 0.03) and adjusted (6.4 and 4.8 of the 11 methodological factors, respectively, p-value = 0.0001) analyses. While journals that promote CONSORT demonstrate superior reporting of RCTs, persistent inadequacies in reporting remain. Until these inadequacies are resolved health-care providers will remain limited in their ability to make informed inferences about the validity of the studies upon which they base their clinical practice. To assess the reporting quality of randomized controlled trials (RCTs) in the primary treatment of brain tumors and to identify significant predictors of quality.
                Two investigators searched MEDLINE, EMBASE, and bibliographies of retrieved articles for RCTs in the primary treatment of brain tumors published between January 1990 and December 2004. We assessed the quality of overall reporting and key methodologic factors reporting (allocation concealment, blinding, and intention to treat [ITT]). Two investigators also rated articles independently using items from the revised Consolidated Standards of Reporting Trials statement. A generalized estimated equation was used to generate regression models that identified significant factors associated with quality of reporting.
                We retrieved 74 relevant RCTs that randomly assigned 14,498 brain tumor patients. The quality of overall reporting has improved during the last 15 years, but eight of the 15 methodologic items were reported in less than 50% of trials. In the appraisal of the reporting quality of key methodologies, allocation concealment, blinding, and adherence to the ITT principle were reported in less than 30% of articles. Multivariable regression models revealed that an impact factor more than 1.66, publication after 1995, and sample size more than 280 were significant factors associated with better overall reporting, whereas complete industrial funding, impact factors more than 2.64, and positive primary outcomes were predictors of higher ratings of the three most important methodologic qualities.
                Despite improvement in general reporting quality, key methodologies that safeguard against biases may still benefit from better description. Significant factors associated with better reporting may act as surrogates for other characteristics. The quality of randomized controlled trials (RCTs) in subfertility and their suitability for inclusion in meta-analyses have been assessed in the past and found to be insufficient. Our aim was to assess whether this quality has improved over time, particularly since the publication of the Consolidated Standards of Reporting Trials (CONSORT) statement, and to assess what proportion of trials could be included in the meta-analyses of pregnancy outcomes such as those included in Cochrane Reviews.
                A selection of subfertility trials published in 1990, 1996 and 2002 was collected from the Cochrane Menstrual Disorder and Subfertility Group (MDSG) database. Only trials published in English as full journal articles, claiming to be randomized and reporting on pregnancy outcomes, were included.
                One hundred and sixty-four trials met our inclusion criteria. Twenty-four (15%) were found not to be randomized, despite claims, and only 10 trials (6%) provided adequate details on the methods of randomization and allocation concealment. Of these, only three had sufficient details extractable to allow for an intention-to-treat analysis of the outcome 'live birth'.
                Although an improvement in some subfertility-specific issues was observed, the quality of reporting of RCTs still needs to improve to make them suitable for inclusion in meta-analyses such as those in the Cochrane Library. Substantial numbers of clinical trials continue to be reported only in summary reports that present insufficient methodological details to permit informed judgments about the likely validity of the conclusions. Using a cohort of 176 controlled trials reported in summary form, we tested the hypotheses that they would be more likely to be followed by full reports if, on the basis of the information provided in the summary report, (1) the trial was judged to be methodologically sound, (2) the results favored the test treatment, and (3) the sample size was relatively large. The results of univariate and multivariate analyses provided support for only the third of these hypotheses. Investigators, as well as those who fund and sanction the conduct of clinical research, should make greater efforts to ensure that clinical trials are reported properly.
The methodological quality and the reporting of the included studies were too poor to draw any useful conclusions.