Assessment of risk of bias among pediatric randomized controlled trials. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Association of funding and conclusions in randomized drug trials: a reflection of treatment effect or adverse events? Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. Abstracts of randomized controlled trials presented at the society for pediatric research meeting: an example of publication bias. Publication bias and meta-analyses: a practical example. Quality of reporting of randomized, controlled trials in cerebral palsy.
The goal was to assess the risk of bias among pediatric, randomized, controlled trials (RCTs) reported in 8 high-impact journals.
                We searched PubMed for all pediatric RCTs reported between July 1, 2007, and June 30, 2008, in 8 journals with high impact factors. Using Cochrane Collaboration methods for risk assessment, we evaluated all reports for risk of bias according to domain (ie, randomized sequence generation, allocation concealment, masking, incomplete outcome data, selective outcome reporting, and other). We used multiple logistic regression to test for associations between the presence of a high risk of bias according to domain and funding source, intervention type, trial registration, and multicenter status.
                Industry-funded RCTs were more likely to show a high risk of bias for sequence generation, compared with government-funded RCTs (adjusted odds ratio [aOR]: 6.1 [95% confidence interval [CI]: 1.70- 21.89]), and behavioral/educational trials were more likely to show a high risk of bias for sequence generation (aOR: 2.8 [95% CI: 1.06-7.36]) and allocation concealment (aOR: 4.09 [95% CI: 1.69-9.90]), compared with drug trials. Registered trials were less likely to have a high risk of bias for sequence generation, compared with nonregistered trials (aOR: 0.33 [95% CI: 0.15-0.71]).
                Overall, we found a large proportion of pediatric RCT reports with a high risk of bias for sequence generation and allocation concealment. Factors associated with a high risk of bias included industry funding and assessment of behavioral/educational interventions, whereas trial registration was associated with a lower risk of bias. To determine if inadequate approaches to randomized controlled trial design and execution are associated with evidence of bias in estimating treatment effects.
                An observational study in which we assessed the methodological quality of 250 controlled trials from 33 meta-analyses and then analyzed, using multiple logistic regression models, the associations between those assessments and estimated treatment effects.
                Meta-analyses from the Cochrane Pregnancy and Childbirth Database.
                The associations between estimates of treatment effects and inadequate allocation concealment, exclusions after randomization, and lack of double-blinding.
                Compared with trials in which authors reported adequately concealed treatment allocation, trials in which concealment was either inadequate or unclear (did not report or incompletely reported a concealment approach) yielded larger estimates of treatment effects (P < .001). Odds ratios were exaggerated by 41% for inadequately concealed trials and by 30% for unclearly concealed trials (adjusted for other aspects of quality). Trials in which participants had been excluded after randomization did not yield larger estimates of effects, but that lack of association may be due to incomplete reporting. Trials that were not double-blind also yielded larger estimates of effects (P = .01), with odds ratios being exaggerated by 17%.
                This study provides empirical evidence that inadequate methodological approaches in controlled trials, particularly those representing poor allocation concealment, are associated with bias. Readers of trial reports should be wary of these pitfalls, and investigators must improve their design, execution, and reporting of trials. To explore whether reported methodologic quality affects estimated intervention effects in randomized trials and contributes to discrepancies between the results of large randomized trials and small randomized trials in meta-analyses.
                Meta-analyses of randomized trials that included at least one large trial (>/=1000 participants) were included, regardless of the therapeutic area. Eligible meta-analyses were identified through electronic searches and bibliographies of relevant articles.
                Full-length randomized trials.
                Methodologic quality was assessed according to reported randomization, double blinding, and follow-up as separate components and by using the Jadad composite scale.
                Fourteen meta-analyses involving 190 randomized trials from eight therapeutic areas were included. Compared with large trials, intervention effects were exaggerated in small trials with inadequate allocation sequence generation (ratio of odds ratios, 0.46 [95% CI, 0.25 to 0.83]; P = 0.011), inadequate allocation concealment (ratio of odds ratios, 0.49 [CI, 0.27 to 0.86]; P = 0.014), and no double blinding (ratio of odds ratios, 0.52 [CI, 0.28 to 0.96]; P = 0.01). Large trials did not differ significantly from small trials with adequate generation of the allocation sequence, adequate allocation concealment, or adequate double blinding. No association was seen between reported follow-up and intervention effects. The Jadad scale provided no additional information because the scale and the quality components overlapped substantially.
                Inadequate generation of the allocation sequence, allocation concealment, and double blinding lead to exaggerated estimates of intervention benefit and may contribute to discrepancies between the results of large randomized trials and small randomized trials in meta-analyses. Previous studies indicate that industry-sponsored trials tend to draw proindustry conclusions.
                To explore whether the association between funding and conclusions in randomized drug trials reflects treatment effects or adverse events.
                Observational study of 370 randomized drug trials included in meta-analyses from Cochrane reviews selected from the Cochrane Library, May 2001. From a random sample of 167 Cochrane reviews, 25 contained eligible meta-analyses (assessed a binary outcome; pooled at least 5 full-paper trials of which at least 1 reported adequate and 1 reported inadequate allocation concealment). The primary binary outcome from each meta-analysis was considered the primary outcome for all trials included in each meta-analysis. The association between funding and conclusions was analyzed by logistic regression with adjustment for treatment effect, adverse events, and additional confounding factors (methodological quality, control intervention, sample size, publication year, and place of publication).
                Conclusions in trials, classified into whether the experimental drug was recommended as the treatment of choice or not.
                The experimental drug was recommended as treatment of choice in 16% of trials funded by nonprofit organizations, 30% of trials not reporting funding, 35% of trials funded by both nonprofit and for-profit organizations, and 51% of trials funded by for-profit organizations (P<.001; chi2 test). Logistic regression analyses indicated that funding, treatment effect, and double blinding were the only significant predictors of conclusions. Adjusted analyses showed that trials funded by for-profit organizations were significantly more likely to recommend the experimental drug as treatment of choice (odds ratio, 5.3; 95% confidence interval, 2.0-14.4) compared with trials funded by nonprofit organizations. This association did not appear to reflect treatment effect or adverse events.
                Conclusions in trials funded by for-profit organizations may be more positive due to biased interpretation of trial results. Readers should carefully evaluate whether conclusions in randomized trials are supported by data. Selective reporting of outcomes within published studies based on the nature or direction of their results has been widely suspected, but direct evidence of such bias is currently limited to case reports.
                To study empirically the extent and nature of outcome reporting bias in a cohort of randomized trials.
                Cohort study using protocols and published reports of randomized trials approved by the Scientific-Ethical Committees for Copenhagen and Frederiksberg, Denmark, in 1994-1995. The number and characteristics of reported and unreported trial outcomes were recorded from protocols, journal articles, and a survey of trialists. An outcome was considered incompletely reported if insufficient data were presented in the published articles for meta-analysis. Odds ratios relating the completeness of outcome reporting to statistical significance were calculated for each trial and then pooled to provide an overall estimate of bias. Protocols and published articles were also compared to identify discrepancies in primary outcomes.
                Completeness of reporting of efficacy and harm outcomes and of statistically significant vs nonsignificant outcomes; consistency between primary outcomes defined in the most recent protocols and those defined in published articles.
                One hundred two trials with 122 published journal articles and 3736 outcomes were identified. Overall, 50% of efficacy and 65% of harm outcomes per trial were incompletely reported. Statistically significant outcomes had a higher odds of being fully reported compared with nonsignificant outcomes for both efficacy (pooled odds ratio, 2.4; 95% confidence interval [CI], 1.4-4.0) and harm (pooled odds ratio, 4.7; 95% CI, 1.8-12.0) data. In comparing published articles with protocols, 62% of trials had at least 1 primary outcome that was changed, introduced, or omitted. Eighty-six percent of survey responders (42/49) denied the existence of unreported outcomes despite clear evidence to the contrary.
                The reporting of trial outcomes is not only frequently incomplete but also biased and inconsistent with protocols. Published articles, as well as reviews that incorporate them, may therefore be unreliable and overestimate the benefits of an intervention. To ensure transparency, planned trials should be registered and protocols should be made publicly available prior to trial completion. The reporting of outcomes within published randomized trials has previously been shown to be incomplete, biased and inconsistent with study protocols. We sought to determine whether outcome reporting bias would be present in a cohort of government-funded trials subjected to rigorous peer review.
                We compared protocols for randomized trials approved for funding by the Canadian Institutes of Health Research (formerly the Medical Research Council of Canada) from 1990 to 1998 with subsequent reports of the trials identified in journal publications. Characteristics of reported and unreported outcomes were recorded from the protocols and publications. Incompletely reported outcomes were defined as those with insufficient data provided in publications for inclusion in meta-analyses. An overall odds ratio measuring the association between completeness of reporting and statistical significance was calculated stratified by trial. Finally, primary outcomes specified in trial protocols were compared with those reported in publications.
                We identified 48 trials with 68 publications and 1402 outcomes. The median number of participants per trial was 299, and 44% of the trials were published in general medical journals. A median of 31% (10th-90th percentile range 5%-67%) of outcomes measured to assess the efficacy of an intervention (efficacy outcomes) and 59% (0%-100%) of those measured to assess the harm of an intervention (harm outcomes) per trial were incompletely reported. Statistically significant efficacy outcomes had a higher odds than nonsignificant efficacy outcomes of being fully reported (odds ratio 2.7; 95% confidence interval 1.5-5.0). Primary outcomes differed between protocols and publications for 40% of the trials.
                Selective reporting of outcomes frequently occurs in publications of high-quality government-funded trials. Publication bias toward studies that favor new therapies has been known to occur for the past 40 years, yet its implications are not well studied in child health. The increased interest in meta-analyses has highlighted the need to identify the totality of evidence when addressing treatment questions.
                To measure the percentage of randomized controlled trials (RCTs) presented at a major pediatric scientific meeting that were subsequently published as full-length articles, to investigate factors associated with publication, and to describe the variables that change from abstract to manuscript form.
                The scientific proceedings from the Society for Pediatric Research were hand searched for RCTs (1992-1995). Subsequent publication was ascertained through a search of various electronic databases. Quality of abstracts and manuscripts was measured, and data were extracted using a structured form.
                A total of 264 (59.1%) of 447 abstracts were subsequently published. Almost 64% of RCTs that were subsequently published favored new therapy compared with 43.5% of studies that were never published (P<.001). Mean effect size for published vs unpublished RCTs was 0.74 vs 0.05 (P<.001). Median sample size was larger in published (n = 45) vs unpublished (n = 34) RCTs (P =.02). Quality was significantly lower for abstracts vs published RCTs (P<.001). For 5% of abstracts that were subsequently published, the conclusion regarding treatment efficacy changed.
                Publication bias is a serious threat to assessing the effectiveness of interventions in child health, as little more than half of RCTs presented at a major scientific meeting are subsequently published. There is a need to institute an international registry of RCTs in children so that the totality of evidence can be accessed when assessing treatment effectiveness. Publication bias is widely appreciated, but considerable time and effort are needed to locate and obtain data from unpublished randomized controlled trials (RCTs), those published in non-English language journals or those reported in the gray literature; for this publication, we will call this collection of trials the "gray+literature." However, excluding such trials from systematic reviews could introduce bias and give rise to misleading conclusions.
                We aimed to explore and quantify the impact of inclusion of gray+ literature on the results of all completed individual patient data (IPD) reviews coordinated by our group (13 meta-analyses). For each IPD review, results were calculated for RCTs fully published in English language journals and RCTs fully published in English language journals and the gray+literature.
                The IPD meta-analyses based only on RCTs that were fully published in English language journals tended to give more favorable results than those that included RCTs from the gray+literature. Although in most cases the addition of gray+data gave less encouraging results, moving the estimated treatment effect toward a null result, the direction of effect was not always predictable.
                We recommend that all systematic reviews should at least attempt to identify trials reported in the gray+literature and, where possible, obtain data from them. In conducting reviews on the effectiveness of physiotherapy interventions on children with cerebral palsy, the assessment of trials can be hampered by problems in reporting. Therefore, we set out to evaluate trial reporting by using the Consolidated Standards of Reporting Trials (CONSORT) statement recommendations.
                Randomized, controlled trials published in 1990 or later were identified in literature searches for reviews on the effectiveness of physiotherapy interventions on children with cerebral palsy. Two independent reviewers evaluated the trial reporting by using a modified 33-item Consolidated Standards of Reporting Trials checklist.
                We identified 15 randomized, controlled trials. Almost half (48%) of the applicable items were reported adequately. Inadequate reporting was found in the following items: outcome measures, sample-size determination, details of the sequence generation, allocation concealment and implementation of the randomization, success of assessor blinding, recruitment and follow-up dates, intention-to-treat analysis, precision of the effect size, co-interventions, and adverse events.
                Only a small number of sufficiently reported trials were found. Because nearly all items had been described in at least 1 article, high-quality reporting seems feasible. Assessment of trials depends on appropriate reporting, and poor reporting jeopardizes judgments on the clinical implications. Authors of randomized, controlled trials are encouraged to follow the Consolidated Standards of Reporting Trials criteria. There is a clear need to improve the quality of reporting of trials in this field.
The reviewed trial presented a high risk of bias, which limited conclusions that could be drawn from the presented data. The inclusion of only one trial indicates the need for more robust randomised controlled trials to determine the efficacy of interventions for this condition.