Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Thirty years of Medical Research Council randomized trials in solid tumours. The quality of reports of randomised trials in 2000 and 2006: comparative study of articles indexed in PubMed. Comparison of registered and published primary outcomes in randomized controlled trials. Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. Factors associated with failure to publish large randomized trials presented at an oncology meeting. Physicians' reasons for not entering eligible patients in a randomized clinical trial of surgery for breast cancer. Selection bias in clinical trials. Association of funding and conclusions in randomized drug trials: a reflection of treatment effect or adverse events?
To explore whether reported methodologic quality affects estimated intervention effects in randomized trials and contributes to discrepancies between the results of large randomized trials and small randomized trials in meta-analyses.
                Meta-analyses of randomized trials that included at least one large trial (>/=1000 participants) were included, regardless of the therapeutic area. Eligible meta-analyses were identified through electronic searches and bibliographies of relevant articles.
                Full-length randomized trials.
                Methodologic quality was assessed according to reported randomization, double blinding, and follow-up as separate components and by using the Jadad composite scale.
                Fourteen meta-analyses involving 190 randomized trials from eight therapeutic areas were included. Compared with large trials, intervention effects were exaggerated in small trials with inadequate allocation sequence generation (ratio of odds ratios, 0.46 [95% CI, 0.25 to 0.83]; P = 0.011), inadequate allocation concealment (ratio of odds ratios, 0.49 [CI, 0.27 to 0.86]; P = 0.014), and no double blinding (ratio of odds ratios, 0.52 [CI, 0.28 to 0.96]; P = 0.01). Large trials did not differ significantly from small trials with adequate generation of the allocation sequence, adequate allocation concealment, or adequate double blinding. No association was seen between reported follow-up and intervention effects. The Jadad scale provided no additional information because the scale and the quality components overlapped substantially.
                Inadequate generation of the allocation sequence, allocation concealment, and double blinding lead to exaggerated estimates of intervention benefit and may contribute to discrepancies between the results of large randomized trials and small randomized trials in meta-analyses. This paper reviews the survival outcome from the randomized Phase III trials in solid tumours published on behalf of, or in collaboration with, the Cancer Therapy Committee (CTC) of the British Medical Research Council over a 30-year period to 31 December 1995. We review briefly the innovations in statistical methodology that have occurred over the period. We also note the ways in which standards of reporting the trials have improved, with more recent publications including, for example, estimates of the size of effect and confidence intervals. In all, 32 trials, involving over 5000 deaths in more than 8000 patients, have been published. Tumour types have included bladder, bone, brain, cervix, colon and rectum, head and neck, kidney, lung, ovary, prostate and skin. This paper presents a bibliography of these trials and gives details of the treatment comparisons made, the numbers of patients randomized and included in the analysis for each treatment arm, the observed numbers of deaths, and an estimate of the hazard ratio with associated 95% confidence intervals. The bibliography also indicates the main endpoint of each trial, whether recurrence-free survival or survival, and whether the trial was aimed at finding a difference or showing equivalence. The MRC trials have made an impact on both clinical practice and research activities. For example, the lung cancer programme has helped to establish the role of chemotherapy in small cell lung cancer and has developed better palliative treatment for non-small cell lung cancer. Trials of the radiosensitizer misonidazole have demonstrated that it has no role in the treatment of a number of cancers, trials of hyperbaric oxygen have defined the biological activity of this approach, and the appropriate dose of radiotherapy in patients with brain tumours has been found. The individual trials recruited between 44 and 824 patients (median 213). A better measure of the information in a trial is the number of deaths reported, which varied from 28 to 661 (median 145). A large proportion of the comparisons (8/29 or 28%) anticipating a survival difference, demonstrated such a difference at the 5% level of significance. Despite this, it is concluded that some of the trials should have been larger. In such cases, hindsight suggests either that an overoptimistic view of the anticipated survival benefit was taken at the design stage, or, for equivalence trials, the planned confidence interval was too wide for definitive statements to be made. As a consequence, the current CTC profolio of ongoing randomized trials open to patient accrual at 1 January 1996 have a projected median size of 600 and range from 120 to 2000 patients. To examine the reporting characteristics and methodological details of randomised trials indexed in PubMed in 2000 and 2006 and assess whether the quality of reporting has improved after publication of the Consolidated Standards of Reporting Trials (CONSORT) Statement in 2001.
                Comparison of two cross sectional investigations. Study sample All primary reports of randomised trials indexed in PubMed in December 2000 (n=519) and December 2006 (n=616), including parallel group, crossover, cluster, factorial, and split body study designs.
                The proportion of general and methodological items reported, stratified by year and study design. Risk ratios with 95% confidence intervals were calculated to represent changes in reporting between 2000 and 2006.
                The majority of trials were two arm (379/519 (73%) in 2000 v 468/616 (76%) in 2006) parallel group studies (383/519 (74%) v 477/616 (78%)) published in specialty journals (482/519 (93%) v 555/616 (90%)). In both 2000 and 2006, a median of 80 participants were recruited per trial for parallel group trials. The proportion of articles that reported drug trials decreased between 2000 and 2006 (from 393/519 (76%) to 356/616 (58%)), whereas the proportion of surgery trials increased (51/519 (10%) v 128/616 (21%)). There was an increase between 2000 and 2006 in the proportion of trial reports that included details of the primary outcome (risk ratio (RR) 1.18, 95% CI 1.04 to 1.33), sample size calculation (RR 1.66, 95% CI 1.40 to 1.95), and the methods of random sequence generation (RR 1.62, 95% CI 1.32 to 1.97) and allocation concealment (RR 1.40, 95% CI 1.11 to 1.76). There was no difference in the proportion of trials that provided specific details on who was blinded (RR 0.91, 95% CI 0.75 to 1.10).
                Reporting of several important aspects of trial methods improved between 2000 and 2006; however, the quality of reporting remains well below an acceptable level. Without complete and transparent reporting of how a trial was designed and conducted, it is difficult for readers to assess its conduct and validity. As of 2005, the International Committee of Medical Journal Editors required investigators to register their trials prior to participant enrollment as a precondition for publishing the trial's findings in member journals.
                To assess the proportion of registered trials with results recently published in journals with high impact factors; to compare the primary outcomes specified in trial registries with those reported in the published articles; and to determine whether primary outcome reporting bias favored significant outcomes.
                MEDLINE via PubMed was searched for reports of randomized controlled trials (RCTs) in 3 medical areas (cardiology, rheumatology, and gastroenterology) indexed in 2008 in the 10 general medical journals and specialty journals with the highest impact factors.
                For each included article, we obtained the trial registration information using a standardized data extraction form.
                Of the 323 included trials, 147 (45.5%) were adequately registered (ie, registered before the end of the trial, with the primary outcome clearly specified). Trial registration was lacking for 89 published reports (27.6%), 45 trials (13.9%) were registered after the completion of the study, 39 (12%) were registered with no or an unclear description of the primary outcome, and 3 (0.9%) were registered after the completion of the study and had an unclear description of the primary outcome. Among articles with trials adequately registered, 31% (46 of 147) showed some evidence of discrepancies between the outcomes registered and the outcomes published. The influence of these discrepancies could be assessed in only half of them and in these statistically significant results were favored in 82.6% (19 of 23).
                Comparison of the primary outcomes of RCTs registered with their subsequent publication indicated that selective outcome reporting is prevalent. Specific features of trial quality may be associated with exaggeration or shrinking of the observed treatment effect in randomized studies. Therefore, assessment of trial quality is often used in meta-analysis. However, the degree to which specific quality measures are associated with treatment effects has not been well established across a broad range of clinical areas.
                To determine if quality measures are associated with treatment effect size in randomized controlled trials (RCTs).
                Quality measures from published quality assessment scales were evaluated in RCTs included in meta-analyses from 4 medical areas (cardiovascular disease, infectious disease, pediatrics, and surgery). Included meta-analyses incorporated at least 6 RCTs, examined dichotomous outcomes, and demonstrated significant between-study heterogeneity in the odds ratio (OR) scale.
                Relative ORs comparing overall treatment effect (summary OR) of high vs low-quality studies, as determined by each quality measure, with relative ORs less than 1 indicating larger treatment effect in low-quality studies.
                Twenty-four quality measures were analyzed for 276 RCTs from 26 meta-analyses. Relative ORs of high vs low-quality studies for these quality measures ranged from 0.83 to 1.26; none was statistically significantly associated with treatment effect. The proportion of studies fulfilling specific quality measures varied widely in the 4 medical areas. In analyses limited to specific medical areas, placebo control, multicenter studies, study country, caregiver blinding, and statistical methods were significantly associated with treatment effect on 7 occasions. These relative ORs ranged from 0.40 to 1.74. However, the directions of these associations were not consistent.
                Individual quality measures are not reliably associated with the strength of treatment effect across studies and medical areas. Although use of specific quality measures may be appropriate in specific well-defined areas in which there is pertinent evidence, findings of associations with treatment effect cannot be generalized to all clinical areas or meta-analyses. Large clinical trials are the criterion standard for making treatment decisions, and nonpublication of the results of such trials can lead to bias in the literature and contribute to inappropriate medical decisions.
                To determine the rate of full publication of large randomized trials presented at annual meetings of the American Society of Clinical Oncology (ASCO), quantify bias against publishing nonsignificant results, and identify factors associated with time to publication.
                Survey of 510 abstracts from large (sample size, > or =200), phase 3, randomized controlled trials presented at ASCO meetings between 1989 and 1998. Trial results were classified as significant (P< or =.05 for the primary outcome measure) or nonsignificant (P>.05 or not reported), and the type of presentation and sponsorship were identified. Subsequent full publication was identified using a search of MEDLINE and EMBASE, completed November 1, 2001; the search was updated in November 2002, using the Cochrane Register of Controlled Trials. Authors were contacted if the searches did not find evidence of publication.
                Publication rate at 5 years; time from presentation to full publication.
                Of 510 randomized trials, 26% were not published in full within 5 years after presentation at the meeting. Eighty-one percent of the studies with significant results had been published by this time compared with 68% of the studies with nonsignificant results (P<.001). Studies with oral or plenary presentation were published sooner than those not presented (P =.002), and studies with pharmaceutical sponsorship were published sooner than studies with cooperative group sponsorship or studies for which sponsorship was not specified (P =.02). These factors remained significant in a multivariable model. The most frequent reason cited by authors for not publishing was lack of time, funds, or other resources.
                A substantial number of large phase 3 trials presented at an international oncology meeting remain unpublished 5 years after presentation. Bias against publishing nonsignificant results is a problem even for large randomized trials. Nonpublication breaks the contract that investigators make with trial participants, funding agencies, and ethics boards. We studied the reasons surgical principal investigators chose not to enter patients in a large, multicenter trial sponsored by a cooperative group. In 1976 the National Surgical Adjuvant Project for Breast and Bowel Cancers (NSABP) initiated a clinical trial to compare segmental mastectomy and postoperative radiation, or segmental mastectomy alone, with total mastectomy. Because the low rates of accrual were threatening to close the trial prematurely, we mailed a questionnaire to the 94 NSABP principal investigators, asking why they were not entering eligible patients in the trial. A response rate of 97 per cent was achieved. Physicians who did not enter all eligible patients offered the following explanations: (1) concern that the doctor-patient relationship would be affected by a randomized clinical trial (73 per cent), (2) difficulty with informed consent (38 per cent), (3) dislike of open discussions involving uncertainty (22 per cent), (4) perceived conflict between the roles of scientist and clinician (18 per cent), (5) practical difficulties in following procedures (9 per cent), and (6) feelings of personal responsibility if the treatments were found to be unequal (8 per cent). Further investigation into the behavioral aspects of the investigator-patient relationship is particularly pressing, since fear of change in this relationship was the most common reason given for not entering eligible patients in the trial. Of 90 patients with intermediate or high-grade sarcoma eligible for a randomized trial of adjuvant doxorubicin (Adriamycin, Adria Laboratories, Columbus, Ohio), 48 were not entered: 24 (27%) by physician's choice and 24 refused randomization. Sixty-five percent of lower stage patients were randomized compared with 37% of those with higher stage (P = .02). Patients with extremity lesions were more frequently offered participation in the study (P = .07). Patients with lower stage lesions accepted randomization more readily than those with higher stage lesions (P = .01). As predicted by the higher stage and percentage of central lesions, the disease-free survival of nonrandomized patients was inferior to that of randomized patients (P = .15). Thus, patients at high risk appeared to avoid randomization and adjuvant doxorubicin in this trial, resulting in an inferior disease-free survival for the nonrandomized control group. Important questions generally require randomized trials that reliably determine relative treatment differences. If, however, the patients in a clinical trial are not representative of the entire patient population because of patient and physician selection biases, the generalizability of the results to the entire patient population may be compromised. For example, the prognosis of the general population cannot necessarily be inferred from the selected group in the study. In this study, the randomized and nonrandomized series yielded differing conclusions regarding treatment efficacy, even when an adjustment was made for known prognostic facts. Previous studies indicate that industry-sponsored trials tend to draw proindustry conclusions.
                To explore whether the association between funding and conclusions in randomized drug trials reflects treatment effects or adverse events.
                Observational study of 370 randomized drug trials included in meta-analyses from Cochrane reviews selected from the Cochrane Library, May 2001. From a random sample of 167 Cochrane reviews, 25 contained eligible meta-analyses (assessed a binary outcome; pooled at least 5 full-paper trials of which at least 1 reported adequate and 1 reported inadequate allocation concealment). The primary binary outcome from each meta-analysis was considered the primary outcome for all trials included in each meta-analysis. The association between funding and conclusions was analyzed by logistic regression with adjustment for treatment effect, adverse events, and additional confounding factors (methodological quality, control intervention, sample size, publication year, and place of publication).
                Conclusions in trials, classified into whether the experimental drug was recommended as the treatment of choice or not.
                The experimental drug was recommended as treatment of choice in 16% of trials funded by nonprofit organizations, 30% of trials not reporting funding, 35% of trials funded by both nonprofit and for-profit organizations, and 51% of trials funded by for-profit organizations (P<.001; chi2 test). Logistic regression analyses indicated that funding, treatment effect, and double blinding were the only significant predictors of conclusions. Adjusted analyses showed that trials funded by for-profit organizations were significantly more likely to recommend the experimental drug as treatment of choice (odds ratio, 5.3; 95% confidence interval, 2.0-14.4) compared with trials funded by nonprofit organizations. This association did not appear to reflect treatment effect or adverse events.
                Conclusions in trials funded by for-profit organizations may be more positive due to biased interpretation of trial results. Readers should carefully evaluate whether conclusions in randomized trials are supported by data.
Despite reviewing 39 eligible trials, few firm conclusions could be reached because of the multiple comparisons considered, the small size of individual trials, and their low quality. Whether or not to use a particular policy is usually a trade-off between the risks of morbidity (especially infection) and risks of recatheterisation.