Author contacts for retrieval of data for a meta-analysis on exercise and diet restriction. Review of the usefulness of contacting other experts when conducting a literature search for systematic reviews. Searching multiple databases for systematic reviews: added value or diminishing returns? Beyond Medline: reducing bias through extended systematic review search. Methodologists and their methods. Do methodologists write up their conference presentations or is it just 15 minutes of fame? The use of dedicated methodology and statistical reviewers for peer review: a content analysis of comments to authors made by methodology and regular reviewers. Effectiveness of various mailing strategies among nonrespondents in a prospective cohort study. Effect of open peer review on quality of reviews and on reviewers' recommendations: a randomised trial. Identifying manuscript reviewers: randomized comparison of asking first or just sending.
The mode of contact and response levels of authors who had been asked to provide missing or incomplete data for a systematic review on diet and exercise interventions for weight loss was examined.
                We contacted authors by electronic mail, letter, or both. Survival analyses were performed with the Kaplan-Meier method to determine differences in the proportion of responders over time among the different modes of contact and to determine whether response rates differed between authors from the United States and those from other countries. Logistic regression was used to determine whether the number of items requested and publication date influenced the likelihood of response.
                Two hundred forty-one (39.9 percent) studies had missing or incomplete data (e.g., sample size, age, caloric restriction, exercise amount, and so on). We were unable to locate ninety-five authors (39.4 percent). Of the remaining authors, forty-six authors (31.5 percent) responded to information requests. Time to respond differed by contact method (p < .05): e-mail (3 +/- 3 days), letter (27 +/- 30 days), and both (13 +/-12 days). Response rates from U.S. authors did not differ from those of other countries.
                Our study suggests poor success in the acquisition of essential information. Given considerable time and resources, weight loss studies require improved reporting standards to minimize the relatively unsuccessful attempt to contact authors for important and necessary information. nan To explore whether searching specialised bibliographic databases identified additional relevant papers to those located by a Medline search for a systematic review of exercise therapy.
                Searches were performed in Medline, two further generalised medical databases (Embase, Cochrane Library) and four specialised databases (CancerLit, Cinahl, PsychInfo, SportDiscus) to identify controlled trials of exercise interventions for cancer patients.
                A total of 749 different publications were located through the search, of which 18 met inclusion criteria. Fifteen (83%) of these were identified through Medline and three (17%) from three individual specialised databases. A further seven studies meeting inclusion criteria were located through reference lists and contact with experts.
                In this example, searching Medline and additional specialised databases along with checking reference lists and contacting experts was the most effective means of ensuring that all relevant papers were included in the review. Searching Medline alone for systematic reviews of exercise or other unconventional therapies is likely to be inadequate. To evaluate the sensitivity and precision of various extended search methods in identifying randomized controlled trials (RCTs) for systematic reviews.
                Prospective analysis of extended search methods (specialized databases or trial registries, reference lists, hand-searching, personal communication, and Internet) used in two systematic reviews of RCTs. The gold standard was the total number of RCTs identified by major databases (MEDLINE, EMBASE, etc.) and extended search strategies combined. Sensitivity was the proportion of all known RCTs identified by any extended search method. Precision reflected the proportion of all items uncovered by any extended search method that actually were RCTs.
                The extended search identified 94 additional RCTs for the systematic reviews beyond those identified with the major databases. Specialized databases and trial registries had the highest sensitivity and precision for the lipid-lowering project (13.6% and 52.7%, respectively; p < .05) followed by scanning of reference lists (7.2% sensitivity and 41.9% precision; p <.05). Hand-searching was more effective than personal communication and Internet searching (1.7% sensitivity and 12.2% precision; p < .05). The acupuncture project had slightly different results, with the specialized databases and trial registries tied with the review of reference lists for highest sensitivity (14.2%). The precision followed the same trend as the lipid-lowering project (17.6% specialized databases; 8.3% reference lists; p < .05). A post-hoc analysis showed that 75 of the 94 RCTs were indexed in the major databases but missed by the major database search.
                Extended searching identified additional RCTs for the systematic reviews beyond those found in major databases. Specialized databases and trial registries were most effective. An important number of RCTs were missed by the major database search. Timing and accuracy of indexing may explain this finding. The definitive measure, whether there is an association between the method used to uncover RCTs, the quality of the items uncovered and their impact on systematic review results, is yet to be determined. To assess the extent to which abstracts of methodological research, initially presented at meetings on systematic reviews, have gone on to be published as full articles.
                Full publication was assessed in three ways: a search was carried out of The Cochrane Library; a search was conducted using MEDLINE; and a questionnaire was sent to the contact author of each abstract.
                Approximately half of the abstracts had not been, or were unlikely ever to be, published in full.
                The rate of full publication of abstracts related to the methodology of systematic reviews seems similar to that for randomized trials. In 1997, Annals of Emergency Medicine initiated a protocol by which every original research article, in addition to each regular review, was concurrently evaluated by 1 of 2 methodology and statistical reviewers. We characterized and contrasted comments made by the methodology and regular peer reviewers.
                After pilot testing, interrater reliability assessment, and revision, we finalized a 99-item taxonomy of reviewer comments organized in 8 categories. Two authors, uninvolved in the writing of reviews, classified each comment from a random sample of methodology reviews from 1999. For 30 of these reviews (15 for each methodology reviewer), the 2 authors also scored all (range 2 to 5) regular reviews.
                Sixty-five reviews by methodologist A, 60 by methodologist B, and 68 by regular reviewers were analyzed. Comments by methodologist A most frequently concerned the presentation of results (33% of all comments) and methods (17%). Methodologist B commented most frequently on presentation of results (28%) and statistical methods (16%). Regular reviewers most frequently made non-methodology/statistical comments (45%) and comments on presentation of results (18%). Of note, comments made by methodology and regular reviewers about methods issues were often contradictory.
                The distributions of comments made by the 2 methodology and statistical reviewers were similar, although reviewer A emphasized presentation and reviewer B stressed statistical issues. The regular reviewers (most of whom were unaware that a dedicated methodology and statistical reviewer would be reviewing the article) paid much less attention to methodology issues. The 2 dedicated methodology and statistical reviewers created reviews that were similarly focused and emphasized methodology issues that were distinct from the issues raised by regular reviewers. Measures of association in prospective studies can be distorted by incomplete follow-up. Various mailing strategies were used to contact 12,233 cohort members of the Health Professionals Follow-up Study who had not responded to three successive bulk-rate mailings. Response rates were highest, 79.5% overall, from participants who were sent a certified mailing in phase 1 (63.2%), followed by a repeat certified mailing to nonrespondents (44.3%). Although altering the physical appearance of the envelope and using other postal rates were tested, certified mail was the most effective approach for reaching study members who were nonrespondents to a mailed questionnaire. To examine the effect on peer review of asking reviewers to have their identity revealed to the authors of the paper.
                Randomised trial. Consecutive eligible papers were sent to two reviewers who were randomised to have their identity revealed to the authors or to remain anonymous. Editors and authors were blind to the intervention.
                The quality of the reviews was independently rated by two editors and the corresponding author using a validated instrument. Additional outcomes were the time taken to complete the review and the recommendation regarding publication. A questionnaire survey was undertaken of the authors of a cohort of manuscripts submitted for publication to find out their views on open peer review.
                Two editors' assessments were obtained for 113 out of 125 manuscripts, and the corresponding author's assessment was obtained for 105. Reviewers randomised to be asked to be identified were 12% (95% confidence interval 0.2% to 24%) more likely to decline to review than reviewers randomised to remain anonymous (35% v 23%). There was no significant difference in quality (scored on a scale of 1 to 5) between anonymous reviewers (3.06 (SD 0.72)) and identified reviewers (3.09 (0.68)) (P=0.68, 95% confidence interval for difference - 0.19 to 0.12), and no significant difference in the recommendation regarding publication or time taken to review the paper. The editors' quality score for reviews (3.05 (SD 0.70)) was significantly higher than that of authors (2.90 (0.87)) (P<0.005, 95%confidence interval for difference - 0.26 to - 0.03). Most authors were in favour of open peer review.
                Asking reviewers to consent to being identified to the author had no important effect on the quality of the review, the recommendation regarding publication, or the time taken to review, but it significantly increased the likelihood of reviewers declining to review. Some journals routinely query potential referees before sending manuscripts ("askfirst"), whereas others just send manuscripts and allow referees to opt out ("justsend"). It is not known which protocol results in more completed reviews or shorter review time.
                Trial to assess proportion of referee turndowns and length of review process, conducted at editorial office of Obstetrics & Gynecology and involving 283 consecutive qualifying manuscripts. For each, a referee was randomly assigned to askfirst (manuscript sent only after affirmative response within 3 days) and another to justsend (manuscript sent with request to review; could opt out).
                Only 64% of askfirst referees assented initially (15% declined [vs 8% for justsend, P =.008] and 21% failed to respond within 3 working days, necessitating a replacement). But once manuscript was mailed, mean time to file a review was significantly shorter for askfirst (21.0 vs 25.0 days, P<.001); thus, overall time to receipt of review did not differ significantly (24.7 vs 25.9 days, P =.19), nor did review quality (P =.39).
                Askfirst led to a higher rate of referee turndown than did justsend, but assenting askfirst referees completed reviews faster. The overall time for the review process did not differ between the 2 protocols.
Those carrying out systematic reviews should continue to contact authors for missing data, recognising that this might not always be successful, particularly for older studies. Contacting authors by e-mail results in the greatest response rate with the fewest number of attempts and the shortest time to respond.