MJA
MJA InSight
MJA Careers
Systematic review

Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review

Amy C Plint, David Moher, Andra Morrison, Kenneth Schulz, Douglas G Altman, Catherine Hill and Isabelle Gaboury
Med J Aust 2006; 185 (5): 263-267.
Abstract

Objective: To determine whether the adoption of the CONSORT checklist is associated with improvement in the quality of reporting of randomised controlled trials (RCTs).

Data sources: MEDLINE, EMBASE, Cochrane CENTRAL, and reference lists of included studies and of experts were searched to identify eligible studies published between 1996 and 2005.

Study selection: Studies were eligible if they (a) compared CONSORT-adopting and non-adopting journals after the publication of CONSORT, (b) compared CONSORT adopters before and after publication of CONSORT, or (c) a combination of (a) and (b). Outcomes examined included reports for any of the 22 items on the CONSORT checklist or overall trial quality.

Data synthesis: 1128 studies were retrieved, of which 248 were considered possibly relevant. Eight studies were included in the review. CONSORT adopters had significantly better reporting of the method of sequence generation (risk ratio [RR], 1.67; 95% CI, 1.19–2.33), allocation concealment (RR, 1.66; 95% CI, 1.37–2.00) and overall number of CONSORT items than non-adopters (standardised mean difference, 0.83; 95% CI, 0.46–1.19). CONSORT adoption had less effect on reporting of participant flow (RR, 1.14; 95% CI, 0.89–1.46) and blinding of participants (RR, 1.09; 95% CI, 0.84–1.43) or data analysts (RR, 5.44; 95% CI, 0.73–36.87). In studies examining CONSORT-adopting journals before and after the publication of CONSORT, description of the method of sequence generation (RR, 2.78; 95% CI, 1.78–4.33), participant flow (RR, 8.06; 95% CI, 4.10–15.83), and total CONSORT items (standardised mean difference, 3.67 items; 95% CI, 2.09–5.25) were improved after adoption of CONSORT by the journal.

Conclusions: Journal adoption of CONSORT is associated with improved reporting of RCTs.

In the mid 1990s, in response to concerns about quality of reporting of randomised controlled trials (RCTs), an international group developed the Consolidated Standards of Reporting Trials (CONSORT) statement.1,2 CONSORT is intended to improve the reporting of an RCT, enabling readers to understand its conduct and to gauge the validity of its results.

The original CONSORT statement, developed for simple two-group parallel RCTs, comprises a checklist and flow diagram.1 Presently, CONSORT includes a 22-item checklist and four-stage flow diagram.2 The items included in the checklist were selected, whenever possible, because empirical evidence indicates that not reporting the information is associated with biased estimates of treatment effectiveness, or because the information is essential to judge the reliability or relevance of the findings.2-6 CONSORT has been endorsed by prominent medical journals and by editorial groups, such as the International Committee of Medical Journal Editors.

Since the publication of CONSORT, several evaluations of its effectiveness have been published.7-10 Two articles have suggested that journal endorsement of CONSORT is associated with improved quality of reporting.7,8 However, other study results have been equivocal.9,10 To gain a broader global perspective of CONSORT’s effectiveness, we conducted a systematic review, examining effectiveness in journals that have formally endorsed CONSORT.

Methods
Study selection

Studies were eligible if they were comparative studies evaluating the quality of RCT reporting; one of the comparator groups included RCTs reported in CONSORT-adopting journals; and outcomes reported included any of the 22 items on the CONSORT checklist, summary scores based on the CONSORT checklist, or any measures of overall trial quality.11 Study quality was not an exclusion criterion, and studies were not excluded based on publication status or language of publication.

Journal adoption of CONSORT was defined as a statement in the “instructions to authors” to follow CONSORT in preparing manuscripts, or a requirement for authors to submit a completed CONSORT checklist with their manuscript.

Eligible studies fell into three groups: those that (a) compared CONSORT adopters and non-adopters after the publication of CONSORT, (b) compared CONSORT-adopting journals before and after publication of CONSORT, or (c) a combination of (a) and (b). We did not specify a primary outcome for our review, but included as outcomes: (1) reporting of any of the 22 items on the CONSORT checklist, (2) summary scores based on the CONSORT checklist, or (3) any measures of overall trial quality.

Study identification

The main search strategy was developed in MEDLINE (1996 to 2005 week 28) and customised for EMBASE (1996 to 2005 week 32); both were executed through the Ovid interface. In addition, the Cochrane Methodology Register (up to 2nd Quarter 2005) and the Cochrane Database of Methodology Reviews (up to 2nd Quarter 2005) were initially searched via Update Software in 2004 and then via Wiley InterScience for the 2005 update. The Science Citation Index, Social Sciences Citation Index and Arts & Humanities Citation Index (June 2005) were searched through the ISI Web of Knowledge interface.

We searched for grey literature, checked the bibliographic references of relevant articles, and consulted with CONSORT experts.

The search was designed to retrieve citations published since 1996, when CONSORT was first published. Keywords included randomised controlled trial, publication bias, CONSORT, and epidemiological methods. The full search strategy is available from the authors.

Study inclusion

Two reviewers (A C P and A M) independently reviewed titles and abstracts using broad screening criteria (comparative study, and outcomes included some measure of trial reporting quality). Articles of potential relevance were further assessed (by A C P and D M) for eligibility. Disagreements were resolved by consensus. The reviewers were not blinded to authors or journals.

Detailed determination of study eligibility

When a comparative study with appropriate outcomes was identified, a detailed assessment was undertaken to determine eligibility. First, the study was reviewed to determine the journals and time frame in which the individual RCTs forming the basis for the comparison were published.

Second, the journals for these individual RCTs were examined to determine whether they met our definition of CONSORT adopter. We reviewed the most recent “instructions to authors” to determine if the journal was currently a CONSORT adopter (as indicated by a statement to use CONSORT in preparing the manuscript or by requiring submission of a completed checklist). We then reviewed previous issues to determine the journal’s CONSORT adoption date, and compared this with the date of RCT publication. The journal had to have adopted the CONSORT statement at least 6 months before the RCT publication to be considered a CONSORT adopter.

If CONSORT use was not currently required by a journal, we assumed the journal had never adopted the CONSORT statement.

Assessment of methodological quality

A single non-blinded reviewer (A C P) assessed reporting quality using a five-item checklist based on principles of internal and external validity.12 Quality assessment included the use of a control group, RCT cohort ascertainment, how the authors reached agreement on whether a checklist item was included, and whether the reviewers were blinded when assessing outcomes.

Data collection and analysis

Data were extracted by one reviewer (A C P) using a standardised data form, and then checked for accuracy by a second reviewer (D M). Only “consensus” data were used in the review. All data were entered in Review Manager 4.2 (Nordic Cochrane Centre, Copenhagen, Denmark). Dichotomous data were expressed as risk ratios (RR) with 95% confidence intervals. Continuous data were expressed as standardised mean differences with 95% confidence intervals. Consistency across the studies was examined using the I-squared test. Data were pooled using fixed effect models. A priori subgroup analyses included a comparison of RCTs published in journals that specifically require submission of a completed CONSORT checklist with journals that are not CONSORT adopters.

Authors of several possibly relevant studies were contacted for clarification of eligibility and invited to contribute unpublished data. We did not contact the authors of all included studies for additional unpublished data.

Results
Description of studies

Our search strategy identified 1128 studies; 248 were potentially relevant and eight of them7,8,13-18 met our inclusion criteria (Box 1). One study was included after the author provided unpublished data.17 Six studies that examined changes in the quality of trial reporting over time or compared quality of trial reporting between journals were excluded, either because they did not include a comparator group or were not published in CONSORT-adopting journals,9,19-21 or because insufficient detail was available to determine eligibility.10,22

The number of RCTs in individual studies ranged from 13 to 240, and the number of journals in individual studies ranged from 1 to 68 (Box 2). All studies were published in English. One study used a different definition of CONSORT adoption (journals in which the editor applied the CONSORT checklist were considered CONSORT adopters),7 but 88% of RCTs in the study met our definition of CONSORT adoption. For all other included studies, the definition of CONSORT adopters matched our definition.

Overall, 37 different outcomes were reported (range, 1–16 per study) (Box 3). This variability resulted in most outcomes being reported in only one study. Outcomes used in more than one study were reporting of allocation concealment (five studies), sequence generation (three studies), blinding (four studies), participant flow (two studies) and overall number of CONSORT checklist items (two studies, one using a total of 30 items, the other 40 items). Blinding as an outcome was assessed as reported or not in one study,17 and blinding of participants and data analysts in three studies.7,17,18

Box 2 summarises the methodological quality of the studies. All studies were quasi-experimental, with only one study having an a priori defined control group.8

CONSORT adopters and non-adopters

CONSORT-adopting journals had better reporting of the method of sequence generation and allocation concealment (Box 4A). For the two studies that reported total number of CONSORT items, the standardised mean difference was 0.83 (95% CI, 0.46–1.19). CONSORT adoption appeared to have less effect on reporting of participant flow and reporting of blinding of participants or data analysts.

Although we planned a priori to determine the influence of journals requiring submission of a completed CONSORT checklist when manuscripts are submitted, this comparison was impossible as very few journals actually require checklist submission. The paucity of data also precluded meaningful examination of publication bias. Tests for heterogeneity were all non-significant.

CONSORT-adopting journals before and after CONSORT publication

The descriptions of the method of sequence generation, participant flow and total CONSORT items (standardised mean difference, 3.67 items; 95% CI, 2.09–5.25) were better after adoption of CONSORT (Box 4B). CONSORT adoption appeared to have less effect on allocation concealment. The paucity of data precluded meaningful examination of publication bias. There was no evidence of heterogeneity.

Discussion

Studies evaluating the effectiveness of the CONSORT checklist are methodologically weak. The eight studies we identified used different quasi-experimental designs, which (with one exception) did not include a control group, addressed different questions, and used a large number of discrete outcomes, negating the possibility of combining them — a strength of the systematic review process. Further, analysis of studies that sample journals should take account of the clustering effect in the analysis, which would widen the confidence intervals. Only one study included in the review considered this effect. Nevertheless, our review suggests some improvement in the quality of reporting RCTs when the CONSORT checklist is used. Although the degree of improvement in quality of reporting of particular trial items differs somewhat between our comparison groups (CONSORT adopters v non-adopters after CONSORT publication, and CONSORT adopters before and after CONSORT publication), the direction of the effect remains the same. The difference in magnitude may be related to the small number of studies.

Our results are encouraging, but provide no definitive answer as to whether the CONSORT checklist improves reporting of RCTs. Perhaps unique to our review is ascertaining whether the RCTs were published in journals that adhered to CONSORT. If journals do not enforce the use of the checklist, it is likely that the effects we observed underestimate the possible benefits with proper enforcement of CONSORT. A review of 167 high-impact journals found that only 22% mentioned CONSORT in their instructions to authors, and 25% of these referred to the obsolete 1996 version.24 In another study involving 15 high-impact journals (five of which were not included in the previously mentioned review) that have reported endorsing CONSORT, only eight referred to the statement in their instructions to authors.25 CONSORT-adopting journals should be more proactive in enforcing adherence to CONSORT.

Although we did not find a strong influence of CONSORT on the quality of reporting of blinding, a recent study has suggested that the CONSORT checklist may have had a positive influence. In a study of RCTs in journals that adopted the original and then the revised CONSORT checklist, the authors found that the quality of reporting of blinding improved by 23%–55% between publication of the two checklists.26 This is particularly important in light of emerging data suggesting investigators, educators and readers vary greatly in their interpretations and definitions of types of blinding.27

Many of the evaluations included in our review were conducted shortly after the introduction of CONSORT. The benefits from CONSORT might take longer to materialise. It might be prudent to conduct further evaluations of the checklist after a longer adoption period, such as 5 years.

The trials reported a large number of discrete outcomes, and it is possible that individual studies selectively reported their outcomes. As we did not contact all authors of included studies for unpublished outcome data, we cannot address this issue.

In summary, it appears that use of the CONSORT checklist is associated with some improvement in reporting of RCTs. With endorsement by more journals, and greater editorial efforts to ensure that authors comply, CONSORT could begin to yield the full benefits for which it was intended.

1 Selection of studies for review

2 Characteristics and quality of included studies

First author, year, country

Number of RCTs

Number of journals

Number of CONSORT adopters*

Clinical content

Study design

More than one outcome reviewer

Interobserver reliability or consensus method

Blinded reviewer

Source of funding


Hewitt,13 2005, UK

234

4

3 (2 follow; 1 checklist)

Multiple areas

Post-intervention

Yes

No

Not reported

University of York

Halpern,14 2004, Canada

99

15

1 (1 checklist)

Obstetrical anaesthesia

Before/after

Not reported

Yes

Not reported

Not reported

Faunce,15 2003, Australia

13

7

2 (2 checklist)

Overdoses in health volunteers

Before/after; post-intervention

Yes

Not reported

Not reported

Not reported

Montori,16 2002, Canada

200

5

4 (2 follow; 2 checklist)

Multiple areas

Post-intervention

Yes

Yes

Not reported

Not reported

Hill,17 2002, US/Australia

240§

68§

2 (2 follow)

Adult rheumatological diseases

Before/after; post-intervention

Yes

Yes

Yes

Arthritis Foundation of Australia

Devereaux,7 2002, Canada

105

7

6 (4 follow; 2 checklist)

Internal medicine

Post-intervention

Yes

Yes

Not reported

Heart and Stroke Foundation of Canada; Canadian Institutes of Health Research (CIHR); Alberta Heritage Foundation; Canada Research Chair

Sanchez-Thorin,18 2001, Colombia

75**

1**

1 (1 checklist)

Ophthalmology

Before/after

Yes

Yes

Not reported

Not reported

Moher,8 2001, Canada

211

4

3 (2 follow; 1 checklist)

Multiple areas

Before/after with control group

Yes

Yes

No

Jefferson Smurfit Foundation, Ireland; US National Library of Medicine; Merck; Glaxo Wellcome; CIHR


* “Follow” means journal instructions to authors include a statement to follow CONSORT; “checklist” means authors must submit a CONSORT checklist with the article. “Source of funding” means salary support to the investigators. The study reported the results of 99 articles from 15 journals. Separate results were reported for journals with more than five articles, for a total of 83 articles from 7 journals — these results were considered for the systematic review. § This study reported the results of 240 papers published in 68 journals. The author provided unpublished data for 133 papers in 66 journals for inclusion in the systematic review. A random selection of articles was reviewed in a blinded manner. ** This study used results from a previously published paper by Scherer and Crawley23 as the comparison group. Sanchez-Thorin et al examined 24 RCTs from one ophthalmology journal and included as a comparison the results of 51 trials published in the same journal and reported by Scherer et al, who examined 125 trials published in three ophthalmology journals. RCT = randomised controlled trial.

3 CONSORT items investigated in the included studies

Study

Methods

Results

Other


Hewitt13

Allocation concealment


Halpern*14

Total number of CONSORT items (out of 30)


Faunce15

Eligibility criteria; stopping rules

Protocol deviation


Montori16

Blinding of participants, data analysts, and outcome assessors


Hill17

Sequence generation; allocation concealment; blinding

Participant flow


Devereaux7

Sequence generation; allocation concealment; blinding of participants, data analysts, intervention administrators, and outcome assessors; method of analysis

Baseline characteristics; participant flow


Sanchez-Thorin18

Eligibility criteria; interventions; objectives; outcomes; sample size determination; sequence generation; allocation concealment; randomisation implementation; blinding of participants, intervention administrators, and outcome assessors; method of analysis

Participant flow diagram; numbers analysed; results for outcome

Description of allocation method in title and abstract


Moher*8

Total score (22 items); allocation concealment

Total score (10 items)

Total number of CONSORT items (out of 40); total number of CONSORT items for title/abstract, introduction, discussion


* Both these papers developed a checklist of items based on the CONSORT checklist and gave an overall score to the trials reviewed using this score. Moher et al also reported the results as a total per section of the RCT, and reported the Jadad score as a quality indicator.

4 Effects of adopting CONSORT

A: Analysis of CONSORT-adopters compared with non-adopters. B: Analysis of CONSORT-adopters before and after adopting CONSORT.

Amy C Plint, MD, Associate Professor, Department of Pediatrics and Department of Emergency Medicine1
David Moher, PhD, Associate Professor, Department of Pediatrics and Department of Epidemiology and Community Medicine,1 and Director2
Andra Morrison, BSc, Associate3
Kenneth Schulz, PhD, MBA, Vice President4
Douglas G Altman, PhD, Professor5
Catherine Hill, MB BS, MSc, Staff Physician6
Isabelle Gaboury, PhD(c), Statistician2
1 University of Ottawa, Ottawa, Ontario, Canada.
2 Chalmers Research Group, Children’s Hospital of Eastern Ontario, Ottawa, Ontario, Canada.
3 Canadian Coordinating Office for Health Technology Assessment, Ottawa, Ontario, Canada.
4 Quantitative Sciences, Family Health International, Durham, NC, USA.
5 Centre for Statistics in Medicine, Oxford, UK.
6 Rheumatology Unit, Queen Elizabeth Hospital, Adelaide, SA.
Correspondence: 
Competing interests: 

None identified.

Reference Text: 
Begg C, Cho M, Eastwood S, et al. Improving the quality of reporting of randomized controlled trials: the CONSORT statement. JAMA 1996; 276: 637-639.
Reference Order: 
1
PubMed ID: 
8773637
Reference Text: 
Moher D, Schulz KF, Altman D for the CONSORT Group. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA 2001; 285: 1987-1991.
Reference Order: 
2
PubMed ID: 
11308435
Reference Text: 
Hollis S, Campbell F. What is meant by intention-to-treat analysis? Survey of published randomized controlled trials. BMJ 1999; 319: 670-674.
Reference Order: 
3
PubMed ID: 
10480822
Reference Text: 
Ruiz-Canela M, Martinez-Gonzalez MA, de Irala-Estevez J. Intention to treat analysis is related to methodological quality [letter]. BMJ 2000; 320: 1007-1008.
Reference Order: 
4
PubMed ID: 
Reference Text: 
Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995; 273: 408-412.
Reference Order: 
5
PubMed ID: 
7823387
Reference Text: 
Moher D, Pham B, Jones A, et al. Does the quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 1998; 352: 609-613.
Reference Order: 
6
PubMed ID: 
9746022
Reference Text: 
Devereaux PJ, Manns BJ, Ghali WA, et al. The reporting of methodological factors in randomized controlled trials and the association with a journal policy to promote adherence to the Consolidated Standards of Reporting Trials (CONSORT) checklist. Control Clin Trials 2002; 23: 380-388.
Reference Order: 
7
PubMed ID: 
12161081
Reference Text: 
Moher D, Jones A, Lepage L for the CONSORT Group. Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. JAMA 2001; 285: 1992-1995.
Reference Order: 
8
PubMed ID: 
11308436
Reference Text: 
Piggot M, McGee H, Feuer D. Has CONSORT improved the reporting of randomized controlled trials in the palliative care literature? A systematic review. Palliat Med 2004; 18: 32-38.
Reference Order: 
9
PubMed ID: 
14982205
Reference Text: 
Bhandari M, Guyaa G, Lochner H, et al. Application of the Consolidated Standards of Reporting Trials (CONSORT) in the fracture care literature. J Bone Joint Surg Am 2002; 84: 485-489.
Reference Order: 
10
PubMed ID: 
11886922
Reference Text: 
Jadad AR, Moore RA, Carroll D, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials 1996; 17: 1-12.
Reference Order: 
11
PubMed ID: 
8721797
Reference Text: 
Campbell DC, Stanley JT. Experimental and quasi-experimental design for research. Boston: Houghton Mifflin Company, 1966.
Reference Order: 
12
PubMed ID: 
Reference Text: 
Hewitt C, Hahn S, Torgerson DJ, et al. Adequacy and reporting of allocation concealment: a review of recent trials published in four general medical journals. BMJ 2005; 330: 1057-1058.
Reference Order: 
13
PubMed ID: 
15760970
Reference Text: 
Halpern SH, Darani R, Douglas MJ, et al. Compliance with the CONSORT checklist in obstetric anaesthesia randomised controlled trials. Int J Obstet Anesth 2004; 13: 207-214.
Reference Order: 
14
PubMed ID: 
15477048
Reference Text: 
Faunce TA, Buckley NA. Of consents and CONSORTS: reporting ethics, law, and human rights in RCTs involving monitored overdose of healthy volunteers pre and post the “CONSORT” guidelines. J Toxicol Clin Toxicol 2003; 41: 93-99.
Reference Order: 
15
PubMed ID: 
12733843
Reference Text: 
Montori VM, Bhandari M, Devereaux PJ, et al. In the dark: the reporting of blinding status in randomized controlled trials. J Clin Epidemiol 2002; 55: 787-790.
Reference Order: 
16
PubMed ID: 
12384193
Reference Text: 
Hill CL, LaValley MP, Felson DT. Secular changes in the quality of published randomized clinical trials in rheumatology. Arthritis Rheum 2002; 46: 779-784.
Reference Order: 
17
PubMed ID: 
11920415
Reference Text: 
Sanchez-Thorin JC, Cortes MC, Montenegro M, Villate N. The quality of reporting of randomized clinical trials published in Ophthalmology. Ophthalmology 2001; 108: 410-415.
Reference Order: 
18
PubMed ID: 
11158822
Reference Text: 
Latronico N, Botteri M, Minelli C, et al. Quality of reporting of randomised controlled trials in the intensive care literature: a systematic analysis of papers published in Intensive Care Medicine over 26 years. Intensive Care Med 2002; 28: 1316-1323.
Reference Order: 
19
PubMed ID: 
12209283
Reference Text: 
Doig GS, Simpson F, Delaney A. A review of the true methodological quality of nutritional support trials conducted in the critically ill: time for improvement. Anesth Analg 2005; 100: 527-533.
Reference Order: 
20
PubMed ID: 
15673887
Reference Text: 
Torgerson CJ, Torgerson DJ, Birks YF, Porthouse J. A comparison of randomised trials in health and education. Br Educ Res J 2005; 31: 761-785.
Reference Order: 
21
PubMed ID: 
Reference Text: 
Stinson JN, McGrath PJ, Yamada JT. Clinical trials in the Journal of Pediatric Psychology: applying the CONSORT statement. J Pediatr Psychol 2003; 28: 159-167.
Reference Order: 
22
PubMed ID: 
12654939
Reference Text: 
Scherer RW, Crawley B. Reporting of randomized clinical trial descriptors and use of structured abstracts. JAMA 1998; 280: 269-272. Correction in JAMA 1998; 280: 1054.
Reference Order: 
23
PubMed ID: 
9676678
Reference Text: 
Altman DG. Endorsement of the CONSORT statement by high impact medical journals: a survey of instructions for authors. BMJ 2005; 330: 1056-1057.
Reference Order: 
24
PubMed ID: 
15879389
Reference Text: 
Mills E, Wu P, Gagnier J, et al. An analysis of general medical and specialist journals that endorse CONSORT found that reporting was not enforced consistently. J Clin Epidemiol 2005; 58: 662-667.
Reference Order: 
25
PubMed ID: 
15939216
Reference Text: 
Mills EJ, Wu P, Gagnier J, et al. The quality of randomized trial reporting in leading medical journals since the revised CONSORT statement. Contemp Clin Trials 2005; 26: 480-487.
Reference Order: 
26
PubMed ID: 
16054580
Reference Text: 
Devereaux PJ, Manns BJ, Ghali WA, et al. Physician interpretations and textbook definitions of blinding terminology in randomized controlled trials. JAMA 2001; 285: 2000-2003.
Reference Order: 
27
PubMed ID: 
11308438