Three major health reform initiatives — the National Health and Hospitals Reform Commission (NHHRC),1 National Primary Health Care Strategy (NPHCS) working group2 and Preventative Health Taskforce (PHT)3 — released their final reports during 2009. Their processes were remarkably open. Consistent with the Australian Government’s commitment to evidence-informed policy,4,5 each working group placed discussion papers, commissioned papers, submissions and reports on websites in the public domain.6-9 This gave us a rare opportunity to identify the evidence and publications that are used by such working groups to develop their health reform options. The inclusion of citations in final reports allowed analysis of the forms of evidence that supported their decisions

As part of a broader study of the contribution of primary health care research to the health reform initiatives,10 we identified which journals were most frequently cited in the health reform documents. We extracted and coded all references cited in the publicly available discussion documents (n = 4), submissions (n = 1264), commissioned papers (n = 25), an interim report, and final reports (n = 3) from the websites of the three reform initiatives.6-8 As the variety of referencing styles limited the accuracy of the dataset, we undertook more intensive data cleaning and coding of the journal subset to identify the most commonly cited journals. We used Microsoft Excel to organise and code data, and SPSS Version 17.0 for Windows (SPSS Inc, Chicago, Ill, USA) to obtain frequencies. Ethics approval was not required, as all data were in the public domain.

Discussion documents, commissioned papers, submissions and the interim NHHRC report11 contained 8143 references, with almost half (46%; 3713) in documents associated with the NHHRC, 35% (2882) with the PHT and 19% (1548) with the NPHCS. Journal articles were cited most frequently, followed by reports and other publications by government organisations such as the Australian Institute of Health and Welfare and Australian Bureau of Statistics, and documents from Australian university research groups, non-government organisations and private companies. The pattern varied little among the three initiatives (Box 1).

Overall, the Medical Journal of Australia (MJA) was the most frequently cited journal, with 392 references to its articles (11.8% of the total 3321 journal articles cited). The second most cited journal was the BMJ, with 150 citations (4.5%), followed by Tobacco Control, with 131 citations (3.9%), which was cited almost exclusively in documents relating to the PHT (Box 2). A total of 817 journals were cited, with the top 10 providing 32% of citations.

Submissions were cited in the final reports of the NHHRC and NPHCS, but not the PHT. Australian government reports dominated the types of document cited in the final reports of the NHHRC and NPHCS, while the PHT final report cited more journal articles than other types of publication.

The MJA was the most frequently cited journal in the NHHRC and NPHCS final reports, but was overtaken by Tobacco Control in the PHT final report. Overall, the MJA was the most frequently cited journal in all three final reports, with 58 references to its articles (13.7% of the 422 journal articles cited in the final reports). Citations in the final reports were spread over more than 160 journals, the top 10 journals providing 44% of citations.

Of the many sources of evidence used to support arguments for reform, peer-reviewed journals were cited most frequently. The “grey” literature, such as government reports from reputable independent organisations, including the Australian Institute of Health and Welfare and Australian Bureau of Statistics, also played a prominent role. The dominance of MJA citations in consultation documents and final reports demonstrates that its published content was accessible, credible, valued and relevant to the broad health reform agenda of these diverse initiatives.

Limitations to our study included the wide range and variable accuracy of referencing styles, which made it difficult to compile the data accurately. Our results should be seen as indicative only, and we therefore avoided ranking journals, apart from the first three, where the margin for error would not have made a difference. Our study examined just the visible tip of the evidence iceberg for the three working groups. The final reports mark the completion of the policy-development stage and the start of the decision-making stage, in which research competes with other forms of evidence and experience.

1 Type of documents cited in discussion documents, commissioned papers and submissions (n = 8143)

NHHRC = National Health and Hospitals Reform Commission. NPHCS = National Primary Health Care Strategy. PHT = Preventative Health Taskforce. WHO = World Health Organization.