Evaluation of NHMRC funded research completed in 1992, 1997 and 2003: gains in knowledge, health and wealth

Bronwyn A Kingwell, Gary P Anderson, Meg E Morris, Stephen J Duckett, Elizabeth A Hoole, Jan Rothwell-Short, Lisa R Jackson-Pulver, Levon M Khachigian, David M Roder and Andrew J Wilson, for the National Health and Medical Research Council Evaluations and Outcomes Working Committee
Med J Aust 2006; 184 (6): 282-286. || doi: 10.5694/j.1326-5377.2006.tb00238.x
Published online: 20 March 2006


Objective: To report on strategies for, and outcomes of, evaluation of knowledge (publications), health and wealth (commercial) gains from medical research funded by the Australian Government through the National Health and Medical Research Council (NHMRC).

Design and methods: End-of-grant reports submitted by researchers within 6 months of completion of NHMRC funded project grants which terminated in 2003 were used to capture self-reported publication number, health and wealth gains. Self-reported gains were also examined in retrospective surveys of grants completed in 1992 and 1997 and awards primarily supporting people (“people awards”) held between 1992 and 2002.

Results: The response rate for the 1992 sample was too low for meaningful analysis. The mean number of publications per grant in the basic biomedical, clinical and health services research areas was very similar in 1997 and 2003. The publication output for population health was somewhat higher in the 2003 than in the 1997 analysis. For grants completed in 1997, 24% (31/131) affected clinical practice; 14% (18/131) public health practice; 9% (12/131) health policy; and 41% (54/131) had commercial potential with 20% (26/131) resulting in patents. Most respondents (89%) agreed that NHMRC people awards improved their career prospects. Interpretation is limited by the relatively low response rates (50% or less).

Conclusions: A mechanism has been developed for ongoing assessment of NHMRC funded research. This process will improve accountability to the community and to government, and refine current funding mechanisms to most efficiently deliver health and economic returns for Australia.

The National Health and Medical Research Council (NHMRC) is Australia’s leading agency responsible for the development and maintenance of public and individual health standards. It funds health and medical research through a highly competitive grant system, and in 2005 awarded grants amounting to $412 million.

The past decade has seen increased pressures for accountability in the public sector, including for research funding.1-4 Australian medical research is highly innovative and translated into medically useful treatments.5-8 However, funders of health and medical research expect quantitative evidence of research impact. Additional funding for the NHMRC will need to be supported by evidence of health and wealth (commercial) gains from the increased funding delivered following the Health and Medical Research Strategic (Wills) Review in 2000.9 In particular, the NHMRC will be required to report against Australia’s National Research Priorities10 and National Health Priorities11 (Box 1). Recognising this challenge, the NHMRC established an Evaluation and Outcomes Working Committee in 2003.

The NHMRC has previously commissioned bibliometric analyses of Australia’s health and medical research publication output.12-14 Published articles indicate knowledge gain, but are not a complete measure of value.15 The task of the Evaluations and Outcomes Working Committee is to evaluate health and wealth gains derived from knowledge gained through NHMRC funding. A key strategy was to enhance the content, compliance rate for completion, and evaluation of end-of-grant reports (Box 2). The new grant reports18 allow better monitoring of compliance, and provide an input to strategic evaluation. They are designed to provide summary statistics of grant outcomes. In addition, database management procedures were strengthened to enable easy retrieval of information. Another strategy was the retrospective follow-up of the outcomes of grants completed in 1992 and 1997 and awards primarily supporting people held between 1992 and 2002 (Box 2). We report the collective outcome of these evaluations.

Grants completed in 1992 and 1997

A contracted web-based survey of NHMRC project grants which terminated in 1992 and 1997 was completed in June 2005.16 A simplified version of the new end-of-grant report was used as the survey instrument. Primary investigators were contacted by email and in some instances by telephone and invited to complete the survey. Current email addresses were sourced from the NHMRC; however, these were unavailable for 35% of the 1992 sample and 26% of the 1997 sample.


One hundred and thirty-one reports of grants completed in 2003 (29% of 454 expected reports) were returned and reviewed by a panel of 18 experts.

For the surveys of grants completed in 1992 and 1997, the response rate was 44% of the contactable sample in 1992 (61/139), and 51% (131/259) in 1997.

Of the eligible population of 1897 recipients of people awards, 596 (31%) completed the survey.

Knowledge gain
Grants completed in 1997

Given the greater eligible sample (259 in 1997 compared with 139 in 1992) and the higher response rate (51% in 1997, 44% in 1992), we focus predominantly on 1997. However, trends were similar for both the 1992 and 1997 analysis.16

In 1997, 68% (89/131) of all grants surveyed focused on one or more of the national research priority areas. There were 206 reports of alignment with national research priorities among these 89 grants. Most focused on one or more aspects of promoting and maintaining good health (125/206, 61%). Of these 125, 45 targeted ageing well, ageing productively; 45 preventive health care; 28 a healthy start to life; and 7 strengthening Australia’s social and economic fabric.

In 1997, 57% of grants (75/131) focused on one or more national health priorities. There were 101 reports of alignment with national health priorities among these 75 grants. Of all grants, 23% (30/131) focused on cancer; followed by cardiovascular disease, 16% (21/131); mental health 13% (17/131); and arthritis and musculoskeletal conditions, 11% (15/131).

The publication number per grant in the basic biomedical sciences, the clinical sciences and health services research was very similar in 1997 and 2003 (Box 3).

In 1997, a mean of 1.3 (range, 0–10) candidates per grant progressed or completed their postgraduate training. This is similar to the results for 2003. For basic biomedical sciences, the mean was 1.5 (range, 0–10) per grant; for clinical medicine, 1 (range, 0–4) per grant; and for public health and health services research, 0.7 (range, 0–3) per grant.


The bibliometric analyses reported previously12-14 and the preliminary NHMRC grant outcome evaluations reported here indicate that NHMRC grants make a significant contribution to knowledge creation through scientific publication. Importantly, knowledge from a substantial number of grants over a 7–12-year time frame is translated to outcomes that affect Australia’s health and economic wellbeing. However, it is important to consider the potential for sample bias, given the relatively low response rates in our preliminary assessments. In the future, it will also be important to determine methods to quantify the economic benefit of changes in practice and policy (in terms, for example, of both changed health care costs and improved patient outcomes leading to human productivity). Likewise, it will be important to capture evidence of the economic effect arising from commercialisation and patenting activity.5

Comparison of results from the 1997 and 2003 analyses is of interest. However, there are two major confounding variables. First, the size of grants and the cost of research both increased from 1997 to 2003. Second, the time frame for assessment in the two sets was different: 6 months after grant completion for the 2003 sample, and 7 years after completion for the 1997 sample. However, it is likely that a 7-year retrospective analysis of the 2003 grants would reveal further publications, and that grant expenditure per publication would be largely similar to that for the 1997 analysis. The publication output for population health was higher in the 2003 compared with the 1997 analysis. This may be due to greater recent investment in this area, reflected by the increase in the average grant expenditure per publication in 2003 compared with 1997.

Although Australian medical research has been translated into medically useful treatments,5-7 most development of Australian medical research was not performed in Australia.8,9 Australian ideas and publications were over-represented in the US patent literature on patent applications filed by non-Australian entities.9 If Australia is to contribute significantly to international health advances, a larger fraction of innovation must be developed in Australia. The level of engagement with the commercialisation process in the 1997 analysis is a gratifying outcome given that this is often the first step in developing a product which may contribute to health gain. The local protection of intellectual property allows inventors to interact with Australian biotechnology or small innovative companies and/or larger multinationals in a way that should bring substantial economic returns. It will be important to track the future development and commercialisation of patents reported in the current cohort of grants.

There was an overwhelmingly positive assessment of NHMRC people awards in contributing to the development of knowledge and skills.19 In addition, the high reputation and competitive standard of these awards was considered very important in improving future career prospects. On the negative side, there was insecurity about career pathways and employment prospects which was perceived to inhibit research development. This probably relates to the highly competitive and fixed term nature of people awards, both of which reflect the limited funding available.

Our first attempt to evaluate outcomes beyond publications is also useful for refining the mechanism of data capture. Measures are currently being implemented to improve the rate of end-of-grant report return, which exceeded 80% for 2005. Although the retrospective analyses collected data from grants completed in 1992 and 1997, it was only possible to obtain an adequate sample size from grants completed 7 years before the survey (1997 sample). Such retrospective analysis is important, as not all manuscripts arising from a grant are accepted for publication at the time a final report is due. Similarly, not all higher degree students will have completed their candidature. The health and wealth outcomes of grants are also unlikely to have developed significantly 6 months after completion of funding. However, future potential is certainly identifiable. Linkage between funding and outcomes over longer periods may be better tracked in the reverse direction from measurable outcomes to their funding sources.17 However, this approach may not be appropriate, given the increased variety of funding contributions that underpin research.

We suggest that a 5–7-year reporting time frame allows a sufficient period for health and wealth outcomes to have appeared, while minimising loss to follow-up. The magnitude and effect of research translation is still difficult to quantify, given both problems with outcome data collection and the time lag between grant completion and research translation. The end-of-grant report and the retrospective survey instrument will be further refined on the basis of this initial experience.

Communication of outcomes of research funded by the NHMRC to the Australian community is an important aspect of medical research accountability. In the 2003 analysis, a panel of experts identified 10 grants for showcasing as case-studies in a booklet highlighting research outcomes to the community (“Ten of the Best”,20 Box 5). This type of community reporting will probably be expanded as the NHMRC moves to an outcome-based approach to research reporting. Strategic groupings, organised according to major disease, population and discipline groups, will be responsible for identifying the strengths and gaps and for recommending strategies to ensure that there is appropriate capacity (people, funds or facilities) to best meet future challenges.

As the evaluation process is expanded, it will be possible to relate outcomes back to specific funding schemes so that the most effective and efficient funding mechanisms can be identified.21 Outcome analyses can then be used to inform future funding strategies. As well as identifying Australian research strengths, outcome evaluation will highlight areas of weakness that require specific investment and capacity-building.

In conclusion, outcome analyses and attribution to the initial funding source presents a significant challenge to all research funding agencies. There is no single established method that provides valid assessment of the specific outcomes of a single grant. In the last triennium (2003–2005), the NHMRC has laid the foundation for a number of different evaluation methodologies. However, further significant work is required to refine databases and outcome analyses. The goal will be to establish the highest standards of accountability and community dialogue, and to refine current funding mechanisms to most efficiently deliver health and economic returns for Australia.

Received 1 November 2005, accepted 12 January 2006

  • Bronwyn A Kingwell1
  • Gary P Anderson2
  • Meg E Morris2
  • Stephen J Duckett3
  • Elizabeth A Hoole4
  • Jan Rothwell-Short4
  • Lisa R Jackson-Pulver5
  • Levon M Khachigian5
  • David M Roder6
  • Andrew J Wilson7
  • for the National Health and Medical Research Council Evaluations and Outcomes Working Committee

  • 1 Alfred and Baker Medical Unit, Baker Medical Research Institute, Melbourne, VIC.
  • 2 University of Melbourne, Melbourne, VIC.
  • 3 Queensland Health, Brisbane, QLD.
  • 4 National Health and Medical Research Council, Canberra, ACT.
  • 5 University of New South Wales, Sydney, NSW.
  • 6 Cancer Council of South Australia, Adelaide, SA.
  • 7 University of Queensland, Brisbane, QLD.



The Evaluations and Outcomes Working Committee is grateful for the support of the office of the National Health and Medical Research Council, and in particular, the secretariat support provided by project officers Lisa Teasdale and Liam Whyte. Thanks also to Bob Williamson (chair) and the grant evaluation committee (Julie Campbell, Allan Carmichael, John Chalmers, Julian Clarke, Mark Cooper, Caroline Finch, Janet Greeley, Christine Jenkin, Tony Jorm, John Hamilton, Michele Kosky, Stephen Leeder, Kerin O’Dea, Allan McLean, Brian Oldenburg, Jeanette Ward, Judy Whitworth) for evaluation of the grants completed in 2003 and for developing “Ten of the Best”.20

  • 1. Ferguson B, Kelly P, Georgiou A, et al. Assessing payback from NHS reactive research programmes. J Manag Med 2000; 14: 25-36.
  • 2. Buxton M, Hanney S. How can payback from health services research be assessed? J Health Serv Res Policy 1996; 1: 35-43.
  • 3. Buxton M, Hanney S, Jones T. Estimating the economic value to societies of the impact of health research: a critical review. Bull World Health Organ 2004; 82: 733-739.
  • 4. Hanney SR, Grant J, Wooding S, Buxton MJ. Proposed methods for reviewing the outcomes of health research: the impact of funding by the UK’s ‘Arthritis Research Campaign’. Health Res Policy Syst 2004; 2: 4.
  • 5. Access Economics. Exceptional returns: the value of investing in health R&D in Australia. Canberra: Australian Society for Medical Research, 2003. Available at: (accessed Feb 2006).
  • 6. Australian Society for Medical Research. Australian case studies. Sydney: ASMR, 2002. Available at: (accessed Feb 2006).
  • 7. Australian Society for Medical Research. Australian case studies II. Sydney: ASMR, 2003. Available at: (accessed Feb 2006).
  • 8. Mattes E, Stacey MC. Australian medical patents granted in the United States in 1984–1999. Med J Aust 2001; 174: 83-87. <MJA full text>
  • 9. Wills PJ (Chair). The virtuous cycle: working together for health and medical research. Health and medical research strategic review discussion document. Canberra: Commonwealth of Australia, 1998. Available at: (accessed Feb 2006).
  • 10. Department of Education, Science and Training. National research priorities and their associated priority goals. Canberra: DEST, 2003. Available at: (accessed Feb 2006)
  • 11. National Health Priority Action Council. National Health Priority Areas [website]. 2002. Available at: (accessed Feb 2006).
  • 12. Bourke P, Butler L. Mapping Australia’s basic research in the medical and health sciences. Med J Aust 1997; 167: 610-613. <MJA full text>
  • 13. Butler L. NHMRC-supported research: the impact of journal publication output 1996–2000. Canberra: NHMRC, 2003. Available at: (accessed Feb 2006).
  • 14. Anderson WP. Funding Australia’s health and medical research. Med J Aust 1997; 167: 608-609. <MJA full text>
  • 15. Butler L. A list of published papers is no measure of value. Nature 2002; 419: 877.
  • 16. Consan Consulting. Analysis of NHMRC funded research in 1992 and 1997. Canberra: NHMRC, 2005. Available at: (accessed Feb 2006).
  • 17. Comroe JH Jr, Dripps RD. Scientific basis for the support of biomedical science. Science 1976; 192: 105-111.
  • 18. National Health and Medical Research Council end of grant report form. 2005. Available at: (accessed Feb 2006).
  • 19. Boreham P, Western J, Laffan W, et al. Survey of NHMRC research workforce outcomes: 1992–2002. Brisbane: University of Queensland Social Research Centre, 2005. Available at: (accessed Feb 2006).
  • 20. National Health and Medical Research Council Evaluations and Outcomes Committee. Ten of the Best. NHMRC funded health and medical research successes. Canberra: NHMRC, 2005. Available at: (accessed Feb 2006).
  • 21. Bourke P, Butler L. The efficacy of different modes of funding research: perspectives from Australian data on the biological sciences. Res Policy 1999; 28: 489-499.


remove_circle_outline Delete Author
add_circle_outline Add Author

Do you have any competing interests to declare? *

I/we agree to assign copyright to the Medical Journal of Australia and agree to the Conditions of publication *
I/we agree to the Terms of use of the Medical Journal of Australia *
Email me when people comment on this article

Online responses are no longer available. Please refer to our instructions for authors page for more information.