The National Health and Medical Research Council (NHMRC) is Australia’s leading agency responsible for the development and maintenance of public and individual health standards. It funds health and medical research through a highly competitive grant system, and in 2005 awarded grants amounting to $412 million.
The past decade has seen increased pressures for accountability in the public sector, including for research funding.1-4 Australian medical research is highly innovative and translated into medically useful treatments.5-8 However, funders of health and medical research expect quantitative evidence of research impact. Additional funding for the NHMRC will need to be supported by evidence of health and wealth (commercial) gains from the increased funding delivered following the Health and Medical Research Strategic (Wills) Review in 2000.9 In particular, the NHMRC will be required to report against Australia’s National Research Priorities10 and National Health Priorities11 (Box 1). Recognising this challenge, the NHMRC established an Evaluation and Outcomes Working Committee in 2003.
The NHMRC has previously commissioned bibliometric analyses of Australia’s health and medical research publication output.12-14 Published articles indicate knowledge gain, but are not a complete measure of value.15 The task of the Evaluations and Outcomes Working Committee is to evaluate health and wealth gains derived from knowledge gained through NHMRC funding. A key strategy was to enhance the content, compliance rate for completion, and evaluation of end-of-grant reports (Box 2). The new grant reports18 allow better monitoring of compliance, and provide an input to strategic evaluation. They are designed to provide summary statistics of grant outcomes. In addition, database management procedures were strengthened to enable easy retrieval of information. Another strategy was the retrospective follow-up of the outcomes of grants completed in 1992 and 1997 and awards primarily supporting people held between 1992 and 2002 (Box 2). We report the collective outcome of these evaluations.
Investigators holding NHMRC project grants that were completed in 2003 were required to submit a report by June 2004.16
A contracted web-based survey of NHMRC project grants which terminated in 1992 and 1997 was completed in June 2005.16 A simplified version of the new end-of-grant report was used as the survey instrument. Primary investigators were contacted by email and in some instances by telephone and invited to complete the survey. Current email addresses were sourced from the NHMRC; however, these were unavailable for 35% of the 1992 sample and 26% of the 1997 sample.
The NHMRC funds individuals, including PhD scholars, postdoctoral fellows, early career researchers and senior fellows. These awards are collectively called “people awards”. A contracted survey of NHMRC workforce outcomes from people awards funded between 1992 and 2002 was completed in June 2005, and was conducted using a computer-assisted telephone interview system.19 In addition to assessing knowledge, health and wealth gains, the survey tracked career paths, benefits of the scheme with regard to collaborations, networking and opportunities, and deficiencies in the system, including barriers and gaps.
For all evaluations, the main output indicator for knowledge gain was the number of publications. No attempt was made to assess publication quality or impact, which has been done more comprehensively using other methods.12-14
The main evaluations reported are for health and wealth gains. Health gain was defined as any research with a self-reported effect on clinical practice or other health service delivery practice or outcomes, changes in public health practice, or changes in health policy. Wealth gain was defined as any research with self-reported commercial activity, including commercial potential and patents.
One hundred and thirty-one reports of grants completed in 2003 (29% of 454 expected reports) were returned and reviewed by a panel of 18 experts.
For the surveys of grants completed in 1992 and 1997, the response rate was 44% of the contactable sample in 1992 (61/139), and 51% (131/259) in 1997.
Of the eligible population of 1897 recipients of people awards, 596 (31%) completed the survey.
In 2003, 56% (73/131) of grants reviewed focused on National Health Priority Areas. Cancer accounted for 23% (30/131) of all grants; cardiovascular disease, 11% (14/131); mental health, 11% (15/131); diabetes 6% (8/131); asthma, 3% (4/131); and arthritis and musculoskeletal conditions, 2% (2/131) (some research grants addressed more than one national health priority area).
The publication output per grant in specific disciplines is shown in Box 3, along with the average grant expenditure per publication. However, this latter figure does not reflect total expenditure per publication, as the amount contributed by other funding sources, including those supporting the salaries of senior investigators, is unknown.
Training of postgraduate students is also an indicator of knowledge gain. In 2003, a mean of 1.5 candidates (range, 0–6) per grant progressed or completed their postgraduate training. For basic biomedical sciences, the mean was 2 (range, 0–6) per grant; for clinical medicine, 1.2 (range, 0–4) per grant; and for public health and health services research, 1.4 (range, 0–4) per grant.
Given the greater eligible sample (259 in 1997 compared with 139 in 1992) and the higher response rate (51% in 1997, 44% in 1992), we focus predominantly on 1997. However, trends were similar for both the 1992 and 1997 analysis.16
In 1997, 68% (89/131) of all grants surveyed focused on one or more of the national research priority areas. There were 206 reports of alignment with national research priorities among these 89 grants. Most focused on one or more aspects of promoting and maintaining good health (125/206, 61%). Of these 125, 45 targeted ageing well, ageing productively; 45 preventive health care; 28 a healthy start to life; and 7 strengthening Australia’s social and economic fabric.
In 1997, 57% of grants (75/131) focused on one or more national health priorities. There were 101 reports of alignment with national health priorities among these 75 grants. Of all grants, 23% (30/131) focused on cancer; followed by cardiovascular disease, 16% (21/131); mental health 13% (17/131); and arthritis and musculoskeletal conditions, 11% (15/131).
The publication number per grant in the basic biomedical sciences, the clinical sciences and health services research was very similar in 1997 and 2003 (Box 3).
In 1997, a mean of 1.3 (range, 0–10) candidates per grant progressed or completed their postgraduate training. This is similar to the results for 2003. For basic biomedical sciences, the mean was 1.5 (range, 0–10) per grant; for clinical medicine, 1 (range, 0–4) per grant; and for public health and health services research, 0.7 (range, 0–3) per grant.
Effect on practice was assessed primarily by principal investigator self-report (Box 4). Examples of clinical or policy effects arising from their grant as delineated by investigators16 included incorporation of outcomes of clinical trials into practice. According to one chief investigator, “the use of antibiotics and particularly beta-lactam antibiotics was reduced in the community studied”. Another reported practice benefit was increased awareness of the importance of rotavirus infection causing hospitalisation of young children.
Health policy outcomes included a more rigorous basis for falls prevention programs (particularly influencing policy in New South Wales and Western Australia).
Public health effects from grants completed in 1992 included reduction in infant mortality and change in advice to parents about the safe sleeping position of infants from the finding that a prone sleeping position increased the risk of sudden infant death syndrome (SIDS).
Research completed in 1992 also led to changes to the design and implementation of control programs worldwide against the hydatid tapeworm.
In 1997, 41% (54/131) of grants were self-reported as having commercial potential. Of all grants, 20% (26/131) resulted in one or more patents (Box 4): 33% were pending, 50% had been granted, and 17% had lapsed.
Patents covered diverse areas such as new diagnostics for liver disease, new methods to prevent and treat inflammatory skin disorders, and new cancer therapeutics. There were also multiple patents around vaccine technology, including vaccines aimed at preventing multiple diseases.
Ninety-eight per cent (577/589) of respondents indicated that their NHMRC award developed expertise which was relevant or highly relevant to Australia’s medical research and development needs, and 95% (560/589) indicated that the expertise was relevant to international needs. Furthermore, 89% (524/589) of respondents indicated that their NHMRC award improved their career prospects.
On the negative side, 81% (478/587) agreed that lack of continuing employment is a barrier to effective medical research, and 72% (423/587) believed that career paths in medical research are poorly defined. Eighty per cent (470/585) disagreed that the Australian health and medical research environment provides a secure and long-term career path. Funding was considered to be the major impediment to the development of research by 61% (362/594) of those surveyed.
The bibliometric analyses reported previously12-14 and the preliminary NHMRC grant outcome evaluations reported here indicate that NHMRC grants make a significant contribution to knowledge creation through scientific publication. Importantly, knowledge from a substantial number of grants over a 7–12-year time frame is translated to outcomes that affect Australia’s health and economic wellbeing. However, it is important to consider the potential for sample bias, given the relatively low response rates in our preliminary assessments. In the future, it will also be important to determine methods to quantify the economic benefit of changes in practice and policy (in terms, for example, of both changed health care costs and improved patient outcomes leading to human productivity). Likewise, it will be important to capture evidence of the economic effect arising from commercialisation and patenting activity.5
Comparison of results from the 1997 and 2003 analyses is of interest. However, there are two major confounding variables. First, the size of grants and the cost of research both increased from 1997 to 2003. Second, the time frame for assessment in the two sets was different: 6 months after grant completion for the 2003 sample, and 7 years after completion for the 1997 sample. However, it is likely that a 7-year retrospective analysis of the 2003 grants would reveal further publications, and that grant expenditure per publication would be largely similar to that for the 1997 analysis. The publication output for population health was higher in the 2003 compared with the 1997 analysis. This may be due to greater recent investment in this area, reflected by the increase in the average grant expenditure per publication in 2003 compared with 1997.
Although Australian medical research has been translated into medically useful treatments,5-7 most development of Australian medical research was not performed in Australia.8,9 Australian ideas and publications were over-represented in the US patent literature on patent applications filed by non-Australian entities.9 If Australia is to contribute significantly to international health advances, a larger fraction of innovation must be developed in Australia. The level of engagement with the commercialisation process in the 1997 analysis is a gratifying outcome given that this is often the first step in developing a product which may contribute to health gain. The local protection of intellectual property allows inventors to interact with Australian biotechnology or small innovative companies and/or larger multinationals in a way that should bring substantial economic returns. It will be important to track the future development and commercialisation of patents reported in the current cohort of grants.
There was an overwhelmingly positive assessment of NHMRC people awards in contributing to the development of knowledge and skills.19 In addition, the high reputation and competitive standard of these awards was considered very important in improving future career prospects. On the negative side, there was insecurity about career pathways and employment prospects which was perceived to inhibit research development. This probably relates to the highly competitive and fixed term nature of people awards, both of which reflect the limited funding available.
Our first attempt to evaluate outcomes beyond publications is also useful for refining the mechanism of data capture. Measures are currently being implemented to improve the rate of end-of-grant report return, which exceeded 80% for 2005. Although the retrospective analyses collected data from grants completed in 1992 and 1997, it was only possible to obtain an adequate sample size from grants completed 7 years before the survey (1997 sample). Such retrospective analysis is important, as not all manuscripts arising from a grant are accepted for publication at the time a final report is due. Similarly, not all higher degree students will have completed their candidature. The health and wealth outcomes of grants are also unlikely to have developed significantly 6 months after completion of funding. However, future potential is certainly identifiable. Linkage between funding and outcomes over longer periods may be better tracked in the reverse direction from measurable outcomes to their funding sources.17 However, this approach may not be appropriate, given the increased variety of funding contributions that underpin research.
We suggest that a 5–7-year reporting time frame allows a sufficient period for health and wealth outcomes to have appeared, while minimising loss to follow-up. The magnitude and effect of research translation is still difficult to quantify, given both problems with outcome data collection and the time lag between grant completion and research translation. The end-of-grant report and the retrospective survey instrument will be further refined on the basis of this initial experience.
Communication of outcomes of research funded by the NHMRC to the Australian community is an important aspect of medical research accountability. In the 2003 analysis, a panel of experts identified 10 grants for showcasing as case-studies in a booklet highlighting research outcomes to the community (“Ten of the Best”,20 Box 5). This type of community reporting will probably be expanded as the NHMRC moves to an outcome-based approach to research reporting. Strategic groupings, organised according to major disease, population and discipline groups, will be responsible for identifying the strengths and gaps and for recommending strategies to ensure that there is appropriate capacity (people, funds or facilities) to best meet future challenges.
As the evaluation process is expanded, it will be possible to relate outcomes back to specific funding schemes so that the most effective and efficient funding mechanisms can be identified.21 Outcome analyses can then be used to inform future funding strategies. As well as identifying Australian research strengths, outcome evaluation will highlight areas of weakness that require specific investment and capacity-building.
In conclusion, outcome analyses and attribution to the initial funding source presents a significant challenge to all research funding agencies. There is no single established method that provides valid assessment of the specific outcomes of a single grant. In the last triennium (2003–2005), the NHMRC has laid the foundation for a number of different evaluation methodologies. However, further significant work is required to refine databases and outcome analyses. The goal will be to establish the highest standards of accountability and community dialogue, and to refine current funding mechanisms to most efficiently deliver health and economic returns for Australia.
1 Priority areas
National Research Priority Areas10
An environmentally sustainable Australia
Promoting and maintaining good health
Frontier technologies for building and transforming Australian industries
Safeguarding Australia
National Health Priority Areas11
2 NHMRC evaluations and outcomes strategy
All assessments are self-reported outcomes from either end-of-grant reports (short-term) or from retrospective outcome surveys (medium-term). |
|||||||||||||||
3 Publication output per grant
* Average expenditure per grant = total $ value of grant/number of publications. Both analyses were conducted in 2005. |
|||||||||||||||
5 “Ten of the Best”, selected from grants completed in 2003
Stem cell transplants to treat leukaemia
New strategies for asthma and allergy treatment
Risk factors for multiple sclerosis
Genetic testing for haemochromatosis
Diabetes self-management in Aboriginal communities
Gene therapy to correct blindness
A percutaneous device to treat mitral valve failure
Nicotine addiction and regulation of the tobacco industry
Imaging to improve diagnosis of brain infections
Tissue engineering for reconstructive surgery
For details see reference.20 A panel of 18 experts reviewed end-of-grant reports completed in 2003 (see Acknowledgements), and identified 10 grants with clinically relevant outcomes for showcasing to the community.
Received 1 November 2005, accepted 12 January 2006
- Bronwyn A Kingwell1
- Gary P Anderson2
- Meg E Morris2
- Stephen J Duckett3
- Elizabeth A Hoole4
- Jan Rothwell-Short4
- Lisa R Jackson-Pulver5
- Levon M Khachigian5
- David M Roder6
- Andrew J Wilson7
- for the National Health and Medical Research Council Evaluations and Outcomes Working Committee
- 1 Alfred and Baker Medical Unit, Baker Medical Research Institute, Melbourne, VIC.
- 2 University of Melbourne, Melbourne, VIC.
- 3 Queensland Health, Brisbane, QLD.
- 4 National Health and Medical Research Council, Canberra, ACT.
- 5 University of New South Wales, Sydney, NSW.
- 6 Cancer Council of South Australia, Adelaide, SA.
- 7 University of Queensland, Brisbane, QLD.
The Evaluations and Outcomes Working Committee is grateful for the support of the office of the National Health and Medical Research Council, and in particular, the secretariat support provided by project officers Lisa Teasdale and Liam Whyte. Thanks also to Bob Williamson (chair) and the grant evaluation committee (Julie Campbell, Allan Carmichael, John Chalmers, Julian Clarke, Mark Cooper, Caroline Finch, Janet Greeley, Christine Jenkin, Tony Jorm, John Hamilton, Michele Kosky, Stephen Leeder, Kerin O’Dea, Allan McLean, Brian Oldenburg, Jeanette Ward, Judy Whitworth) for evaluation of the grants completed in 2003 and for developing “Ten of the Best”.20
- 1. Ferguson B, Kelly P, Georgiou A, et al. Assessing payback from NHS reactive research programmes. J Manag Med 2000; 14: 25-36.
- 2. Buxton M, Hanney S. How can payback from health services research be assessed? J Health Serv Res Policy 1996; 1: 35-43.
- 3. Buxton M, Hanney S, Jones T. Estimating the economic value to societies of the impact of health research: a critical review. Bull World Health Organ 2004; 82: 733-739.
- 4. Hanney SR, Grant J, Wooding S, Buxton MJ. Proposed methods for reviewing the outcomes of health research: the impact of funding by the UK’s ‘Arthritis Research Campaign’. Health Res Policy Syst 2004; 2: 4.
- 5. Access Economics. Exceptional returns: the value of investing in health R&D in Australia. Canberra: Australian Society for Medical Research, 2003. Available at: http://www.accesseconomics.com.au/reports/asmreport.pdf (accessed Feb 2006).
- 6. Australian Society for Medical Research. Australian case studies. Sydney: ASMR, 2002. Available at: http://www.asmr.org.au/news/CS.pdf (accessed Feb 2006).
- 7. Australian Society for Medical Research. Australian case studies II. Sydney: ASMR, 2003. Available at: http://www.asmr.org.au/news/CSMay03.pdf (accessed Feb 2006).
- 8. Mattes E, Stacey MC. Australian medical patents granted in the United States in 1984–1999. Med J Aust 2001; 174: 83-87. <MJA full text>
- 9. Wills PJ (Chair). The virtuous cycle: working together for health and medical research. Health and medical research strategic review discussion document. Canberra: Commonwealth of Australia, 1998. Available at: http://www.health.gov.au/internet/wcms/publishing.nsf/content/hmrsr.htm (accessed Feb 2006).
- 10. Department of Education, Science and Training. National research priorities and their associated priority goals. Canberra: DEST, 2003. Available at: http://www.dest.gov.au/sectors/research_sector/publications_resources/profiles/national_research_priorities_and_priority_goals.htm (accessed Feb 2006)
- 11. National Health Priority Action Council. National Health Priority Areas [website]. 2002. Available at: http://www.nhpac.gov.au/nhpas.htm (accessed Feb 2006).
- 12. Bourke P, Butler L. Mapping Australia’s basic research in the medical and health sciences. Med J Aust 1997; 167: 610-613. <MJA full text>
- 13. Butler L. NHMRC-supported research: the impact of journal publication output 1996–2000. Canberra: NHMRC, 2003. Available at: http://www7.health.gov.au/nhmrc/publications/_files/butler03.pdf (accessed Feb 2006).
- 14. Anderson WP. Funding Australia’s health and medical research. Med J Aust 1997; 167: 608-609. <MJA full text>
- 15. Butler L. A list of published papers is no measure of value. Nature 2002; 419: 877.
- 16. Consan Consulting. Analysis of NHMRC funded research in 1992 and 1997. Canberra: NHMRC, 2005. Available at: http://www.nhmrc.gov.au/publications/_files/r36.pdf (accessed Feb 2006).
- 17. Comroe JH Jr, Dripps RD. Scientific basis for the support of biomedical science. Science 1976; 192: 105-111.
- 18. National Health and Medical Research Council end of grant report form. 2005. Available at: http://www.nhmrc.gov.au/publications/_files/nhmrcfinrep05.doc (accessed Feb 2006).
- 19. Boreham P, Western J, Laffan W, et al. Survey of NHMRC research workforce outcomes: 1992–2002. Brisbane: University of Queensland Social Research Centre, 2005. Available at: http://www.nhmrc.gov.au/publications/_files/r37.pdf (accessed Feb 2006).
- 20. National Health and Medical Research Council Evaluations and Outcomes Committee. Ten of the Best. NHMRC funded health and medical research successes. Canberra: NHMRC, 2005. Available at: http://www.nhmrc.gov.au/publications/_files/thebest.pdf (accessed Feb 2006).
- 21. Bourke P, Butler L. The efficacy of different modes of funding research: perspectives from Australian data on the biological sciences. Res Policy 1999; 28: 489-499.
Abstract
Objective: To report on strategies for, and outcomes of, evaluation of knowledge (publications), health and wealth (commercial) gains from medical research funded by the Australian Government through the National Health and Medical Research Council (NHMRC).
Design and methods: End-of-grant reports submitted by researchers within 6 months of completion of NHMRC funded project grants which terminated in 2003 were used to capture self-reported publication number, health and wealth gains. Self-reported gains were also examined in retrospective surveys of grants completed in 1992 and 1997 and awards primarily supporting people (“people awards”) held between 1992 and 2002.
Results: The response rate for the 1992 sample was too low for meaningful analysis. The mean number of publications per grant in the basic biomedical, clinical and health services research areas was very similar in 1997 and 2003. The publication output for population health was somewhat higher in the 2003 than in the 1997 analysis. For grants completed in 1997, 24% (31/131) affected clinical practice; 14% (18/131) public health practice; 9% (12/131) health policy; and 41% (54/131) had commercial potential with 20% (26/131) resulting in patents. Most respondents (89%) agreed that NHMRC people awards improved their career prospects. Interpretation is limited by the relatively low response rates (50% or less).
Conclusions: A mechanism has been developed for ongoing assessment of NHMRC funded research. This process will improve accountability to the community and to government, and refine current funding mechanisms to most efficiently deliver health and economic returns for Australia.