Connect
MJA
MJA

Evaluation of the impact of National Breast Cancer Foundation-funded research

Claire Donovan, Linda Butler, Alison J Butt, Teresa H Jones and Stephen R Hanney
Med J Aust 2014; 200 (4): 214-218. || doi: 10.5694/mja13.10798
Published online: 3 March 2014

Abstract

Objective: To evaluate the impact of the National Breast Cancer Foundation’s (NBCF’s) research investment.

Design and participants: Surveys based on the Payback Framework were sent to chief investigators involved in research funded by the NBCF during 1995–2012; a bibliometric analysis of NBCF-funded publications in 2006–2010 was conducted; and a purposive, stratified sample of case studies was obtained.

Main outcome measures: Research impact on knowledge production, the research system, informing policy, product development and broader health and economic benefits.

Results: Of 242 surveys sent, 153 (63%) were returned. The average impact of journals in which NBCF publications appeared was double that of world publications. Seventy surveys (46%) reported career progression, and 185 higher degrees were obtained or expected, including 121 PhDs. One hundred and one grants (66%) produced tools that built capacity across the research system, and research teams leveraged an additional $1.40 in funding for every dollar invested. Fifteen applied grants and one basic grant impacted on policy. Ten basic and four applied grants led to the development of drugs, prognostic tools or diagnostic technologies. Twenty applied and two basic grants led to changes in practice and behaviour of health care staff, consumers and the public, with further impacts anticipated. Case studies provided illustrations of high impact.

Conclusions: NBCF’s strategy of investing in a mixed portfolio of research areas and mechanisms encouraged a broad range of impacts across all Payback categories. The impacts from basic research tended to focus on knowledge production and drug development; while applied research generated greater impacts within the other Payback categories. The funding of shared infrastructure stimulated impact across the research system.

The role of charities in supporting health and medical research is substantial. In 2011, Australia’s total federal, state and non-government organisation spending on cancer research projects, programs and infrastructure was $300 million, of which $83 million was awarded by major national cancer research funding charities, including $9 million by the National Breast Cancer Foundation (NBCF).1 With growing competition for the charity dollar, it is becoming increasingly important for medical research charities to demonstrate the outcomes or benefits of the research that they fund and to identify research strategies that are most likely to produce future benefits.2-4

The NBCF was established in 1994, and by early 2012 had invested over $81 million in over 300 Australian-based research projects. The NBCF seeks to invest in breast cancer research that will have the greatest impact, aiming to reduce the burden of a breast cancer diagnosis for all those living with or at high risk of developing breast cancer, and their families. All funds are raised from and within the Australian community. The NBCF’s research investment strategy supports a commitment to funding across the research continuum from basic to applied research, and the translation of research into clinical settings.5,6 In early 2012, the NBCF’s board of directors sought an independent evaluation of its total investment in breast cancer research to identify and communicate the impacts of that research to the Australian community, and to use the findings as an evidence base to inform and guide the NBCF’s future funding strategies.

Traditional approaches to evaluating the benefits of health and medical research focus on impacts on knowledge production and the research system. However, research funders in Australia have increasingly sought to demonstrate the wider benefits of research, such as improved treatments and health gains.7,8 This has resulted in interest in developments such as the Payback Framework, which fuses traditional measures with broader categorisations of the benefits from health and medical research,9 and is internationally acknowledged as the most widely used comprehensive method for assessing returns from research.10-13

The NBCF commissioned the Health Economics Research Group (HERG) at Brunel University to conduct a Payback study to record the range of impacts across its research investment portfolio. It is the first study of its kind of a program of breast cancer research.

Methods

The Payback Framework9 was applied to the portfolio of NBCF-funded research from 1 January 1995 to 31 March 2012. Ethics approval was obtained from Brunel University’s Research Ethics Committee. Data were collected and analysed in 2012 from March onwards.

Data were collected to measure the impact of NBCF research in terms of:

  • scientific peer-reviewed publications as the central means of sharing knowledge with the research community;

  • dissemination of knowledge produced to academic and non-academic audiences;

  • interaction with the potential end users and beneficiaries of research, which increases the scientific and broader impacts of research;13,14

  • research training and career advancement;

  • capacity building and critical mass to undertake effective research;

  • translation of research into clinical practice, evident in changes to health service policy and decision making and best practice in diagnosis and treatment;

  • development of drugs, prognostic tools or diagnostic technologies;

  • actual health gain, which is often hard to show, but may be evidenced in changes in the behaviour or practice of health care staff, consumers or the public.

Data sources
Results

The response rate for the CI surveys was 63% (153/242). For numbers of not sent, not delivered and non-returned surveys, see Appendix 1 (pdf). The mean grant start date of non-returned surveys was in the range 2004–2005 and returned surveys reported on grants that started in the range 2006–2007. The mean individual value of grants related to returned surveys was $348 096, compared with $170 203 for those related to non-returned surveys. The grants covered by survey responses represented $53 million (66%) of NBCF expenditure.

In the survey data, the balance of funding mechanisms and of applied and basic research was representative of the NBCF’s investment portfolio, as was the focus according to Common Scientific Outline (CSO) classification, except for a slight underrepresentation of “Biology” and overrepresentation of “Cancer control, survivorship and outcomes research” (Box 1).

The mean end year of grants covered by the surveys was in the range 2008–2009, compared with 2007 for all NBCF grants awarded between 1995 and 2012. Of the grants evaluated, 111 (73%) ended from 2008 onwards, and 42 (27%) of all surveyed grants were still in progress when the survey was administered (Box 2).

Case study data provided illustrations of high impact for each Payback category (Box 3). Three case studies are outlined in more detail in Box 4.

Discussion

This assessment was based on information collected from 153 of 242 completed CI surveys (63%), a comparatively good response rate on which to base valid conclusions.15 The surveys were broadly representative of the whole NBCF portfolio in terms of grant mechanisms, balance of applied and basic research, and areas of focus (determined by CSO coding). Non-returned surveys tended to relate to grants that were older and smaller than those covered in the completed surveys, being on average 2 years older and half the value.

The relatively short time frame that had elapsed since many of the surveyed grants had been awarded or completed, with a demonstrable bias of returned surveys towards grants awarded in recent years, may have limited the assessment of impact, as most grants were yet to realise their promise. This was especially the case for basic research, for which downstream impacts take longer to accrue.14

The study also relied on subjective reporting of impacts in CI survey responses and case study interviews. However, multiple data collection approaches allowed comparisons to be made with objective measures such as citation data on the scientific impact of publications, or the impact of research on policy or practice in the form of citations in clinical practice guidelines. Interestingly, triangulation of data sources showed that in surveys, if anything, CIs tended to underreport the impact of their research, as has been noted in previous studies.4,15

While taking these potential limitations into account, this study, nonetheless, found a wide range of impacts across the Payback categories from citations through to informing policy — consistent with the NBCF’s investment in breast cancer research across the whole spectrum of the disease. It found that a mixed portfolio of research areas and mechanisms favoured a broad range of impacts. Because the NBCF is a community-funded organisation, a focus on generating health benefits to those impacted by breast cancer is particularly important. The evaluation found that applied research was effective in producing changes in policy and in the behaviour and practice of health care professionals, consumers and the public; thus generating wider health and health sector benefits, such as improvements in service delivery. The study found that basic research was more closely linked to impacts relating to knowledge production, and to product development in terms of developing drugs, prognostic tools or diagnostic technologies. It also highlighted that the funding of shared resources such as biobanks and other infrastructure was highly valuable to the research system for both basic and applied research.

The 16 case studies validated the survey data and provided rich illustrations of the full range of impacts.

The study builds on previous published examples of Australian research funding evaluations7,8 by adopting a more systematic, multidimensional and comprehensive approach. This is the first Payback study of a program of breast cancer research. Except for the separately commissioned bibliometric analysis, there are no benchmarking data for comparing the impacts with those of other breast cancer research organisations; however, this study creates a baseline for the future. Research supported by the NBCF attracted funds from other sources, such as state-based Cancer Councils, Cancer Australia and the National Health and Medical Research Council, and direct attribution of impacts to particular funders is complex.

This study is particularly timely as there is increasing pressure on both government and non-government research funders to provide evidence of the wider impacts of research. There is a renewed focus in the university sector on assessing the wider impacts of research, with trials of a proposed new initiative, Excellence in Innovation for Australia (EIA), having recently been completed. Many proponents hope to see EIA sitting alongside the existing Excellence in Research for Australia framework for assessing academic research performance.

3 Description of case studies and high impact Payback categories

Case study

Research type

Funding

Duration

Cost

Payback categories


A: Novel approaches towards targeting normoxic and hypoxic cancer tissue

Basic

Novel concept award

24 months (2011–13)

$185 200

Product development

B: Germline BRCA1 and BRCA2 mutations in the Australian Ovarian Cancer Study

Basic

Priority-driven collaborative cancer research scheme

35 months (2008–11)

$200 000

Practice/behaviour

C: The development and evaluation of a menopause information tool for young women after breast cancer

Applied

Novel concept award

36 months (2005–08)

$119 784

Policy/decision making

D: Compliance with breast cancer management guidelines and related health outcomes

Applied

Project grant

36 months (2000–03)

$137 045

Health/health service

E: A twin study of mammographic breast density and the risk of breast cancer

Applied

Project grant

24 months (1996–98)

$180 000

Knowledge production; research system

F: c-KIT: a novel target for the treatment of basal-like breast cancer

Basic

Novel concept award

24 months (2011–13)

$200 000

Product development

G: Investigation of the role of the prolactin receptor in mammary gland development and cancer using prolactin receptor knockout mice

Basic

Project grant

24 months (1997–99)

$183 101

Knowledge production

H: Nanoscaled biosensors: reading epigenetic signatures to improve breast cancer diagnosis and treatment

Basic

Priority-driven collaborative cancer research scheme

36 months (2008–11)

$400 000

Research system; product development

I: Novel strategies for prediction and control of advanced breast cancer via nanoscaled epigenetic-based biosensors

Basic

National collaborative breast cancer research grant program

60 months (2008–13)

$5 000 000

Knowledge production; research system; product development; health/health service

J: Multiple perspectives on sexuality and intimacy post-cancer, leading to the development and evaluation of supportive interventions

Applied

Australian research council linkage grant

48 months (2008–12)

$90 000

Research system; policy/decision making; practice/behaviour; health/health service

K: Identification of new therapeutic targets for metastatic breast cancer

Basic

Career fellowship

60 months (2009–2013)

$995 720 

Knowledge production; research system; product development

L: Physical activity and breast cancer recovery: research to reality

Applied

Postdoctoral training fellowship

48 months (2005–09)

$260 000

Policy/decision making

M: Physical rehabilitation following breast cancer

Applied

Career fellowship

48 months (2009–13)

$1 000 000

Practice/behaviour; health/health service

N: Breast cancer and pregnancy: how does a concurrent or subsequent pregnancy affect breast cancer diagnosis, management and outcomes?

Applied

Doctoral research scholarship

24 months (2006–08)

$61 200

Policy/decision making; practice/behaviour; health/health service

O: The fertility and menopause-related information needs of younger women with a diagnosis of breast cancer

Applied

Doctoral research scholarship

36 months (2002–2004)

$76 800

Knowledge production; health/health service

P: Breast cancer biospecimen resource

Applied

Enabling grant

36 months (2005–08)

$380 000

Research system

4 Case studies

Case Study G

  • The aim of the grant was to define the role of prolactin in mammary gland development

  • 15 journal articles 1997–1999 (808 citations)

  • The prolactin receptor knockout mouse developed is now a widely used model in endocrinology

  • Provided key preclinical model efficacy data supporting development of antiprolactin receptor antibody for treatment of breast cancer

  • Drug development by multinational pharmaceutical company now in Phase II clinical trials

Case Study L

  • The aim of the grant was to investigate the role of physical activity in enhancing functional capacity and quality of life among breast cancer survivors

  • 16 journal articles (289 citations)

  • > 10 presentations to practitioners and consumer groups

  • Informed three different sets of clinical practice guidelines issued by professional associations in Australia

  • Findings adopted in clinical practice

  • Key public health gain is recognition that treatment-related concerns exist for breast cancer survivors and influence quality of life

Case Study P

  • The Australian Breast Cancer Tissue Bank (ABCTB) is a research-enabling infrastructure. The National Breast Cancer Foundation grant provided for collection of tissues, blood and clinical data from newly diagnosed patients with breast cancer; distribution of biospecimens and clinical data to Australian researchers; support of translational research

  • 5200 donors, > 100 000 biospecimens collected, 7000 samples accessed by researchers

  • Around 30 organisations engaged with ABCTB

  • Instrumental in 20 research teams obtaining grants, ranging from Mayo Clinic (USA) to Canberra Hospital

Received 14 June 2013, accepted 24 October 2013

  • Claire Donovan1
  • Linda Butler2
  • Alison J Butt3
  • Teresa H Jones1
  • Stephen R Hanney1

  • 1 Health Economics Research Group, Brunel University, London, UK.
  • 2 School of Politics and International Relations, Australian National University, Canberra, ACT.
  • 3 Research Investment, National Breast Cancer Foundation, Sydney, NSW.


Correspondence: claire.donovan@brunel.ac.uk

Acknowledgements: 

This study was funded by the NBCF. Claire Donovan was funded as the study’s CI, Teresa Jones was funded as a Research Fellow, and Stephen Hanney as a co-investigator. Linda Butler was funded by NBCF to conduct a bibliometric analysis of the scientific impact of recent NBCF-linked publications. We thank NBCF staff for preparing and distributing the surveys, and all the NBCF-funded researchers who contributed to this evaluation. We also acknowledge the input of the Steering Committee.

Competing interests:

Alison Butt is Director of Research Investment at the NBCF.

  • 1. Cancer Research Leadership Forum. Towards a national cancer research plan. White paper. Sydney: National Breast Cancer Foundation, 2012. http://www.prostate.org.au/articleLive/attachments/1/research/crlf_2012_jan.pdf (accessed Oct 2013).
  • 2. Hanney SR, Grant J, Wooding S, Buxton MJ. Proposed methods for reviewing the outcomes of research: the impact of funding by the UK’s Arthritis Research Campaign. Health Res Policy Syst 2004; 2: 4.
  • 3. UK Evaluation Forum. Medical research: assessing the benefits to society. London: UK Evaluation Forum, 2006. http://www.acmedsci. ac.uk/download.php?file=/images/project/Medicalr.pdf (accessed Oct 2013).
  • 4. Hanney SR, Watt A, Jones TH, Metcalf L. Conducting retrospective impact analysis to inform a medical research charity’s funding strategies: the case of Asthma UK. Allergy Asthma Clin Immunol 2013; 9: 17.
  • 5. National Breast Cancer Foundation. Challenging breast cancer: a national action plan for breast cancer research and funding. Sydney: NBCF, 2004. http://www.nbcf.org.au/Research/National-Strategy.aspx (accessed Oct 2013).
  • 6. National Breast Cancer Foundation. Prevention has to start somewhere: our 30 year challenge. Sydney: NBCF, 2010. http://www.nbcf.org.au/Research/National-Strategy.aspx (accessed Oct 2013).
  • 7. Clay MA, Donovan C, Butler L, Oldenburg BF. The returns from cardiovascular research: the impact of the National Heart Foundation of Australia’s investment. Med J Aust 2006; 185: 209-212. <MJA full text>
  • 8. Kingwell BA, Anderson GP, Duckett SJ, et al; NHMRC Evaluations and Outcomes Working Committee. Evaluation of NHMRC funded research completed in 1992, 1997 and 2003: gains in knowledge, health and wealth. Med J Aust 2006; 184: 282-286. <MJA full text>
  • 9. Donovan C, Hanney S. The ‘Payback Framework’ explained. Res Eval 2011; 20: 181-183. doi: 10.3152/095820211X13118583635756.
  • 10. Boaz A, Fitzpatrick S, Shaw B. Assessing the impact of research on policy: a literature review. Science and Public Policy 2009; 36: 255-270. doi: 10.3152/030234209X436545.
  • 11. Banzi R, Moja L, Pistotti V, et al. Conceptual frameworks and empirical approaches used to assess the impact of health research: an overview of reviews. Health Res Policy Syst 2011; 9: 26.
  • 12. Panel on Return on Investment in Health Research. Making an impact: a preferred framework and indicators to measure returns on investment in health research. Ottawa, ON: Canadian Academy of Health Sciences, 2009. http://www.cahs-acss.ca/making-an-impact-a-preferred-framework-and-indicators-to-measure-returns-on-investment-in-health-research-8 (accessed Oct 2013).
  • 13. Milat AJ, Laws R, King L, et al. Policy and practice impacts of applied research: a case study analysis of the New South Wales Health Promotion Demonstration Research Grants Scheme 2000-2006. Health Res Policy Syst 2013; 11: 5.
  • 14. Wooding S, Hanney S, Pollitt A, et al. Project Retrosight: understanding the returns from cardiovascular and stroke research. Cambridge, UK: RAND Europe, 2011. http://www.rand.org/pubs/research_briefs/RB9573.html (accessed Oct 2013).
  • 15. Hanney S, Buxton M, Green C, et al. An assessment of the impact of the NHS Health Technology Assessment Programme. Health Technol Assess 2007; 11: iii-iv, ix-xi, 1-180.

Author

remove_circle_outline Delete Author
add_circle_outline Add Author

Comment
Do you have any competing interests to declare? *

I/we agree to assign copyright to the Medical Journal of Australia and agree to the Conditions of publication *
I/we agree to the Terms of use of the Medical Journal of Australia *
Email me when people comment on this article

Online responses are no longer available. Please refer to our instructions for authors page for more information.