Connect
MJA
MJA

Evaluation of the impact of National Breast Cancer Foundation-funded research

Claire Donovan, Linda Butler, Alison J Butt, Teresa H Jones and Stephen R Hanney
Med J Aust 2014; 200 (4): 214-218. || doi: 10.5694/mja13.10798
Published online: 3 March 2014

Abstract

Objective: To evaluate the impact of the National Breast Cancer Foundation’s (NBCF’s) research investment.

Design and participants: Surveys based on the Payback Framework were sent to chief investigators involved in research funded by the NBCF during 1995–2012; a bibliometric analysis of NBCF-funded publications in 2006–2010 was conducted; and a purposive, stratified sample of case studies was obtained.

Main outcome measures: Research impact on knowledge production, the research system, informing policy, product development and broader health and economic benefits.

Results: Of 242 surveys sent, 153 (63%) were returned. The average impact of journals in which NBCF publications appeared was double that of world publications. Seventy surveys (46%) reported career progression, and 185 higher degrees were obtained or expected, including 121 PhDs. One hundred and one grants (66%) produced tools that built capacity across the research system, and research teams leveraged an additional $1.40 in funding for every dollar invested. Fifteen applied grants and one basic grant impacted on policy. Ten basic and four applied grants led to the development of drugs, prognostic tools or diagnostic technologies. Twenty applied and two basic grants led to changes in practice and behaviour of health care staff, consumers and the public, with further impacts anticipated. Case studies provided illustrations of high impact.

Conclusions: NBCF’s strategy of investing in a mixed portfolio of research areas and mechanisms encouraged a broad range of impacts across all Payback categories. The impacts from basic research tended to focus on knowledge production and drug development; while applied research generated greater impacts within the other Payback categories. The funding of shared infrastructure stimulated impact across the research system.

The role of charities in supporting health and medical research is substantial. In 2011, Australia’s total federal, state and non-government organisation spending on cancer research projects, programs and infrastructure was $300 million, of which $83 million was awarded by major national cancer research funding charities, including $9 million by the National Breast Cancer Foundation (NBCF).1 With growing competition for the charity dollar, it is becoming increasingly important for medical research charities to demonstrate the outcomes or benefits of the research that they fund and to identify research strategies that are most likely to produce future benefits.2-4

The NBCF was established in 1994, and by early 2012 had invested over $81 million in over 300 Australian-based research projects. The NBCF seeks to invest in breast cancer research that will have the greatest impact, aiming to reduce the burden of a breast cancer diagnosis for all those living with or at high risk of developing breast cancer, and their families. All funds are raised from and within the Australian community. The NBCF’s research investment strategy supports a commitment to funding across the research continuum from basic to applied research, and the translation of research into clinical settings.5,6 In early 2012, the NBCF’s board of directors sought an independent evaluation of its total investment in breast cancer research to identify and communicate the impacts of that research to the Australian community, and to use the findings as an evidence base to inform and guide the NBCF’s future funding strategies.

Traditional approaches to evaluating the benefits of health and medical research focus on impacts on knowledge production and the research system. However, research funders in Australia have increasingly sought to demonstrate the wider benefits of research, such as improved treatments and health gains.7,8 This has resulted in interest in developments such as the Payback Framework, which fuses traditional measures with broader categorisations of the benefits from health and medical research,9 and is internationally acknowledged as the most widely used comprehensive method for assessing returns from research.10-13

The NBCF commissioned the Health Economics Research Group (HERG) at Brunel University to conduct a Payback study to record the range of impacts across its research investment portfolio. It is the first study of its kind of a program of breast cancer research.

Methods

The Payback Framework9 was applied to the portfolio of NBCF-funded research from 1 January 1995 to 31 March 2012. Ethics approval was obtained from Brunel University’s Research Ethics Committee. Data were collected and analysed in 2012 from March onwards.

Data were collected to measure the impact of NBCF research in terms of:

  • scientific peer-reviewed publications as the central means of sharing knowledge with the research community;

  • dissemination of knowledge produced to academic and non-academic audiences;

  • interaction with the potential end users and beneficiaries of research, which increases the scientific and broader impacts of research;13,14

  • research training and career advancement;

  • capacity building and critical mass to undertake effective research;

  • translation of research into clinical practice, evident in changes to health service policy and decision making and best practice in diagnosis and treatment;

  • development of drugs, prognostic tools or diagnostic technologies;

  • actual health gain, which is often hard to show, but may be evidenced in changes in the behaviour or practice of health care staff, consumers or the public.

Data sources
Data held by the NBCF

Archival material that was made available comprised a database with descriptive information about all NBCF grants, a database of publications linked to NBCF grants, and copies of grant proposals, annual reports and end-of-grant reports for the case studies.

Survey of chief investigators

A questionnaire was structured around the multidimensional categorisation of impacts in the Payback Framework. It also investigated factors influencing the dissemination and utilisation of research. A total of 242 surveys were sent to the chief investigators (CIs) involved in current and previously awarded grants. This excluded grants that had not yet started or had recently commenced and CIs who were unavailable for personal reasons.

Case studies

Sixteen case studies were conducted to provide validation of data collected from the survey and to supply richer information about the full extent of wider impacts of NBCF research. The HERG compiled a shortlist of potential case studies based on reported high impact in at least one of the Payback categories in the CI survey and high-impact project publications identified in Elsevier’s Scopus database. The case studies were selected using a purposive, stratified approach to ensure all Payback categories were represented across the range of grant types funded.

Each case study included document and archival review, citation analysis, searches for citations in clinical practice guidelines, and semi-structured telephone interviews with CIs. Each case study was described in a narrative account organised according to the stages of the Payback model, and sent to CIs for final validation. All CIs gave informed consent to participate in the study, and approved the narrative accounts.

Bibliometric analysis

Bibliometric analysis of the scientific impact of recent NBCF-linked publications was conducted. This used an existing database of publications from Elsevier’s Scopus citation index developed to analyse the impact of research conducted by the Victorian Comprehensive Cancer Centre (VCCC). Approval was obtained from Elsevier and VCCC for its use in this study. The publication window was the 5-year period 2006 to 2010. Two hundred and sixty-two NBCF publications were identified. The citation window was 1 January 2006 to 1 May 2011. Elsevier calculated world and Australian benchmarks for aggregate citation rates, and thresholds to identify highly cited journal articles.

Results

The response rate for the CI surveys was 63% (153/242). For numbers of not sent, not delivered and non-returned surveys, see Appendix 1 (pdf). The mean grant start date of non-returned surveys was in the range 2004–2005 and returned surveys reported on grants that started in the range 2006–2007. The mean individual value of grants related to returned surveys was $348 096, compared with $170 203 for those related to non-returned surveys. The grants covered by survey responses represented $53 million (66%) of NBCF expenditure.

In the survey data, the balance of funding mechanisms and of applied and basic research was representative of the NBCF’s investment portfolio, as was the focus according to Common Scientific Outline (CSO) classification, except for a slight underrepresentation of “Biology” and overrepresentation of “Cancer control, survivorship and outcomes research” (Box 1).

The mean end year of grants covered by the surveys was in the range 2008–2009, compared with 2007 for all NBCF grants awarded between 1995 and 2012. Of the grants evaluated, 111 (73%) ended from 2008 onwards, and 42 (27%) of all surveyed grants were still in progress when the survey was administered (Box 2).

Case study data provided illustrations of high impact for each Payback category (Box 3). Three case studies are outlined in more detail in Box 4.

Knowledge production
Publications

The bibliometric study found that between 2006 and 2010, the relative citation impact of the 262 NBCF-linked publications was 2.03, double the world benchmark (1.0) and exceeding the Australian average (1.40). The journals in which NBCF-linked publications were cited had an impact of double the world average. The analysis also found almost double the expected number of NBCF publications among the most cited articles (12 in the top 5%).

Dissemination

NBCF grant holders had made considerable efforts to disseminate the knowledge produced to academic and non-academic audiences (Appendix 2 (pdf)). Case studies showed that the findings of NBCF research were broadcast in Australia and internationally through a wide variety of platforms, thereby contributing to the pool of global scientific knowledge, supplying an evidence base for consumer groups, policymakers and clinical practice, and also raising community awareness of breast cancer and related concerns.

Interaction with research users

Data on interaction with research users can be found in Appendix 3 (pdf).

Research system
Research training and career development

As a direct consequence of the NBCF grant, 185 higher degrees were gained or expected, including 121 PhDs, and almost half of CI surveys (70 [46%]) reported career progression for at least one team member, such as promotion or gaining tenure.

Capacity building

The surveyed CIs reported that 101 grants (66%) generated tools for future research that would help to build capacity across the research system, such as new or significantly improved animal models, cell lines, physiological or biochemical markers, assays, models of disease, or validated patient questionnaires.

Targeting further research and attracting further income

The NBCF’s research investment of $53 million in the surveyed grants had generated significant opportunities to attract further funding totalling $74.5 million: for every dollar the NBCF invested, research teams leveraged an additional $1.40.

Informing policy development

The survey found that 15 applied grants had impacted on policy and 32 expected to do so, compared with one basic research grant impacting on policy and 13 expecting to do so. Data on actual and expected wider impacts can be found in Appendix 4 (pdf).

Informing product development

Ten basic grants and four applied grants had led to product development of drugs, prognostic tools or diagnostic technologies. Three applied research projects had informed a broader spectrum of product development; for example, development of a decision aid, a psychological treatment protocol and a sports bra.

Health gain and broader economic benefits

The survey results showed that applied research grants had made the largest contribution to changes in behaviour or practice, with 20 applied grants having realised impacts and 41 expecting to do so, compared with two and 18 basic grants, respectively.

Discussion

This assessment was based on information collected from 153 of 242 completed CI surveys (63%), a comparatively good response rate on which to base valid conclusions.15 The surveys were broadly representative of the whole NBCF portfolio in terms of grant mechanisms, balance of applied and basic research, and areas of focus (determined by CSO coding). Non-returned surveys tended to relate to grants that were older and smaller than those covered in the completed surveys, being on average 2 years older and half the value.

The relatively short time frame that had elapsed since many of the surveyed grants had been awarded or completed, with a demonstrable bias of returned surveys towards grants awarded in recent years, may have limited the assessment of impact, as most grants were yet to realise their promise. This was especially the case for basic research, for which downstream impacts take longer to accrue.14

The study also relied on subjective reporting of impacts in CI survey responses and case study interviews. However, multiple data collection approaches allowed comparisons to be made with objective measures such as citation data on the scientific impact of publications, or the impact of research on policy or practice in the form of citations in clinical practice guidelines. Interestingly, triangulation of data sources showed that in surveys, if anything, CIs tended to underreport the impact of their research, as has been noted in previous studies.4,15

While taking these potential limitations into account, this study, nonetheless, found a wide range of impacts across the Payback categories from citations through to informing policy — consistent with the NBCF’s investment in breast cancer research across the whole spectrum of the disease. It found that a mixed portfolio of research areas and mechanisms favoured a broad range of impacts. Because the NBCF is a community-funded organisation, a focus on generating health benefits to those impacted by breast cancer is particularly important. The evaluation found that applied research was effective in producing changes in policy and in the behaviour and practice of health care professionals, consumers and the public; thus generating wider health and health sector benefits, such as improvements in service delivery. The study found that basic research was more closely linked to impacts relating to knowledge production, and to product development in terms of developing drugs, prognostic tools or diagnostic technologies. It also highlighted that the funding of shared resources such as biobanks and other infrastructure was highly valuable to the research system for both basic and applied research.

The 16 case studies validated the survey data and provided rich illustrations of the full range of impacts.

The study builds on previous published examples of Australian research funding evaluations7,8 by adopting a more systematic, multidimensional and comprehensive approach. This is the first Payback study of a program of breast cancer research. Except for the separately commissioned bibliometric analysis, there are no benchmarking data for comparing the impacts with those of other breast cancer research organisations; however, this study creates a baseline for the future. Research supported by the NBCF attracted funds from other sources, such as state-based Cancer Councils, Cancer Australia and the National Health and Medical Research Council, and direct attribution of impacts to particular funders is complex.

This study is particularly timely as there is increasing pressure on both government and non-government research funders to provide evidence of the wider impacts of research. There is a renewed focus in the university sector on assessing the wider impacts of research, with trials of a proposed new initiative, Excellence in Innovation for Australia (EIA), having recently been completed. Many proponents hope to see EIA sitting alongside the existing Excellence in Research for Australia framework for assessing academic research performance.

1 All National Breast Cancer Foundation (NBCF) grants and grants reported in survey responses, by mode of funding and research type, 1995–2012, and by Common Scientific Outline (CSO) classification, 2000 onwards*

All NBCF grants, no. (%)

Surveyed grants, no. (%)

Funding mode

n = 313

n = 153


Research projects

231 (74%)

111 (73%)

Fellowships

38 (12%)

21 (14%)

Doctoral scholarships

30 (10%)

16 (10%)

National resources

14 (4%)

5 (3%)

Research type

Applied

125 (40%)

66 (43%)

Basic

177 (57%)

84 (55%)

Equipment

11 (3%)

3 (2%)

CSO classification

n = 220

n = 122

Biology

54 (25%)

25 (20%)

Aetiology

22 (10%)

12 (10%)

Prevention

11 (5%)

7 (6%)

Early detection, diagnosis, prognosis

37 (17%)

18 (15%)

Treatment

26 (12%)

15 (12%)

Cancer control, survivorship and outcomes research

51 (23%)

37 (30%)

Scientific model systems

18 (8%)

9 (7%)


* Since 2000, NBCF has classified all grants awarded according to the International Cancer Research Partnership’s Common Scientific Outline. Includes three top-up grants. Includes two top-up grants.

2 Distribution of the 153 grants covered by survey responses, by research type and end year of funding

3 Description of case studies and high impact Payback categories

Case study

Research type

Funding

Duration

Cost

Payback categories


A: Novel approaches towards targeting normoxic and hypoxic cancer tissue

Basic

Novel concept award

24 months (2011–13)

$185 200

Product development

B: Germline BRCA1 and BRCA2 mutations in the Australian Ovarian Cancer Study

Basic

Priority-driven collaborative cancer research scheme

35 months (2008–11)

$200 000

Practice/behaviour

C: The development and evaluation of a menopause information tool for young women after breast cancer

Applied

Novel concept award

36 months (2005–08)

$119 784

Policy/decision making

D: Compliance with breast cancer management guidelines and related health outcomes

Applied

Project grant

36 months (2000–03)

$137 045

Health/health service

E: A twin study of mammographic breast density and the risk of breast cancer

Applied

Project grant

24 months (1996–98)

$180 000

Knowledge production; research system

F: c-KIT: a novel target for the treatment of basal-like breast cancer

Basic

Novel concept award

24 months (2011–13)

$200 000

Product development

G: Investigation of the role of the prolactin receptor in mammary gland development and cancer using prolactin receptor knockout mice

Basic

Project grant

24 months (1997–99)

$183 101

Knowledge production

H: Nanoscaled biosensors: reading epigenetic signatures to improve breast cancer diagnosis and treatment

Basic

Priority-driven collaborative cancer research scheme

36 months (2008–11)

$400 000

Research system; product development

I: Novel strategies for prediction and control of advanced breast cancer via nanoscaled epigenetic-based biosensors

Basic

National collaborative breast cancer research grant program

60 months (2008–13)

$5 000 000

Knowledge production; research system; product development; health/health service

J: Multiple perspectives on sexuality and intimacy post-cancer, leading to the development and evaluation of supportive interventions

Applied

Australian research council linkage grant

48 months (2008–12)

$90 000

Research system; policy/decision making; practice/behaviour; health/health service

K: Identification of new therapeutic targets for metastatic breast cancer

Basic

Career fellowship

60 months (2009–2013)

$995 720 

Knowledge production; research system; product development

L: Physical activity and breast cancer recovery: research to reality

Applied

Postdoctoral training fellowship

48 months (2005–09)

$260 000

Policy/decision making

M: Physical rehabilitation following breast cancer

Applied

Career fellowship

48 months (2009–13)

$1 000 000

Practice/behaviour; health/health service

N: Breast cancer and pregnancy: how does a concurrent or subsequent pregnancy affect breast cancer diagnosis, management and outcomes?

Applied

Doctoral research scholarship

24 months (2006–08)

$61 200

Policy/decision making; practice/behaviour; health/health service

O: The fertility and menopause-related information needs of younger women with a diagnosis of breast cancer

Applied

Doctoral research scholarship

36 months (2002–2004)

$76 800

Knowledge production; health/health service

P: Breast cancer biospecimen resource

Applied

Enabling grant

36 months (2005–08)

$380 000

Research system

4 Case studies

Case Study G

  • The aim of the grant was to define the role of prolactin in mammary gland development

  • 15 journal articles 1997–1999 (808 citations)

  • The prolactin receptor knockout mouse developed is now a widely used model in endocrinology

  • Provided key preclinical model efficacy data supporting development of antiprolactin receptor antibody for treatment of breast cancer

  • Drug development by multinational pharmaceutical company now in Phase II clinical trials

Case Study L

  • The aim of the grant was to investigate the role of physical activity in enhancing functional capacity and quality of life among breast cancer survivors

  • 16 journal articles (289 citations)

  • > 10 presentations to practitioners and consumer groups

  • Informed three different sets of clinical practice guidelines issued by professional associations in Australia

  • Findings adopted in clinical practice

  • Key public health gain is recognition that treatment-related concerns exist for breast cancer survivors and influence quality of life

Case Study P

  • The Australian Breast Cancer Tissue Bank (ABCTB) is a research-enabling infrastructure. The National Breast Cancer Foundation grant provided for collection of tissues, blood and clinical data from newly diagnosed patients with breast cancer; distribution of biospecimens and clinical data to Australian researchers; support of translational research

  • 5200 donors, > 100 000 biospecimens collected, 7000 samples accessed by researchers

  • Around 30 organisations engaged with ABCTB

  • Instrumental in 20 research teams obtaining grants, ranging from Mayo Clinic (USA) to Canberra Hospital

Received 14 June 2013, accepted 24 October 2013

  • Claire Donovan1
  • Linda Butler2
  • Alison J Butt3
  • Teresa H Jones1
  • Stephen R Hanney1

  • 1 Health Economics Research Group, Brunel University, London, UK.
  • 2 School of Politics and International Relations, Australian National University, Canberra, ACT.
  • 3 Research Investment, National Breast Cancer Foundation, Sydney, NSW.


Acknowledgements: 

This study was funded by the NBCF. Claire Donovan was funded as the study’s CI, Teresa Jones was funded as a Research Fellow, and Stephen Hanney as a co-investigator. Linda Butler was funded by NBCF to conduct a bibliometric analysis of the scientific impact of recent NBCF-linked publications. We thank NBCF staff for preparing and distributing the surveys, and all the NBCF-funded researchers who contributed to this evaluation. We also acknowledge the input of the Steering Committee.

Competing interests:

Alison Butt is Director of Research Investment at the NBCF.

  • 1. Cancer Research Leadership Forum. Towards a national cancer research plan. White paper. Sydney: National Breast Cancer Foundation, 2012. http://www.prostate.org.au/articleLive/attachments/1/research/crlf_2012_jan.pdf (accessed Oct 2013).
  • 2. Hanney SR, Grant J, Wooding S, Buxton MJ. Proposed methods for reviewing the outcomes of research: the impact of funding by the UK’s Arthritis Research Campaign. Health Res Policy Syst 2004; 2: 4.
  • 3. UK Evaluation Forum. Medical research: assessing the benefits to society. London: UK Evaluation Forum, 2006. http://www.acmedsci. ac.uk/download.php?file=/images/project/Medicalr.pdf (accessed Oct 2013).
  • 4. Hanney SR, Watt A, Jones TH, Metcalf L. Conducting retrospective impact analysis to inform a medical research charity’s funding strategies: the case of Asthma UK. Allergy Asthma Clin Immunol 2013; 9: 17.
  • 5. National Breast Cancer Foundation. Challenging breast cancer: a national action plan for breast cancer research and funding. Sydney: NBCF, 2004. http://www.nbcf.org.au/Research/National-Strategy.aspx (accessed Oct 2013).
  • 6. National Breast Cancer Foundation. Prevention has to start somewhere: our 30 year challenge. Sydney: NBCF, 2010. http://www.nbcf.org.au/Research/National-Strategy.aspx (accessed Oct 2013).
  • 7. Clay MA, Donovan C, Butler L, Oldenburg BF. The returns from cardiovascular research: the impact of the National Heart Foundation of Australia’s investment. Med J Aust 2006; 185: 209-212. <MJA full text>
  • 8. Kingwell BA, Anderson GP, Duckett SJ, et al; NHMRC Evaluations and Outcomes Working Committee. Evaluation of NHMRC funded research completed in 1992, 1997 and 2003: gains in knowledge, health and wealth. Med J Aust 2006; 184: 282-286. <MJA full text>
  • 9. Donovan C, Hanney S. The ‘Payback Framework’ explained. Res Eval 2011; 20: 181-183. doi: 10.3152/095820211X13118583635756.
  • 10. Boaz A, Fitzpatrick S, Shaw B. Assessing the impact of research on policy: a literature review. Science and Public Policy 2009; 36: 255-270. doi: 10.3152/030234209X436545.
  • 11. Banzi R, Moja L, Pistotti V, et al. Conceptual frameworks and empirical approaches used to assess the impact of health research: an overview of reviews. Health Res Policy Syst 2011; 9: 26.
  • 12. Panel on Return on Investment in Health Research. Making an impact: a preferred framework and indicators to measure returns on investment in health research. Ottawa, ON: Canadian Academy of Health Sciences, 2009. http://www.cahs-acss.ca/making-an-impact-a-preferred-framework-and-indicators-to-measure-returns-on-investment-in-health-research-8 (accessed Oct 2013).
  • 13. Milat AJ, Laws R, King L, et al. Policy and practice impacts of applied research: a case study analysis of the New South Wales Health Promotion Demonstration Research Grants Scheme 2000-2006. Health Res Policy Syst 2013; 11: 5.
  • 14. Wooding S, Hanney S, Pollitt A, et al. Project Retrosight: understanding the returns from cardiovascular and stroke research. Cambridge, UK: RAND Europe, 2011. http://www.rand.org/pubs/research_briefs/RB9573.html (accessed Oct 2013).
  • 15. Hanney S, Buxton M, Green C, et al. An assessment of the impact of the NHS Health Technology Assessment Programme. Health Technol Assess 2007; 11: iii-iv, ix-xi, 1-180.

Author

remove_circle_outline Delete Author
add_circle_outline Add Author

Comment
Do you have any competing interests to declare? *

I/we agree to assign copyright to the Medical Journal of Australia and agree to the Conditions of publication *
I/we agree to the Terms of use of the Medical Journal of Australia *
Email me when people comment on this article

Responses are now closed for this article.