Connect
MJA

Digital health benefits evaluation frameworks: building the evidence to support Australia's National Digital Health Strategy

Janice S Biggs, Andrea Willcocks, Mitchell Burger and Meredith AB Makeham
Med J Aust 2019; 210 (6): S9-S11. || doi: 10.5694/mja2.50034
Published online: 31 March 2019

Sophisticated methodological approaches and measures for scaling up are key elements of effective evaluation

Digital health technologies and services are significant contributors to the transformation of health care delivery. It has been estimated that 80% of technology projects fail1 due to uncertainty, abandonment and lack of organisational willingness to adopt.2 In response to the high failure rate, the discipline of benefits management has emerged, with the aim of measuring and optimising the value of digital health initiatives. The development and application of benefits management has received some attention,3 but owing to the infancy of the discipline there has been limited assessment of methodological frameworks and their application.

In this article, we describe the framework that is being used to measure and quantify the benefits of the My Health Record system in Australia. We consider the strengths and limitations of this framework in the context of existing frameworks, and its ability to demonstrate digital health system benefits. We also identify priority areas for further development of digital health benefit evaluation frameworks. Further, we provide an overview of the approach to digital health system benefits evaluation in Australia, in the context of the My Health Record system.

The National Digital Health Strategy and the My Health Record system

A role of the Australian Digital Health Agency has been to lead the development of the National Digital Health Strategy, to lay the foundations for a safe, seamless and secure health system.4 This strategy comprises seven priority areas to be achieved by 2022. A key strategic priority is to provide health information whenever and wherever it is needed, and this is underpinned by the My Health Record system.

My Health Record is a secure online summary of health information which can be accessed by people and their health care providers, and is patient‐controlled. It is a personal health record, enabling people to access, manage and share their health information with their clinicians using a range of privacy controls. This functionality includes the ability to decline access to specific documents, set a control to restrict access to the entire record, see an audit trail of any organisations that have accessed the record, and block organisations from viewing the record. The record can contain summary information from general practice, hospital, pharmacy and other health care settings. It may also contain results of investigations, plus documents that patients create themselves (such as advance care plans and personal notes). In addition, it can provide access to Medicare documents such as Medicare Benefits Schedule and Pharmaceutical Benefits Scheme information, the Australian Immunisation Register and the Australian Organ Donor Register.

The Australian Digital Health Agency's benefits evaluation framework

A multimethod evaluation framework has been developed to comprehensively evaluate the benefits of the My Health Record system. It draws on approaches that have been used overseas and assesses the range of clinical contexts in which the system is used.

Connecting people to their own health information has been shown to produce a range of benefits relating to patient engagement and a 60–80% improvement in their adherence to treatment regimens.5 In addition, enabling self‐management has been associated with significant financial savings in terms of health care costs and avoided hospital admissions.6 It is also theorised that enabling clinicians across different health care settings to share information will result in improved patient safety (eg, fewer medication errors), improved care coordination, a reduction in unnecessarily duplicated investigations, and efficiency gains for clinicians in terms of time savings. In Australia, 2–3% of hospital admissions each year relate to medication misadventure, costing an estimated $1.2 billion annually.7 Improved access to medication information from a range of settings through the medicines information view in My Health Record should provide clinicians with more comprehensive information, and it is hoped that this will result in a reduction of medication misadventure events. Moreover, the use of digital health records to enable test results to be shared has been shown to reduce duplicated pathology tests by 18% per week,8 and in primary care settings it has been found that 13.6% of visits were missing important clinical information.9

Several lessons relating to the application of digital health research and evaluation frameworks have been learned. Many different “key measures” have been described, and there is an overall lack of consensus as to the “who, why, how, when and what” that should apply to an evaluation.10 The impact of this has been a failure to capture the complete range of players involved in the successful delivery of a system — players who do not necessarily share identical perspectives. A recent systematic review recommends that future frameworks present better methods for stakeholder identification and have a greater focus on understanding the context in which the system is delivered. From a systems perspective, this includes usability and organisational impact.10

The foundations of Australia's digital health system benefits evaluation framework are based on applying a range of measurement methods to capture a broad variety of outcome measures, reflecting a multistakeholder National Digital Health Strategy which offers different types of benefits to its different stakeholders. Five benefit workstreams have been introduced to evaluate the My Health Record system using qualitative, quantitative and mixed‐method designs, as well as behavioural economic and health economic evaluation methods (Box). These workstreams are:

 

  • customer and market insights;

  • behavioural economics;

  • data analytics;

  • impact evaluations; and

  • health economics evaluations.

 

The workstreams have been designed with data sources in mind to assist with planning and prioritising evaluation measures. These have been ranked by feasibility11 and impact, taking into account the relative importance of each measure to different stakeholders.10

To support two of these workstreams — customer and market insights, and impact evaluations — we have taken a similar approach to Canada's Clinical Adoption Framework.3 The impact evaluations workstream in particular focuses on outcomes and how these are being realised. For example, it focuses on measuring usability and adoption of digital tools from clinicians and consumers from their inception and throughout their development. The behavioural economics workstream refers to the evaluation methods which aim to measure change in the behaviour of system users (eg, clinicians and patients), organisational issues which can affect adoption, and indicators that would inform change management requirements to improve usability and adoption of digital health services.12 The data analytics workstream enables monitoring of trends in adoption and usage. For example, this workstream is investigating, through data modelling, whether My Health Record use is associated with fewer medication errors and reduced unnecessary duplication of pathology tests, to provide comprehensive results from a range of settings. The health economics evaluations workstream evaluates and forecasts indirect population health outcomes and downstream economic benefits using health economics modelling. For example, it is developing a health economics model to inform the ongoing business case for the My Health Record system, and conducting modelling that will support measurement of current and future benefits.

To support an iterative process in product and service design, the Australian Digital Health Agency also undertakes user experience research and agile project management methods.13 The advantage of this approach is that it can deliver user insights which inform continual design and development of new features. Further, conducting field research to observe a range of digital system users in their environment is a way of identifying attributes that can influence the methods and measures used more broadly in our benefits evaluation framework.

Discussion

Current approaches and frameworks that have been developed overseas to support evaluation and benefits measurement of digital health services are at various stages of maturity. An increasing range of methods to evaluate digital health technologies is being supported by publications, which recognise that they are being deployed in complex health systems that require a contextual understanding of users, clinical settings and the policy environment in which they operate.

We must not underestimate the challenges of evaluating benefits of digital health system delivery, and new methods to support evaluations continue to be developed and validated.2 Benefits evaluation frameworks offer a platform that can guide researchers and policy makers in generating and translating evidence to support future directions and ongoing investment in digital health services. Recent debates highlight the importance of fostering evaluation designs which combine different research methods, using qualitative, quantitative and co‐design principles, as well as process measures1 that we have embedded into our benefits framework.

To strengthen our current benefits evaluation framework and overall approach to digital health service evaluation, we are actively introducing methods that ascertain how services can be scaled up, to identify enablers and barriers to implementation across a range of settings.1 In this context, we define scalability as “the ability … to be expanded under real‐world conditions to reach a greater proportion of the eligible population, while retaining effectiveness”.14 Measuring scalability is not a commonly undertaken process and has been described as poorly understood.15 But the Australian Digital Health Agency is working towards applying these methods to build an understanding of impact. These findings form a picture of how, where and for whom the intervention could have the greatest impact and, conversely, what adaptations are needed for interventions to work across different population groups. A practical example of how this is being undertaken through the Agency is the introduction of and investment in a range of “test bed” studies. Test beds are projects assessing new digital‐enabled models of care that are instigated and delivered cooperatively through sustainable and viable partnerships between industry, government and other organisations. Their purpose is to promote innovation to address Australia's highest priority health challenges and generate evidence on how the new approaches improve health outcomes. Currently, there are 15 test beds across Australia which are testing digital infrastructure and integration of digital health into clinical workflows.16

Conclusion

Despite ongoing interest in digital health benefits evaluation frameworks, few examples of their use in evaluation of digital health services have been published. The Australian Digital Health Agency's benefits evaluation framework will be used to justify future funding of digital health and to inform community and clinical education material. Moreover, the findings will be used to inform enhancements of the My Health Record system, ensuring that its progress is relevant and appropriate for clinicians and consumers.

Box – Activities within the Australian Digital Health Agency's benefits management framework

Workstream

Key benefits being measured

Methods used


Customer and market insights

Perceived improvement in access to patients’ information and reduction in the need to order pathology tests and diagnostic imaging
Self‐reported experience of being able to view information that was previously unknown, and saving time requesting information

Survey and analysis of service users (consumers and clinicians) to generate insights into attitudes, experiences and behaviours

Behavioural economics

Changes in clinical practice that lead to a reduction in unnecessary pathology tests and diagnostic imaging

Laboratory‐based scientific evaluation of service use to reveal insights into experience and behaviour and to promote behavioural change

Data analytics

Cross‐sector flows of information facilitated by My Health Record leading to improved access to patients’ information and reduced time spent requesting information

Analysis and modelling of available big data assets (eg, data routinely collected by state and territory departments of health, such as hospital admissions) under ethics approval

Impact evaluations

Reduction in adverse medicine‐related events through having access to a patient's medicines information
Reduction in unnecessary duplication of pathology tests and diagnostic imaging

Within settings that have implemented My Health Record, these evaluations focus on where health outcomes are being realised, and validation of the link between process indicators and outcomes

Health economics evaluations

Economic value of avoided unnecessary duplication of pathology tests and diagnostic imaging, and reduction in adverse medicine‐related events

Evaluation and forecasting of indirect population health outcomes and downstream economic benefits using health economic modelling


 


Provenance: Commissioned; externally peer reviewed.

  • Janice S Biggs
  • Andrea Willcocks
  • Mitchell Burger
  • Meredith AB Makeham

  • Australian Digital Health Agency, Sydney, NSW


Acknowledgements: 

We are grateful to Darian Eckersley, from the Australian Digital Health Agency, for advice on product design methods. Many thanks to the broader Research Programs team at the Agency for reviewing parts of this manuscript and commenting on an earlier version.

Competing interests:

We are all employed by the Australian Digital Health Agency.

  • 1. Greenhalgh T. Why do so many technology projects in healthcare fail? A new framework for studying the non‐adoption, abandonment and failure of scale‐up, spread and sustainability (NASSS) of health and care technologies. https://new.talks.ox.ac.uk/talks/id/4e41675d-317c-467f-8c70-96918ca171eb (viewed May 2018).
  • 2. Greenhalgh T, Wherton J, Papoutsi C, et al. Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale‐up, spread, and sustainability of health and care technologies. J Med Internet Res 2017; 19: e367.
  • 3. Lau F, Kuziemsky C, editors. Handbook of ehealth evaluation: an evidence‐based approach. Victoria, Canada: University of Victoria, 2016.
  • 4. Australian Digital Health Agency. Australia's National Digital Health Strategy. https://www.digitalhealth.gov.au/about-the-agency/publications/australias-national-digital-health-strategy (viewed Jan 2018).
  • 5. Delbanco T, Walker J, Bell SK, et al. Inviting patients to read their doctors’ notes: a quasi‐experimental study and a look ahead. Ann Intern Med 2012; 157: 461–470.
  • 6. Jovicic A, Holroyd‐Leduc JM, Straus SE. Effects of self‐management intervention on health outcomes of patients with heart failure: a systematic review of randomized controlled trials. BMC Cardiovasc Disord 2006; 6: 43.
  • 7. Roughead L, Semple S, Rosenfeld E. Literature review: medication safety in Australia. Sydney: Australian Commission on Safety and Quality in Health Care, 2013.
  • 8. Zlabek JA, Wickus JW, Mathiason MA. Early cost and safety benefits of an inpatient electronic health record. 2011; 18: 169–172.
  • 9. Smith PC, Araya‐Guerra R, Bublitz C, et al. Missing clinical information during primary care visits. JAMA 2005; 293: 565.
  • 10. Eslami Andargoli A, Scheepers H, Rajendran D, Sohal A. Health information systems evaluation frameworks: a systematic review. Int J Med Inform 2017; 97: 195–209.
  • 11. Pan E, Byrne C, Damico D, Crimmins M. Guide to evaluating health information exchange projects. AHRQ Publication No. 14‐0015‐EF. Rockville: Agency for Healthcare Research and Quality, 2014.
  • 12. Murray E, Hekler EB, Andersson G, et al. Evaluating digital health interventions: key questions and approaches. Am J Prev Med 2016; 51: 843–851.
  • 13. Jacobs MA, Graham AL. Iterative development and evaluation methods of mHealth behavior change interventions. Curr Opin Psychol 2016; 9: 33–37.
  • 14. Milat A, Newson R, King L, et al. A guide to scaling up population health interventions. Public Health Res Pract 2016; 26: e2611604.
  • 15. Wang S, Moss JR, Hiller JE. Applicability and transferability of interventions in evidence‐based public health. Health Promot Int 2006; 21: 76–83.
  • 16. Australian Digital Health Agency. Digital Health Test Beds program. https://conversation.digitalhealth.gov.au/digital-health-test-beds-program (viewed Dec 2018).

Author

remove_circle_outline Delete Author
add_circle_outline Add Author

Comment
Do you have any competing interests to declare? *

I/we agree to assign copyright to the Medical Journal of Australia and agree to the Conditions of publication *
I/we agree to the Terms of use of the Medical Journal of Australia *
Email me when people comment on this article

Online responses are no longer available. Please refer to our instructions for authors page for more information.