Clinical quality registries for clinician-level reporting: strengths and limitations

Australia should learn from overseas experience of reporting clinician outcomes before considering a similar program

Clinical registries systematically collect clinically relevant data regarding specific diseases or health events using standard procedures and definitions across multiple institutions. They originated as an epidemiological construct, and were designed to measure health outcomes across whole populations, originally for epidemiological and health planning purposes. More recently, the term “clinical quality registry” (CQR) has been introduced to define registries that use specific clinical quality indicators for regular confidential and benchmarked reporting to participating sites.1 CQR reporting at a hospital level acknowledges the often inextricable links between the many factors affecting patient care, including practitioner performance, support staff, facilities, care processes, and pre- and postoperative care. CQR reporting may provide early warning of potential quality issues, and when hospitals with outlying results internally review their data and processes, it may be an effective stimulus for clinical practice change.2

Clinician-level reporting from clinical quality registries

Internationally, there is growing interest in clinician-level reporting. This is driven by diverse stakeholders:

• the medical specialty colleges and health care providers, for quality improvement purposes;

• regulators and jurisdictions, to identify underperforming clinicians and risks to patient safety;

• health insurers, to identify costs associated with variation in practice;3

• consumers, to inform clinician choice; and

• CQRs themselves, to support clinician engagement.

The Australian Commission on Safety and Quality in Health Care Framework for Australian clinical quality registries1 recommends that CQRs produce routine risk-adjusted clinician reports. However, the benefits, limitations and risks associated with clinician-level reporting are not necessarily clear, and opinions vary on whether clinician-level reporting is appropriate.4 Currently in Australia, participation in CQRs is voluntary, and clinician-level reporting is based on a model of confidential feedback to individuals. Because of this, the number and composition of clinical registries that report outcomes to clinicians is unclear, although examples include the Victorian Prostate Cancer Outcomes Registry,5 and the Australian and New Zealand Society of Cardiac and Thoracic Surgeons database.6

In general, CQR data are sufficiently granular to allow reporting of clinician-level outcomes. CQRs are often based around high volume, well defined episodes of care, including procedures for which clinician-level reporting is appropriate. However, when clinical care is undertaken by generalists and non-procedural practitioners and trainees, and through multidisciplinary team-based practice, the potential implementation of broad scale clinician-level reporting is limited.

Public reporting

The first public reporting of clinician-level outcomes internationally arose following media requests for information regarding surgeon outcomes.7,8 The New York State Department of Health first published data from its cardiac surgery database for hospital-level and surgeon-level performance in 1990 and 1992, respectively. It reports data on a rolling 3-year basis (reports are available at https://www.health.ny.gov/statistics/diseases/cardiovascular), which has served as a model for other United States state benchmarking and reporting initiatives. The United Kingdom National Adult Cardiac Surgery Audit has publicly reported at a unit level since 2010–11, and about 80% of surgeons now participate in voluntary clinician-level public reporting.8 The UK National Health Service has published clinical outcome information for individual surgeons across ten specialties since 2013.9

Through its MyHospitals website and jurisdictional reports, the Australian National Health Performance Authority publicly reports and benchmarks various hospital-level indicators such as readmission rates, infection rates, waiting times and financial performance.10 Australia does not currently report clinician performance outcome data publicly. However, as CQRs continue to mature it is likely that the capability to provide clinician-level reports will eventuate.

International experience

Public reporting of clinician performance has been shown to lead to improved patient outcomes, particularly in adult and paediatric cardiac surgery.11,12 A recent systematic review of 28 peer-reviewed articles identified some evidence that public reporting can be an incentive for low-performing surgeons to improve.13 Nevertheless, international experience has highlighted a number of important considerations. These can help inform Australia’s approach to reporting clinician-level outcomes.

Low procedural numbers

Clinician activity volume may be substantially smaller than total activity volume at participating sites, leading to difficulties with statistical analysis of low volume datasets, particularly with less sensitive clinical indicators such as mortality. One UK study determined that the number of procedures necessary for the reliable detection of poor clinician performance (defined as double the national mortality rate), with a statistical power of 80%, well exceeded the typical annual numbers of procedures generally performed.14 This demonstrated that low procedural numbers may mask poor performance and lead to false complacency. Detection of poor clinician performance increased, however, when aggregated data were reported over a 3- or 5-year period.

While pooling of data over long periods reduces the timeliness of reported data and may mask deterioration in a clinician’s performance, low volume activities or low sensitivity indicators may require practical compromises. These include publishing a rolling dataset (eg, the New York State cardiac surgery database), or choosing quality indicators that occur with greater frequency than mortality rates, such as compliance with other measures that have an established association with mortality (eg, major complications or incomplete surgical resection).

Clinical indicator selection

Selection of clinical indicators appropriate to clinician-level reporting is complex, as some factors, including teamwork, communication, organisational culture and environment, may be beyond the individual clinician’s control. While benchmarked outcome measures should always be risk-adjusted to account for inherent differences in patient risk and to minimise the potential for clinician avoidance of high risk patients,15 the determination of and access to appropriate risk-adjustment factors may not always be possible. An alternative approach is to use composite measures, including a range of clinical and patient-reported outcomes that aim to provide a broader overall picture of performance. Such measures are recommended by the US Institute of Medicine,15 and may be appropriate for consideration in the Australian context, recognising however that this too is a complex field and such indicators may be difficult to construct.

Clinician performance management

Benchmarked reporting of sites and clinicians can be used for quality improvement and potentially for performance management. Registry data that highlight poorly performing clinicians may lead to hospitals reviewing or restricting clinical privileges, and clinicians retiring or resigning from the service. Following New York State’s first cardiac surgery reports, a greater proportion of surgeons with higher risk-adjusted mortality rates ceased practising cardiac surgery within 2 years.7

In the US, health service or clinician performance in relation to benchmarked quality indicators is additionally used to determine reimbursement.4 The importance of using high quality clinical indicators developed by consensus and applicable in the real world setting cannot be overstated in any potential funding and reimbursement decisions.

Recommendations

A summary of the potential advantages and limitations of clinician-level reporting is presented in the Box.

In the absence of mandatory clinician participation in CQRs, we are cautious about introducing public reporting of clinician-level data in Australia at this point in time. However, confidential clinician-level reporting may have a place if there is confidence in the accuracy of the data, and if outcomes occur with a level of frequency that would enable outliers to be detected in a timely manner.

It is therefore recommended that mature Australian CQRs adhere to the following principles, drawn from international experience, before considering clinician-level reporting:

• Clinician involvement in clinical indicator and report development is essential. The degree of acceptance and use of reports is dependent on the degree to which the key constituencies are part of their development.

• CQRs should identify the statistical requirements to enable detection of poor performance depending on the clinical indicator. To avoid false complacency from low volume activity, rolling datasets or reporting to the level of the team, unit or the hospital may be considered.

• Careful selection of evidence-based clinical indicators and appropriate risk adjustment where benchmarking is undertaken are critical, and CQRs should monitor for unintended effects such as avoidance of high risk patients.

• CQRs should ensure that systems are in place for professional support and remediation of clinicians identified as outliers. This requires vocational college or specialty society engagement, and collaboration with employing health services and jurisdictions.

Any move toward mandatory participation and public reporting of clinician-level outcomes should only be considered after a robust program of confidential clinician reporting is rigorously evaluated to assess for bias, “cherry picking” of patients, and adequacy of risk adjustment. The required support for culture and practice change will also need to have been in place for a reasonable period of time. Any consideration of performance-based incentives should proceed cautiously as Australia’s CQRs and other clinical information systems are still maturing. Given the potential benefits of public reporting, Australian CQRs should support this eventuality. However, further discussion and collaboration with government, clinicians and other key stakeholders is required to determine the most effective way forward.