Connect
MJA
MJA

Using the quality improvement cycle on clinical indicators — improve or remove?

Kirstine M Sketcher-Baker, Maarten C Kamp, Julia A Connors, Don J Martin and Justin E Collins
Med J Aust 2010; 193 (8): S104. || doi: 10.5694/j.1326-5377.2010.tb04023.x
Published online: 18 October 2010

Prior to 2007, Queensland Health clinicians selected a suite of clinical indicators from a range of sources and refined these to ensure their relevance to the Queensland context for inclusion in hospital-specific annual reports. Annual reports were initially disseminated to hospital executives only, presenting cross-sectional analysis of 12-month risk-adjusted outcome rates and highlighting the statistical significance of variation between hospital and state rates. Although the intention of these reports was to prompt local review to determine the cause of variation, it was recognised that the reports were not used as intended because the data were at least 1 year old at publication and the methodology summarised results over 1 year, hiding trends.

Since 2007, Queensland Health has employed a statistical process control methodology that has been adapted for clinical utility, the variable life-adjusted display (VLAD), to systematically monitor patient outcomes (Box).1-3 VLAD methodology based on available administrative data has been found to provide results equivalent to more resource-intensive clinical audit methodology that uses individual chart review.4 Queensland Health hospitals have undertaken numerous quality initiatives as a result of the use of VLADs, leading to improvements in discharge processes, clinician documentation, patient education, implementation of and adherence to standard clinical pathways, and allocation of resources to areas of most need.

One of the key elements that led to these initiatives is having well defined indicators that are clinically relevant, clinically significant in terms of burden of disease, and clinically responsive to changes in practice. Queensland Health has used the quality improvement cycle (plan, do, check and act cycle)5 to improve or remove indicators, suspending two indicators, developing two new indicators and refining three indicators. A review of the balance of indicators is currently underway. This process of development and refinement has led to a closer working relationship with clinicians, and resulted in improved indicator definitions to more effectively monitor safety and quality.

Methods

To overcome issues of timeliness and sensitivity, the VLAD methodology was applied to the 31 existing clinical indicator definitions and disseminated monthly to Queensland public and private hospitals. VLADs graphically display patient outcomes, “flagging” if predetermined levels of variation in patient outcomes are exceeded. Flags require a VLAD review to examine data and clinical practice to detect potential patient safety and quality issues and identify action to resolve issues.6-8 Formal documentation of the VLAD review is required within 30 days of notification and is evaluated within a governance structure that was introduced concurrently with the VLADs.9 Indicator definition issues identified by hospital staff are also captured in the VLAD review documentation. This feedback, and issues captured through an online feedback form,10 is then used to improve or remove clinical indicator definitions.

Results

Two indicators for which the quality improvement cycle was recently applied within Queensland were laparoscopic cholecystectomy complications of surgery and third- and fourth-degree perineal tears during childbirth. The “plan” and “act” parts of the cycle had been in place for at least 12 months, with hospitals using the VLAD indicators to examine data and review practice. During this time, feedback from clinicians, quality coordinators and coders suggested these indicator definitions were problematic, triggering a review of the indicator definition — the “check” part of the quality improvement cycle.

Laparoscopic cholecystectomy

Clinical improvement opportunities were identified as a result of clinical reviews of laparoscopic cholecystectomy undertaken by hospitals following a VLAD flag. One hospital instituted a strict policy of single use when it identified that a laparoscopic clip applicator that had been reused a number of times failed to secure the cystic duct. Another hospital, as a result of a significant increase in bleeding complications, changed practice to ensure locum surgeons were supervised by a senior surgeon and given less complex cases to manage.

However, although quality improvement opportunities were identified using this indicator, feedback in reviews from several hospitals indicated that the definition was problematic as it also captured minor complications (eg, small liver capsule lacerations and gallbladder puncture), and it identified complications not relevant to the laparoscopic cholecystectomy (eg, complications from a previous procedure). Another hospital reported that due to its rigour in documenting minor complications in the operative notes, the reported complication rate was elevated. Unfortunately, the initial intention of this indicator (to monitor general complications of surgery [eg, viscus perforation and intra-abdominal haemorrhage] and complications specific to the procedure [eg, damage to the common bile duct]) was being masked by the minor complications.

As a result of this feedback, the definition was suspended to enable a critical review of the definition; this is underway with an indicator working group comprising surgeons, coders, data analysts, and data collection staff. They are considering clinical review findings and definition issues that were raised, and will agree to a solution by consensus. Initial findings of the review indicate that a more clinically appropriate complication of surgery definition (ie, less sensitive) will not be suitable for the VLAD methodology; however, a definition combining readmissions, complications and other outcomes as well as an alternative monitoring technique is being explored.

Perineal tears

The Australian Council on Healthcare Standards, Women’s Hospitals Australasia (WHA) and the WHA’s Core Maternity Indicators Project11 have used third- and fourth-degree perineal tears as clinical indicators for some years. The usefulness of the perineal tear indicator was challenged by some Queensland Health clinicians, who argued that there was no evidence that intervention could reduce the perineal tear rate.

Consequently, the VLAD perineal tear indicator was suspended and an indicator working group comprising rural and metropolitan midwives, data analysts, data collection staff, an obstetrician, a consumer and a rural general practitioner-obstetrician was formed as a subgroup of the Statewide Maternity and Neonatal Clinical Network to assess the perineal tear and existing VLAD maternity definitions. The working group initially undertook an education session on defining clinical indicators and understanding VLAD methodology, and progressed to robust discussions on indicator definition specifics, relevance, clarity, reflected priorities of clinicians, and responsive potential. The Queensland Centre for Mothers and Babies also helped conduct a literature review of evidence regarding prevention of perineal tears.

During discussions with the clinicians who initially objected to the inclusion of the indicator, it became apparent that the perineal tear indicator had been promoted by their hospital management as a performance tool rather than a monitoring tool. This incorrect use of the tool led to the indicator being viewed as a definitive measure of the level of safety and quality of care provided at the hospital. However, VLADs are not direct measures of quality; they simply flag variation from the mean and prompt further review.

The other significant issue raised in objection to inclusion of this indicator was also based on the misunderstanding that each obstetric VLAD indicator was viewed in isolation, whereas the indicators were intended to be viewed in conjunction with related indicators (eg, caesarean section, episiotomy). Although the indicator definition changes did not occur as a result of these misconceptions, it was important from a clinician engagement perspective to acknowledge and address them. This was achieved with a presentation to the stakeholders at the Statewide Maternity and Neonatal Clinical Network forum and through targeted education to quality improvement staff.

Following the literature review and analysis of the coding definitions, the perineal tears VLAD indicator was refined and reinstated,12 induction of labour and caesarean section indicators were refined, and episiotomy and instrumental delivery VLAD indicators were developed and introduced, completing the “act” part of the quality improvement cycle. Within 3 months of reintroducing the perineal tear indicator, a VLAD review raised a clinical practice issue identifying the use of midline episiotomies causing third-degree tears.

Discussion

Interestingly, issues with the indicator definitions only became apparent when the VLAD methodology was introduced, perhaps confirming a more comprehensive use of the monthly disseminated VLAD monitoring tool, compared with the annual outcome rates. In this sense, a lack of feedback regarding a given indicator should not be interpreted as the sign of an effective indicator. A lack of feedback may be a result of the statistical methodology used or the frequency or timeliness of dissemination.

Use of the original laparoscopic cholecystectomy definition had identified quality improvement opportunities and demonstrated the importance of monitoring significant surgical complications. Despite this, it was agreed that continued use of the indicator could lead to unnecessary reviews, wasting reviewers’ and coders’ valuable time and damaging the perception of the monitoring program’s credibility. Continued use of the indicator could have created a disincentive for detailed documentation in the operative notes. Ultimately, ensuring stakeholders remain engaged by having a definition and monitoring tool that is clinically reliable and statistically sensitive in detecting safety and quality issues far outweighs the short period during which the indicator is suspended.

Using working groups to review and refine indicator definitions not only results in improving the clinical relevance of indicators, but has the additional benefit of increasing the working group participants’ knowledge in the VLAD methodology and indicator definition constructs, thereby creating VLAD experts in various hospitals who can assist others within their hospital to increase knowledge and help with VLAD reviews.

Above all, the most important message is that quality improvement indicators should not to be used as performance measures.13-15 Indicators (eg, VLADs, annual rates) are tools that help identify variation. Review of the variation is required to determine its cause. Indicator variation (without the benefit of review) cannot be equated with either high or low hospital performance. The notion that hospitals with higher risk-adjusted mortality have poorer-quality care is neither consistent nor reliable.16

The quality improvement cycle is the mantra that patient safety and quality programs promote to stakeholders. As patient safety and quality program leaders, expert in employing the quality improvement cycle, it is imperative that we live it, not just preach it. Quality improvement indicators are not beyond improvement and should also be regularly reviewed for validity and utility, with improvements developed and implemented using the same plan, do, check and act cycle expected of users of the indicators.

Laparoscopic cholecystectomy complications of surgery variable life-adjusted display (VLAD) for ABC Hospital, July 2006 – April 2010

The VLAD plot (black trendline) remains fairly flat from cases zero to 57, indicating that ABC Hospital has a similar complication of surgery rate to the state. From cases 58 to 102, it trends downward, indicating a higher number of complications of surgery than the state for this period.


* Control limits are level 1 limits (50% lower or higher than expected).

  • Kirstine M Sketcher-Baker1
  • Maarten C Kamp2,3
  • Julia A Connors1
  • Don J Martin1
  • Justin E Collins1

  • 1 Centre for Healthcare Improvement, Queensland Health, Brisbane, QLD.
  • 2 School of Medicine, University of Queensland, Brisbane, QLD.
  • 3 Metro North Health Service, Queensland Health, Brisbane, QLD.


Acknowledgements: 

We thank Queensland clinicians and safety and quality staff who participate in conducting VLAD reviews to ultimately improve the delivery of patient care to all Queenslanders.

Competing interests:

None identified.

  • 1. Sherlaw-Johnson C. A method for detecting runs of good and bad clinical outcomes on variable life-adjusted display (VLAD) charts. Health Care Manag Sci 2005; 8: 61-65.
  • 2. Duckett SJ, Coory M, Sketcher-Baker K. Identifying variations in quality of care in Queensland hospitals. Med J Aust 2007; 187: 571-575. <MJA full text>
  • 3. Woodall W. The use of control charts in health-care and public-health surveillance. J Qual Technol 2006; 38: 89-104.
  • 4. Scott IA, Thomson PL, Narasimhan S. Comparing risk-prediction methods using administrative or clinical data in assessing excess in-hospital mortality in patients with acute myocardial infarction. Med J Aust 2008; 188: 332-336. <MJA full text>
  • 5. Shewhart W. Economic control of quality of manufactured product. New York: Van Nostrand, 1931. (Reprinted Milwaukee, Wis: American Society for Quality Control, 1980.)
  • 6. Coory M, Duckett S, Sketcher-Baker K. Using control charts to monitor quality of hospital care with administrative data. Int J Qual Health Care 2008; 20: 31-39.
  • 7. Mohammed MA, Stevens AJ. Performance monitoring in Australia and England: from scandals to action [editorial]. Med J Aust 2007; 187: 549-550. <MJA full text>
  • 8. Mohammed MA, Rathbone A, Myers P, et al. An investigation into general practioners associated with high patient mortality flagged up through the Shipman inquiry: retrospective analysis of routine data. BMJ 2004; 328: 1474-1477.
  • 9. Queensland Health. Queensland Health Clinical Governance Implementation Standard. 2008; Jun. http://www.health.qld.gov.au/cpic/pdf/clingovimpstand4_v3.5.pdf (accessed Mar 2010).
  • 10. Queensland Health. Existing Indicator Feedback Form. 2009; Apr. http://www.health.qld.gov.au/quality/existing_indicator.asp (accessed Jul 2010).
  • 11. Women’s Hospitals Australasia. Supporting Excellence in Maternity Care: the Core Maternity Indicators Project. 2007; Jan. http://www.health.gov.au/internet/safety/publishing.nsf/Content/3A59DB5FECB57A99CA25753C001B50ED/$File/CMI-Report-Jan2007.pdf (accessed Mar 2010).
  • 12. Queensland Health. Statewide Maternity and Neonatal Network VLAD Working Group. Summary of activities to-date: November 2008 – July 2009. 2009; Sep. http://www.health.qld.gov.au/quality/docs/vlad_wrgrp_smry_0709.pdf (accessed Mar 2010).
  • 13. Lilford R, Mohammed MA, Spiegelhalter D, et al. Use and misuse of process and outcome data in managing performance of acute medical care: avoid institutional stigma. Lancet 2004; 363: 1147-1154.
  • 14. Mainz J. Developing evidence-based clinical indicators: a state of the art methods primer. Int J Qual Health Care 2003; 15 Suppl 1: i5-i11.
  • 15. Scott IA, Ward M. Public reporting of hospital outcomes based on administrative data: risks and opportunities. Med J Aust 2006; 184: 571-575. <MJA full text>
  • 16. Pitches DW, Mohammed MA, Lilford RJ. What is the empirical evidence that hospitals with higher-risk adjusted mortality rates provide poorer quality care? A systematic review of literature. BMC Health Serv Res 2007; 7: 91.

Author

remove_circle_outline Delete Author
add_circle_outline Add Author

Comment
Do you have any competing interests to declare? *

I/we agree to assign copyright to the Medical Journal of Australia and agree to the Conditions of publication *
I/we agree to the Terms of use of the Medical Journal of Australia *
Email me when people comment on this article

Online responses are no longer available. Please refer to our instructions for authors page for more information.