Using the quality improvement cycle on clinical indicators — improve or remove?

Prior to 2007, Queensland Health clinicians selected a suite of clinical indicators from a range of sources and refined these to ensure their relevance to the Queensland context for inclusion in hospital-specific annual reports. Annual reports were initially disseminated to hospital executives only, presenting cross-sectional analysis of 12-month risk-adjusted outcome rates and highlighting the statistical significance of variation between hospital and state rates. Although the intention of these reports was to prompt local review to determine the cause of variation, it was recognised that the reports were not used as intended because the data were at least 1 year old at publication and the methodology summarised results over 1 year, hiding trends.

Since 2007, Queensland Health has employed a statistical process control methodology that has been adapted for clinical utility, the variable life-adjusted display (VLAD), to systematically monitor patient outcomes (Box).1-3 VLAD methodology based on available administrative data has been found to provide results equivalent to more resource-intensive clinical audit methodology that uses individual chart review.4 Queensland Health hospitals have undertaken numerous quality initiatives as a result of the use of VLADs, leading to improvements in discharge processes, clinician documentation, patient education, implementation of and adherence to standard clinical pathways, and allocation of resources to areas of most need.

One of the key elements that led to these initiatives is having well defined indicators that are clinically relevant, clinically significant in terms of burden of disease, and clinically responsive to changes in practice. Queensland Health has used the quality improvement cycle (plan, do, check and act cycle)5 to improve or remove indicators, suspending two indicators, developing two new indicators and refining three indicators. A review of the balance of indicators is currently underway. This process of development and refinement has led to a closer working relationship with clinicians, and resulted in improved indicator definitions to more effectively monitor safety and quality.

To overcome issues of timeliness and sensitivity, the VLAD methodology was applied to the 31 existing clinical indicator definitions and disseminated monthly to Queensland public and private hospitals. VLADs graphically display patient outcomes, “flagging” if predetermined levels of variation in patient outcomes are exceeded. Flags require a VLAD review to examine data and clinical practice to detect potential patient safety and quality issues and identify action to resolve issues.6-8 Formal documentation of the VLAD review is required within 30 days of notification and is evaluated within a governance structure that was introduced concurrently with the VLADs.9 Indicator definition issues identified by hospital staff are also captured in the VLAD review documentation. This feedback, and issues captured through an online feedback form,10 is then used to improve or remove clinical indicator definitions.

Two indicators for which the quality improvement cycle was recently applied within Queensland were laparoscopic cholecystectomy complications of surgery and third- and fourth-degree perineal tears during childbirth. The “plan” and “act” parts of the cycle had been in place for at least 12 months, with hospitals using the VLAD indicators to examine data and review practice. During this time, feedback from clinicians, quality coordinators and coders suggested these indicator definitions were problematic, triggering a review of the indicator definition — the “check” part of the quality improvement cycle.

Interestingly, issues with the indicator definitions only became apparent when the VLAD methodology was introduced, perhaps confirming a more comprehensive use of the monthly disseminated VLAD monitoring tool, compared with the annual outcome rates. In this sense, a lack of feedback regarding a given indicator should not be interpreted as the sign of an effective indicator. A lack of feedback may be a result of the statistical methodology used or the frequency or timeliness of dissemination.

Use of the original laparoscopic cholecystectomy definition had identified quality improvement opportunities and demonstrated the importance of monitoring significant surgical complications. Despite this, it was agreed that continued use of the indicator could lead to unnecessary reviews, wasting reviewers’ and coders’ valuable time and damaging the perception of the monitoring program’s credibility. Continued use of the indicator could have created a disincentive for detailed documentation in the operative notes. Ultimately, ensuring stakeholders remain engaged by having a definition and monitoring tool that is clinically reliable and statistically sensitive in detecting safety and quality issues far outweighs the short period during which the indicator is suspended.

Using working groups to review and refine indicator definitions not only results in improving the clinical relevance of indicators, but has the additional benefit of increasing the working group participants’ knowledge in the VLAD methodology and indicator definition constructs, thereby creating VLAD experts in various hospitals who can assist others within their hospital to increase knowledge and help with VLAD reviews.

Above all, the most important message is that quality improvement indicators should not to be used as performance measures.13-15 Indicators (eg, VLADs, annual rates) are tools that help identify variation. Review of the variation is required to determine its cause. Indicator variation (without the benefit of review) cannot be equated with either high or low hospital performance. The notion that hospitals with higher risk-adjusted mortality have poorer-quality care is neither consistent nor reliable.16

The quality improvement cycle is the mantra that patient safety and quality programs promote to stakeholders. As patient safety and quality program leaders, expert in employing the quality improvement cycle, it is imperative that we live it, not just preach it. Quality improvement indicators are not beyond improvement and should also be regularly reviewed for validity and utility, with improvements developed and implemented using the same plan, do, check and act cycle expected of users of the indicators.

Laparoscopic cholecystectomy complications of surgery variable life-adjusted display (VLAD) for ABC Hospital, July 2006 – April 2010

The VLAD plot (black trendline) remains fairly flat from cases zero to 57, indicating that ABC Hospital has a similar complication of surgery rate to the state. From cases 58 to 102, it trends downward, indicating a higher number of complications of surgery than the state for this period.

* Control limits are level 1 limits (50% lower or higher than expected).