Identifying variations in quality of care in Queensland hospitals

In Australia and elsewhere, hospital-specific comparisons based on routine data have relied on cross-sectional analysis.6,7 This involves aggregating data for all patients over a set period, say 12 months, and determining whether the number of adverse outcomes (eg, in-hospital deaths after admission for stroke) is higher than expected based on the average for all hospitals. By definition, these cross-sectional analyses can only occur at the end of some set period, and provide average results for all patients admitted to the hospital during that time. In contrast, statistical process control is a continuous approach, and displays data on outcomes of care of individual patients. The method can identify changes in outcomes relatively quickly and is more sensitive to such changes than less regular, cross-sectional approaches, which can obscure important patterns in the data.8 Statistical process control also highlights the dynamic nature of health care: that patterns of care can change over time and a negative signal at some point in the past can be rectified.

Statistical process control was developed several decades ago to improve the quality of manufactured products. Its application in health is complicated by the need to adjust for risk to ensure that hospitals or doctors who see sicker patients are not unfairly penalised. Several methods have been proposed that incorporate risk adjustment;9 the method adopted by Queensland Health follows Sherlaw-Johnson’s approach,10 and is known as variable life-adjusted display (VLAD) (Box 2). Display charts are provided to Queensland hospitals each month, with the first distribution of charts providing trend data from July 2003 to late 2006.

The first step in plotting the VLAD is to calculate, for each patient in the instant month, the expected risk (probability) of a particular outcome (death, complication, readmission, and long-stay are used in Queensland), adjusting for age, sex, and selected comorbidities specific to each indicator. This can be thought of as the average risk of an adverse outcome across all hospitals for patients with the same age, sex and comorbidity profile as the patient in question. It is estimated using a logistic regression model for the index month plus the previous 11 months of data. These data include patients as defined by the indicator admitted to Queensland hospitals (public and private) with an average of at least 20 separations a year for the relevant indicator.

Next, the expected risk is subtracted from the observed outcomes (coded as 0 or 1 for presence or absence of the outcome) and plotted sequentially. An upward movement of the chart indicates that, for the patients in question, the number of outcomes (eg, deaths) was less than that expected, while a downward movement indicates that the number of outcomes was greater than that expected.

Finally, thresholds are calculated where the chart is said to flag. Critical in quality improvement approaches is not just providing data, but ensuring that aberrant patterns identified by monitoring are investigated, and that practice changes occur.9 Queensland Health has developed hierarchical flagging criteria that signal closer scrutiny, depending on the extent of variation from the state average and whether the indicator incorporates a fatal or non-fatal outcome (Box 3).

For example, if the trend line shows that the cumulative experience of outcomes of care is more than 30% worse than the state average (for an indicator with a fatal outcome), the indicator is flagged for internal hospital review. The Queensland Health VLAD policy requires identification of “clinician leads” to facilitate clinician involvement in the review process.11 In addition to reporting through the various organisational structures of Queensland Health, public hospitals are required to report remedial action to the local consumer consultative group.

The statistical process control methods used in industry were developed to help identify special (also called assignable) cause variation,12 which is defined as variation that warrants further investigation. Standard methods of frequentist inference (P values and confidence intervals) are not suitable for identifying such variation. Instead, likelihood methods, which are not affected by the problem of multiple looks at the data,13 are used, and within this framework the characteristics of the VLAD are usually described in terms of the average run length to true or false alarm. We used standard methods based on simulations14 to identify average run lengths to true and false alarm.

The flagging criteria were set to balance the costs of investigating false alarms (where the change in outcomes is simply a statistical artefact) against the need to identify special or assignable cause variation, which might benefit from further investigation.15 As is the case in industry,16 this was a policy decision; the reasoning is similar to that used to decide on a balance between sensitivity and specificity for a screening test.17

The control limits are reset each time a trigger point is reached. For example, when monitoring using the Tier 1, non-fatal flag, if a case is flagged as hitting 50% deviation from the average, the control limits are reset and the hospital could be flagged a second time if there is a cumulative run of cases which is again 50% deviation from average (starting at the first trigger point).

As the VLAD approach is continuous, with monthly dissemination of data, there are likely to be more frequent investigations by hospitals and clinicians than with an annual, cross-sectional approach. A key characteristic of continuous improvement is “closing the loop”, ensuring appropriate investigation and actions run on.

There is an ambiguous relationship between outcome and process measures of quality of care,18 so a pyramid model, which recognises multiple explanations for variation in recorded outcomes, is recommended as a focus for the investigation process.19 The first investigation should be whether the data have been coded accurately. A second screen is whether there is casemix variation that has not been fully accounted for in the risk adjustment process (eg, Indigenous status is not incorporated in the risk adjustment model, but is often associated with worse clinical outcomes). Box 5 shows the stages in the pyramid model, and typical questions that should be asked as part of an investigation.

The flags are a way of standardising the process for deciding when the data are worth a closer look. A virtue of the VLAD approach is that it encourages visual inspection of data and, in many cases, a more detailed look at the data could be instigated without using flags; for example, if a downward slope appeared abruptly. The VLAD can take many possible forms, depending on the length and clustering of runs of good or poor performance. However, in terms of actions that should be taken, VLADs can be grouped into four basic patterns (Box 6).

As recommended in the pyramid model of investigation, the first round of investigations highlighted many data coding issues, and this was frustrating to hospitals and clinicians. An outcome of this review is likely to be improved data quality that will enhance the credibility of the clinical indicators.

Routine data are limited and cannot provide risk adjustment for the full range of factors known to affect outcomes,20 and this is recognised in the second stage of the pyramid model of investigation. That is, more detailed investigation at the local level might reveal that a run of poor outcomes at a particular hospital might be due to a run of sicker patients. This should not undermine the utility of statistical process control approaches: the aim is to identify causes of variation in outcomes, be they variations in data quality, casemix, or quality of care.

The standard format for reporting on the outcome of flags is still evolving. The current requirement is that the report highlights what reviews were conducted (coding audits, clinical review) and their findings, together with what management action has been undertaken. A more structural reporting format is being developed.