Main outcome measures: Frequency of episodes of death in LM-DRGs (defined as DRGs with mortality < 0.5% over the previous 3 years or < 0.5% in any of the previous 3 years); associations between characteristics of patients and hospitals with deaths in LM-DRGs.
Results: There were 1 008 816 LM-DRG episodes with 0–15 LM-DRG deaths per hospital in the 2006–07 financial year and 0–20 deaths per hospital in the 2007–08 financial year. Increased age, level of comorbidity, being male, admission from a residential aged care facility, interhospital transfer, emergency admission and lower hospital volume were associated with an increased risk of death in LM-DRG episodes in both years. Metropolitan location and teaching/major provider status were not associated with LM-DRG deaths (P > 0.10). More than 40% of LM-DRG deaths were among patients aged 83 years or over, who had a length of stay of less than 1 day and had a medical DRG classification. Standardised mortality ratios (SMRs) that adjusted for the patient and hospital characteristics identified nine outlier hospitals with high frequencies of deaths in LM-DRGs in the 2006–07 and six in the 2007–08 financial year compared with 59 hospitals flagged by the death-in-LM-DRG indicator.
Conclusions: The use of the LM-DRG indicator requires further investigation to test its validity. LM-DRG deaths are infrequent, making it difficult to identify temporal changes and outlier hospitals. Patient characteristics unrelated to quality of care increase the likelihood of death among LM-DRG patients. The SMR analysis showed that failure to adjust for these characteristics may result in unfair and inaccurate identification of outlier hospitals. The increased risk of death associated with interhospital transfer patients and low-volume hospitals requires further investigation.
Monitoring and reporting of patient safety indicators (PSIs) at a national level has been identified as one of the major components of health care reform in Australia.1 Australia is following the trend of other Organisation for Economic Co-operation and Development (OECD) countries, such as the United States, which first introduced national PSIs in 20032 under the premise that collection and analysis of patient safety incident data would facilitate learning and the development of solutions that result in improved quality of care. “Death in low-mortality diagnosis-related groups” (LM-DRGs) is a quality and safety indicator used in health care services worldwide. The indicator identifies inhospital deaths that occur among patients assigned to a DRG with a low associated risk of death, and that are therefore more likely to be attributable to a safety or quality issue.3 In 2009, the indicator was included in the 25 national safety and quality in health care indicators proposed for Australian hospitals, and was recommended to be publicly reported at a national and individual hospital level.4 Since these recommendations were made, a comprehensive review has concluded that “higher quality, prospective, analytic studies are required before death in LM-DRG is used as an indicator of quality and safety in health care”.5
PSIs are designed to provide information about the relative quality-of-care performance of health care services. A recent article describes a framework to drive quality-indicator development and evaluation.6 The authors of this article state that, for an indicator to be scientifically acceptable, it must have several attributes. Important ones are:
validity (it measures the intended aspect of quality, and accurately represents the concept being evaluated);
sensitivity (sufficient variation can be explained by provider performance after patients’ characteristics are taken into account); and
event frequency (the sample size is large enough to detect actual differences, and risk adjustment is adequate to address confounding bias).
Current use of the death-in-LM-DRG indicator does not include any risk adjustment. This assumes the indicator is unaffected by patient or hospital factors unrelated to quality of care. Given the heterogeneity of hospital service provision across Australia, it is likely is that these factors could be associated with LM-DRG deaths, and that failure to adjust for their influence in analysing rates may lead to inaccurate identification of hospital outliers. Meaningful hospital performance monitoring using PSIs also requires a sufficiently large number of events to be able to differentiate random noise from special-cause variation. A genuine rate increase (special-cause variation) should be identified promptly so that its cause can be investigated and eliminated.
In Australia, there is no published data on the frequency of deaths flagged by the death-in-LM-DRG indicator or on patient and hospital characteristics that influence their occurrence. In this study, we sought to identify the incidence of LM-DRG deaths, the impact of patient and hospital characteristics on their occurrence and the demographic, admission and clinical profile of patients in LM-DRGs who died.
We undertook a retrospective cohort study of discharge episodes from 122 public hospitals in Victoria. The analysis sample included all 2 400 089 discharge episodes for people aged 18 years or older recorded in the Victorian Admitted Episodes Dataset (VAED) from 1 July 2006 to 30 June 2008. The VAED is the administrative data set for Victorian hospitals that includes demographic, administrative and clinical information. Since 1998, clinical information in the VAED has been coded using the International statistical classification of diseases and related health problems, 10th revision, Australian modification (ICD-10-AM). Good to excellent coding quality of ICD-10-AM has been demonstrated for coding of diagnoses.7
The primary outcome was episodes of death in LM-DRGs. The Agency for Healthcare Research and Quality (AHRQ) death-in-LM-DRG indicator,3 translated by the Victorian Department of Health (International classification of diseases, injuries and causes of death. 9th revision [ICD-9] to ICD-10-AM),8 was applied to the VAED. LM-DRGs are defined as DRGs with a total mortality rate of < 0.5% over the previous 3 years, or < 0.5% in any of the previous 3 years.2 As specified by the AHRQ, episodes with any code for trauma, immunocompromised state or cancer were excluded from the analysis as patients with these conditions have higher non-preventable mortality. Episodes with a care type indicative of a posthumous organ donor, hospital boarder (receives food and/or accommodation, but for whom the hospital does not accept responsibility for treatment and/or care)9 or unqualified neonate (neonates who are 9 or fewer days old and do not meet the criteria for admission)9 were also excluded.8
The patient characteristics of age, sex, admitted from a residential aged care facility, interhospital transfer, emergency admission and level of comorbidity (Elixhauser score10) were generated from VAED variables. Age was coded in 5-year age groups. Elixhauser scores were generated with published algorithms for ICD-10-coded diagnoses.11 The VAED includes up to 40 ICD-10 diagnoses for each patient episode, which are accompanied by a condition onset code.12 These are: P for primary diagnosis, A for associated conditions present on admission, C for complications occurring during the admission, and M for morphology. Diagnoses coded P and A were used to compute Elixhauser comorbidity scores. Thus, Elixhauser scores were generated from diagnoses that were recorded as being present when the patient presented to hospital rather than any diagnoses that were hospital acquired. The Elixhauser Comorbidity Index has been reported to have greater associations with inhospital death than other indexes such as the Charlson method of comorbidity measurement.10 A hospital volume variable was generated based on tertiles of all-episode admission volumes. Each episode was coded as occurring in a “low”, “medium” or “high” volume hospital. Two additional hospital variables were computed for each episode — metropolitan location and major provider/teaching hospital status.
Conventional predictive modelling involves development of a model in one dataset and validation in a second. However, as LM-DRGs were not consistent over the 2 financial years, analysis was completed on each year separately rather than combining the two cohorts. Analysis of each financial-year cohort separately provides information about the extent to which our findings can be generalised through examination of the consistency of associations identified over the 2 years.
LM-DRG deaths were identified for each hospital. Associations of LM-DRG deaths with patient and hospital characteristics were examined using hierarchical multivariable logistic regression models, in which analysis was clustered by hospital to account for possible correlation of patient episodes within hospital groupings. Clustering can occur at several levels. At the patient level, patients may be admitted multiple times during the observation period, especially older patients and those with chronic disease. Multiple admissions of individuals create data clusters — observations are not independent. Clustering can also occur at the unit level where patients have similar characteristics (eg, a neurological diagnosis), which, again, creates dependence between observations. The statistical techniques applied in this study incorporate adjustment for clustering to avoid incorrect associations being identified.
All variables except age and level of comorbidity were entered as dichotomous or ordinal variables. Assumptions of linearity were tested for age and comorbidity variables. A two-stage analysis was then undertaken. First, univariable associations between characteristic variables and the outcome variable were calculated by logistic regression. Variables found to have significant associations in these analyses were then entered into a multivariable logistic regression model. Colinearity checks of variables entered into the multivariable model were undertaken to ensure variables were not significantly correlated and would therefore compromise the stability of the model. A P value of ≤ 0.05 was considered statistically significant, and confidence intervals were calculated at the 95% level. Associations were reported as odds ratios. Descriptive statistics were used to compile a profile of LM-DRG deaths.
The logistic regression models were then used to generate an expected probability of death for each LM-DRG episode that included adjustment for risk factors found to have a significant association with LM-DRG deaths. The expected probabilities were then summed to compute the LM-DRG standardised mortality ratio (SMR) for each hospital, where the SMR is equal to the sum of observed deaths divided by the sum of the expected deaths. Exact 95% CIs for the SMR were then computed. Standard interpretations of the SMR were applied whereby an SMR 95% CI below unity is taken to represent lower than expected mortality, and a CI above unity a higher than expected or excess mortality.
There were 1 008 816 LM-DRG episodes over the 2 years. Box 1 outlines the patient discharge episodes included in the analysis. Box 2 presents the demographic characteristics of each financial-year cohort. More than 40% of patients with LM-DRGs who died were aged 83 years or older, and 39% or more had a length of stay of 1 day or less. In both cohorts, many LM-DRG deaths occurred in patients whose admissions were unplanned and classified as emergency admissions (> 74%) and as medical DRGs. Almost 20% of LM-DRG deaths in the 2006–07 financial year occurred in patients transferred from other hospitals. This proportion was lower (less than 12%) in the 2007–08 financial year. The DRG, primary diagnoses, procedures and complications recorded for LM-DRG deaths in the VAED were highly variable with no single DRG, diagnosis, procedure or complication being reported in more than 10% of cases.
LM-DRG deaths were infrequent, ranging from zero to 15 deaths per hospital in the 2006–07 financial year and zero to 20 in the 2007–08 financial year. There were 225 deaths among the 460 836 LM-DRG episodes in 2006–07 and 217 deaths among the 547 980 LM-DRG episodes in 2007–08. Sixty-three hospitals (51.64%) in 2006–07 and 62 hospitals (51.24%) in 2007–08 had no LM-DRG deaths.
Box 3 shows the associations between patient and hospital characteristics and death in LM-DRG episodes. Increased age and level of comorbidity, being male, admission from a residential aged care facility, interhospital transfer and emergency admission were independently associated with an increased risk of death in LM-DRG episodes (odds ratio 95% CI above 1.0; P < 0.05). Lower hospital volume was also found to be associated with an increased risk of death in LM-DRG episodes, compared with higher volume hospitals (P < 0.05). Hospital metropolitan location and major provider/teaching hospital status had no significant association with risk of death in LM-DRG episodes (P > 0.10), and were therefore not entered into the multivariable model. There was a high level of consistency in the characteristics identified in each cohort and the levels of association with risk of death in LM-DRG episodes, suggesting that our findings can be generalised across yearly cohorts.
Box 4 shows the SMR results. The SMR of each hospital is plotted with bars representing the 95% CI of the SMR. These show that when the rates of LM-DRG deaths are adjusted for the significant patient and hospital characteristics described above, the number of hospitals flagged as high outliers (those with SMR 95% confidence intervals above 1 — nine hospitals in 2006–7 and six in 2007–08) is far fewer than the 59 identified by the death-in-LM-DRG indicator.
Although the death-in-LM-DRG indicator has good face validity and is easy to generate from hospital administrative datasets, it is premature to infer differences in the safety of hospital patient care based on information yielded by this indicator. Our study shows that there are several patient-episode characteristics unrelated to quality of care that influence the likelihood of death in LM-DRG episodes. One main methodological challenge — the low frequency of LM-DRG deaths — suggests that this indicator is likely to be insensitive for true variations in quality of care in the hospital setting. The findings of this study highlight that the death-in-LM-DRG indicator requires further refinement before it can be employed broadly as a quality and safety metric.
An important consideration in the use of quality and safety indicators like death in LM-DRG is that their benefit must outweigh their burden. This study raises questions about the benefit of this indicator. Less than 50% of adult patient episodes are included in its generation (Box 1). Consequently, the indicator provides no information about the quality of hospital care for more than 50% of patient episodes. Patient age, sex, level of comorbidity and admission source and type were significantly associated with the risk of death in LM-DRG episodes in our sample. This indicates that several factors unrelated to the quality of care provided by a hospital may be the source of variations in this indicator.
The finding that more than 40% of LM-DRG deaths occurred in patients aged 83 years and over, and that these patients were also more likely to be medical patients admitted from residential aged care facilities and to die shortly after being admitted, suggests that there may be many false positives with this indicator. The indicator may be biased towards detecting deaths in older hospitalised patients for whom death may be an expected outcome. The finding that LM-DRG patients who were transferred from a residential aged care facility were more likely to die requires further investigation, especially given that this was independent of age and level of comorbidity. Our findings align with a study evaluating the impact of living in a nursing home on mortality within two hospital internal medicine services. The study found inhospital mortality was significantly associated with living in a nursing home, independent of age, sex, condition, level of comorbidity and hospital of admission.13 As highlighted above, death may be an expected outcome for many of these patients, and not an outcome of poor quality of care.
Another interesting finding of our study was that LM-DRG patients who were transferred from another hospital were three to six times more likely to die than those admitted from other sources. There are several reasons for patients to be transferred from one hospital to another. For example, a patient may require procedures or services not available at the hospital that they were first admitted to, such as dialysis or intensive care, or the transfer may be a natural care transition from acute to subacute care for rehabilitation purposes. Consequently, explaining the finding of increased likelihood of death in LM-DRG patients transferred from another hospital requires more detailed investigation of cases and interhospital transfer practices. Use of 30-day mortality, as opposed to inhospital mortality, should be investigated as this may show different associations.
We also explored the application of a risk-adjusted death-in-LM-DRG indicator, and found a very different picture of outlier hospitals than found when considering unadjusted LM-DRG deaths. Risk-adjusted death-in-LM-DRG results flagged substantially fewer hospitals as potential high outliers than the unadjusted death-in-LM-DRG indicator. This analysis shows that failure to adjust for patient and hospital characteristics that are unrelated to quality of care may result in unfair and inaccurate identification of outlier hospitals using the death-in-LM-DRG indicator. However, Box 4 shows that many hospitals have very wide SMR confidence intervals (hence the presentation on a log scale) suggesting that even the risk-adjusted death-in-LM-DRG indicator results are imprecise. The imprecision may be partially explained by the low event rates creating the situation of low “signal-to-noise” ratios, and smaller hospitals displaying greater variability in mortality performance indicators.14,15
We found that admission to a low-volume hospital (< 1000 episodes per year) significantly increased the risk of death in LM-DRG episodes. This risk was independent of age, sex, level of comorbidity and admission from residential aged care. Hospital type, classified as major provider/teaching hospital, and metropolitan location were found to have little association with death in LM-DRG episodes. These findings are consistent with a US study that also found teaching and rural/urban status were not associated with death-in-LM-DRG episodes.16 The finding of increased risk of death in low-volume hospitals is novel in the LM-DRG literature, but consistent with studies in several other populations. One systematic review, of studies across 33 diagnoses and interventions, investigated associations between hospital volume and risk-adjusted mortality.17 This review concluded that higher hospital volume is associated with higher survival. The relationship we found between volume and outcome is potentially important, but further investigation is required as differences in operational aspects of hospitals and casemix that we did not measure may have confounded the results.
Our findings, in conjunction with those of a growing number of other studies, raise questions about the validity of the death-in-LM-DRG indicator as a quality-of-care metric. A recent review provides a comprehensive synthesis of findings of 12 previous studies evaluating the indicator;5 only three of these studies provided some evidence that there were greater quality-of-care deficiencies in death-in-LM-DRG cases than other cases. A large-scale US study in 4504 hospitals found the indicator had a weak and sometimes inverse relationship with other measures of hospital quality, such as risk-adjusted mortality, Hospital Quality Alliance scores and US News and World Reports ratings systems.18 The indicator has also been found to have no association with hospital accreditation scores.19 Another US study reported on a medical record audit of 110 paediatric death-in-LM-DRG cases occurring in 14 hospitals.20 This study found that the indicator was not suitable for use in paediatric populations as 71.8% of the deaths were categorised as unpreventable and 13.6% as unable to be determined.20 A United Kingdom study found no association between hospital SMRs and the number of deaths in low-mortality health care resource groups.21
In consideration of the burden of the use of the death-in-LM-DRG indicator, we identify several issues. In Victoria, this indicator has been used as a screening tool to flag deaths that should undergo medical-record review to determine if there were deficiencies in the quality of care. Medical-record review by hospitals is a resource-intensive process. It has been recommended that to achieve higher levels of reliability in determining quality-of-care issues, the review should be undertaken by one registered nurse and at least two physicians.22 Our findings suggest that further validation of the indicator should be undertaken before valuable and often limited hospital health care staff resources are directed to the medical-record review of LM-DRG deaths.
Past literature suggests that differences in documentation, length of stay (time at risk) and coding practice (resulting from how ICD codes are used to set hospital payments) may cause variability in indicator events derived from administrative datasets.23,24 Also to be considered is that the death-in-LM-DRG indicator may have limited ability to detect quality-of-care deficiencies because of a low signal-to-noise ratio.
There are several limitations of our study that should be considered. First, analysis was based on episode-level and not patient-level data, so adjustment for clustering of observations using the hierarchical regression models may not have been optimal. However, these models are commonly applied in situations such as this where analysis is undertaken on de-identified administrative data. Second, there may be deficiencies in the quality of the underlying data source. There may have been differences in coding practices between hospitals.23 Finally, other factors, such as disease severity, ethnic disparities and socio-demographic characteristics, may also affect the likelihood of death-in-LM-DRG episodes, but we did not measure them.
The focus of our investigation was simply to establish characteristics of patient discharge episodes and of hospitals that influence the likelihood of death in LM-DRG episodes. To date, there is no proven link between deaths flagged by this indicator and the existence of quality-of-care problems in the Australian setting. In addition, it has not been proven, using either unadjusted or risk-adjusted methods, that outlier hospitals (those with higher rates of deaths in LM-DRGs) have higher proportions of cases with quality-of-care problems than non-outlier hospitals.
Further analyses that focus on determining the criterion validity of the indicator should be undertaken. These studies should employ medical-record review of LM-DRG and non-LM-DRG deaths to ascertain if deficiencies in the quality of care are more frequent in LM-DRG deaths. Risk adjustment models should be considered to prevent hospitals being wrongly being identified as outliers. In addition, the indicator should be evaluated using 30-day mortality to assess whether this provides a different profile and is a more precise indicator of quality of care. However, an important caveat in considering the further development of the death-in-LM-DRG indicator is that these events are rare. This suggests the indicator has limited utility as a quality-of-care metric. It is important to note that rare or sentinel events should not be ignored at a hospital level — local hospital quality systems should be developed to identify and investigate these events. It is using indicators of these events to measure and benchmark quality of care that is problematic. Alternative indicators that: (i) include a larger proportion of patient episodes; (ii) for which there are sufficient data to enable meaningful measurement and monitoring; and (iii) most importantly, that have demonstrated validity for measuring quality of care, should ultimately be the focus of future research.
1 Flow diagram of episodes of patient discharge from 122 hospitals in Victoria that were included in the analysis by year, for the financial years 2006–07 and 2007–08
2 Characteristics of episodes of patient discharge from 122 hospitals in Victoria by year, for the financial years 2006–07 and 2007–08
3 Odds ratios (and 95% CIs) for estimated associations between risk of death in low-mortality diagnosis-related groups (LM-DRGs) and selected patient and hospital characteristics in122 hospitals in Victoria for the financial years 2006–07 and 2007–08
4 Hospital standardised mortality ratio results by year
Provenance: Not commissioned; externally peer reviewed.
- 1. National Health and Hospitals Reform Commission. A healthier future for all Australians: final report June 2009. Canberra: Commonwealth of Australia, 2009. http://www.yourhealth.gov.au/internet/yourhealth/publishing. nsf/Content/nhhrc-report-toc (accessed Jun 2011).
- 2. Agency for Healthcare Research and Quality. Advancing patient safety: a decade of evidence, design and implementation. Rockville: US Department of Health and Human Services, 2009.
- 3. Agency for Healthcare Research and Quality. AHRQ quality indicators. Guide to patient safety indicators. Version 3.1. Rockville: Department of Health and Human Services, 2007. http://hcupnet.ahrq.gov/psi_guide_v31.pdf (accessed Jun 2011).
- 4. Australian Institute of Health and Welfare. Towards national indicators of safety and quality in health care. Canberra, AIHW, 2009. (AIHW Cat. No. HSE 75.) http://www.aihw.gov.au/publication-detail/?id=6442468285 (accessed Jun 2011).
- 5. Mihrshahi S, Brand C, Ibrahim JE, et al. Validity of the indicator “death in low-mortality diagnosis-related groups” for measuring patient safety and healthcare quality in hospitals. Intern Med J 2010; 40: 250-257.
- 6. Evans S, Lowinger J, Sprivulis P, et al. Prioritising quality indicator development across the healthcare system: identifying what to measure. Intern Med J 2008; 39: 648-654. doi: 10.1111/j.1445-5994.2008.01733.x.
- 7. Henderson T, Shepheard J, Sundararajan V. Quality of diagnosis and procedure coding in ICD-10 administrative data. Med Care 2006; 44: 1011-1019.
- 8. State Government of Victoria, Australia Department of Health. Victorian Government health information. Patient safety indicators. AusPSI. http://www.health.vic.gov.au/psi/auspsi (accessed Jun 2011).
- 9. Australian Institute of Health and Welfare. Hospital service—care type. http://meteor.aihw. gov.au/content/index.phtml/itemId/270174 (accessed Mar 2011).
- 10. Southern DA, Quan H, Ghali WA. Comparison of the Elixhauser and Charlson/Deyo methods of comorbidity measurement in administrative data. Med Care 2004; 42: 355-360.
- 11. Quan H, Sundararajan V, Halfon P, et al. Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care 2005; 43: 1130-1139.
- 12. State Government of Victoria, Australia Department of Health. Victorian Government health information. Archived. VAED 16th edition user manual — July 2006. http://www.health. vic.gov.au/hdss/archive/vaed/2006/manual/index.htm (accessed Jun 2011).
- 13. Barba R, Losa JE, Canora J, et al. The influence of nursing homes in the functioning of internal medicine services. Eur J Intern Med 2009; 20: 85-88.
- 14. Tu YK, Gilthorpe MS. The most dangerous hospital or the most dangerous equation? BMC Health Serv Res 2007; 7: 185.
- 15. Scott IA, Brand CA, Phelps GE, et al. Using hospital standardised mortality ratios in assessing quality of care — proceed with extreme caution. Med J Aust 2011; 194: 645-648. <MJA full text>
- 16. Thornlow DK, Stukenborg GJ. The association between hospital characteristics and rates of preventable complications and adverse events. Med Care 2006; 44: 265-269.
- 17. Gandjour A, Bannenberg A, Lauterbach KW. Threshold volumes associated with higher survival in health care: a systematic review. Med Care 2003; 41: 1129-1141.
- 18. Isaac T, Jha AK. Are patient safety indicators related to widely used measures of hospital quality? J Gen Intern Med 2008; 23: 1373-1378.
- 19. Miller MR PP, Donithan M, Zeger S, et al. Relationship between performance measurement and accreditation: implications for quality of care and patient safety. Am J Med Qual 2005; 20: 239-252.
- 20. Scanlon MC, Miller M, Harris JM II, et al. Targeted chart review of pediatric patient safety events identified by the Agency for Healthcare Research and Quality’s Patient Safety Indicators methodology. J Patient Saf 2006; 2: 191-197.
- 21. Hutchinson A, Young TA, Cooper KL, et al. Trends in healthcare incident reporting and relationship to safety and quality data in acute hospitals: results from the National Reporting and Learning System. Qual Saf Health Care 2009; 18(1): 5-10.
- 22. Lilford R, Edwards A, Girling A, et al. Inter-rater reliability of case-note audit: a systematic review. J Health Serv Res Policy 2007; 12: 173-180.
- 23. Scott IA, Ward M. Public reporting of hospital outcomes based on administrative data: risks and opportunities. Med J Aust 2006; 184: 571-575. <MJA full text>
- 24. Drosler SE, Klazinga NS, Romano PS, et al. Application of patient safety indicators internationally: a pilot study among seven countries. Int J Qual Health Care 2009; 21: 272-278.
Publication of your online response is subject to the Medical Journal of Australia's editorial discretion. You will be notified by email within five working days should your response be accepted.