eMJA     The Medical Journal of Australia

Home | Issues | eMJA shop | My account | Classifieds | Contact | More... | Topics | Search   

Achieving better practice – The Clinical Support Systems Program

Achieving better in-hospital and after-hospital care of patients with acute cardiac disease

Ian A Scott, Charles P Denaro, Cameron J Bennett, Annabel C Hickey, Alison M Mudge, Judy L Flores, Daniela C J Sanders, Justine M Thiele, Beres Wenck, John W Bennett and Mark A Jones
MJA 2004; 180 (10 Suppl): S83-S88
Abstract
  • In patients hospitalised with acute coronary syndromes (ACS) and congestive heart failure (CHF), evidence suggests opportunities for improving in-hospital and after-hospital care, patient self-care, and hospital–community integration.

  • A multidisciplinary quality improvement program was designed and instigated in Brisbane in October 2000 involving 250 clinicians at three teaching hospitals, 1080 general practitioners (GPs) from five Divisions of General Practice, 1594 patients with ACS and 904 patients with CHF.

  • Quality improvement interventions were implemented over 17 months after a 6-month baseline period and included:

    • clinical decision support (clinical practice guidelines, reminders, checklists, clinical pathways);

    • educational interventions (seminars, academic detailing);

    • regular performance feedback;

    • patient self-management strategies; and

    • hospital–community integration (discharge referral summaries; community pharmacist liaison; patient prompts to attend GPs).

  • Using a before–after study design to assess program impact, significantly more program patients compared with historical controls received:

    • ACS: Angiotensin-converting enzyme (ACE) inhibitors and lipid-lowering agents at discharge, aspirin and β-blockers at 3 months after discharge, inpatient cardiac counselling, and referral to outpatient cardiac rehabilitation.

    • CHF: Assessment for reversible precipitants, use of prophylaxis for deep-venous thrombosis, β-blockers at discharge, ACE inhibitors at 6 months after discharge, imaging of left ventricular function, and optimal management of blood pressure levels.

  • Risk-adjusted mortality rates at 6 and 12 months decreased, respectively, from 9.8% to 7.4% (P = 0.06) and from 13.4% to 10.1% (P = 0.06) for patients with ACS and from 22.8% to 15.2% (P < 0.001) and from 32.8% to 22.4% (P = 0.005) for patients with CHF.

  • Quality improvement programs that feature multifaceted interventions across the continuum of care can change clinical culture, optimise care and improve clinical outcomes.

Acute cardiac disease in the form of acute coronary syndromes (ACS) and congestive heart failure (CHF) remains the single largest cause of death in Australia, accounting for 197 000 hospitalisations per year at a cost of $579 million.1

Achieving optimal patient outcomes requires delivery of multiple forms of care by many clinicians over a care continuum spanning from presentation to emergency department to hospital discharge to general practice. Because of this complexity, several system failures can compromise quality and safety of care:2,3

  • delayed or incomplete diagnostic evaluation and risk-factor assessment;

  • omission, or delayed delivery, of effective treatments;

  • insufficient counselling of patients and carers about their condition and its management;

  • imperfect transfer of patient care information between hospital clinicians and general practitioners;

  • lack of timely, credible performance data for alerting health professionals to evidence–practice gaps; and

  • non-existent or underdeveloped methods for remediating identified deficiencies in care.

Audits of the quality of in-hospital care of patients with ACS and CHF admitted to three Brisbane teaching hospitals in late 2000 – early 2001 revealed opportunities for care improvement.2,3 Furthermore, after hospital discharge, patients with ACS do not always receive secondary prevention measures such as lipid-lowering therapy4 or cardiac rehabilitation,5 or achieve optimal control of risk factors.6 Population-based studies in Australia also suggest there are problems in ambulatory care of patients with CHF.7

Addressing the problem

The Brisbane Cardiac Consortium Clinical Support Systems Program (BCC-CSSP) sought to develop and implement a systematic, collaborative approach to achieving better in-hospital and after-hospital care of patients admitted with ACS or CHF, and to evaluate the impact of this program.

Program participants

The Program involved three teaching hospitals (Royal Brisbane and Women’s [900 beds], Princess Alexandra [640 beds] and Queen Elizabeth II [165 beds]) and five Divisions of General Practice (Brisbane North, Brisbane Inner South, Brisbane Southside Central, Bayside, and Logan). The consortium undertook a 2-year quality improvement program coordinated by a multidisciplinary steering group of leading clinicians and supported by district health service managers, divisional boards, and senior executives of Queensland Health. Five working groups, each with hospital and general practice representation, undertook the following program tasks: decision support, performance measurement and feedback, clinical information systems, patient self-management (assisted by patient representation), and hospital–general practice integration.

Program patients

Consecutive patients admitted to any of the three hospitals between 1 October 2000 and 31 August 2002, and who met the case definitions (Box 1), entered the program. Patients registered between 1 October 2000 and 17 April 2001 comprised the routine care (or baseline) group; those registered on or after 18 April 2001 and whose care involved the quality improvement program were the intervention group. Quality of in-hospital care was assessed in all patients; quality of after-hospital care was assessed in a subset of patients who met eligibility criteria (Box 2) and provided written informed consent.

Program professionals

The targeted professionals were all consultant physicians, medical registrars and residents, nurses, clinical pharmacists and allied health professionals (n 450) working in cardiology, general medicine and emergency departments of participating hospitals; and all GPs (n 1080) who were members of the participating Divisions.

Program interventions

Quality improvement interventions (QIIs) selected for implementation after reviewing published literature8 and expert opinion included the following (resource materials and technical reports are available from our website <www.heath.qld.gov.au/bcc>).

Clinical decision support
  • Locally endorsed, evidence-based, clinical-practice guidelines disseminated to hospital clinicians and GPs as full-text guidelines, flow-chart pocket cards, laminated desktop guides, wall posters, and computer-based formats.

  • Guideline-based reminders such as management checklists attached to hospital observation charts and chart stickers invoking consideration of specific treatments in eligible patients.

  • Clinical pathways for chest pain assessment and management in the emergency department.

  • Discharge summaries and practice evaluation forms issued to GPs, which acted to reinforce key management recommendations.

Educational interventions
  • Periodic educational presentations (eg, “grand rounds”, seminars, and workshops), hospital newsletter articles (eg, hospital drug and therapeutics bulletins, and divisional newsletters), case-based unit and departmental meetings.

  • Academic detailing of hospital clinicians by trained clinical pharmacists, and of GPs, individually or in small groups (about 4–5), by trained detailers with no links to pharmaceutical companies.

Performance feedback
  • Feedback to target clinicians of quality indicators (see below) for in-hospital care every 6 months, and for after-hospital care at 8, 16 and 24 months after program commencement.

  • Feedback of personalised quality indicators to individual in-hospital consultant units.

Patient-directed disease management
  • Provision of patient self-management kits, comprising educational booklets (with treatment-specific risk-reduction estimates), patient diaries, individualised management plans, and self-explanatory medication lists.

  • Pharmacist counselling of individuals about lifestyle changes, drug therapy, and risk-factor modification.

  • After-discharge telephone follow-up by clinical pharmacists of high-risk patients (more than five classes of drugs, admission within previous 6 months, non-compliance, multiple comorbidities, age > 75 years, social isolation, risk of drug–drug interactions).

Hospital–community integration
  • Standardised discharge referral summaries with individualised treatment targets, advice on drug indications and monitoring, and follow-up arrangements.

  • Copies of medication lists forwarded to patients’ community pharmacists at discharge, with telephone call from hospital pharmacist in selected cases.

  • Telephone prompts to consenting patients to undertake follow-up visits to GPs.

Indicators for program performance feedback

Evidence-based, process-of-care indicators for in-hospital and after-hospital care were developed by consensus, as described elsewhere.2,3,9 In-hospital indicators were reported as the proportions of highly eligible patients (definite indication; no contraindication) who received specific care processes. Defining the indicator using eligibility criteria circumvented criticism from treating clinicians that indicators reported for their patients had not been adjusted for variation in casemix or risk. After-hospital care indicators were reported as overall rates, as detailed eligibility data could not be collected. In-hospital care data were abstracted retrospectively from patient records by trained nurses; after-hospital care data were obtained from GPs using standardised forms issued at 3, 6 and 12 months after discharge.

Measuring changes in program indicators

We assessed changes in the following indicators, measured on all patients, between routine care (1/10/00 – 17/4/01) and final intervention (15/2/02 – 30/8/02) cohorts:

  • proportions of inpatients receiving specific forms of care;

  • risk-adjusted, all-cause mortality at 30 days, 6 months and 12 months after discharge;

  • median length of index hospital stay; and

  • rates of same-cause readmission at 30 days.

Changes in after-hospital care were assessed by comparing indicators, measured at 3 and 6 months after discharge, between all evaluable routine care and intervention patients.

Proportions and medians were compared using χ2 and Mann–Whitney tests, respectively. Mortality rates were risk adjusted using logistic regression analysis models (C-statistic10 0.80 for ACS, and 0.75 for CHF).

Over a period of 23 months, 1594 patients with ACS and 904 patients with CHF met the case definitions and were registered. In-hospital data and all-cause mortality up to 12 months after discharge were collected for all patients. In all, 662 (42%) patients with ACS and 364 (40%) with CHF were eligible and consented to undergo follow-up to assess their after-hospital care. Evaluable data were obtained from 405 (61%) and 183 (50%) of these patients at 3 months, and from 344 (52%) and 151 (42%) at 6 months, respectively. Patient characteristics, risk factors and specialty of admitting clinician for both conditions were not significantly different between the routine care and the intervention groups (data not shown).

Program outcomes

A brief summary of program effects follows; more detailed results are available at <www.health.qld.gov.au/bcc>.

Process-of-care indicators

In-hospital care (Boxes 3, 4): Compared with the routine-care group, significant increases were seen in the number of intervention patients with ACS receiving the following: timely electrocardiography after emergency department presentation; angiotensin-converting enzyme (ACE) inhibitors and lipid-lowering agents at discharge; inpatient cardiac counselling; and referral to outpatient cardiac rehabilitation.

For patients with CHF, significant increases were seen in the numbers of intervention patients receiving the following: assessment for reversible, acute precipitants of cardiac decompensation; prophylaxis for deep venous thrombosis; requests for thyroid function tests (in those with atrial fibrillation); β-blocker prescription at discharge; and a recorded review of patient medications by clinical pharmacists. There was a trend towards increased use of imaging to assess left ventricular function.

After-hospital care: Relative to usual-care patients, more intervention patients with ACS were prescribed aspirin at 3 months (89% v 82%; P = 0.05), and, among those prescribed aspirin and β-blockers at discharge, more continued to receive these drugs at 3 months (92% v 84% [P = 0.03] and 85% v 76% [P = 0.05], respectively). For patients with CHF, more intervention than usual-care patients received β-blockers at 3 months (60% v 39%; P = 0.01) and at 6 months received ACE inhibitors (85% v 70%; P = 0.04); achieved ideal blood pressure levels (68% v 44%; P = 0.01); and were monitored for weight and salt/fluid intake (57% v 44%; P = 0.02).

Mortality

Risk-adjusted mortality rates for patients with ACS trended downward from 5.0% to 3.8% at 30 days from 9.8% to 7.4% (P = 0.06) at 6 months and from 13.4% to 10.1% (P = 0.06) at 12 months. More notable reductions were seen for patients with CHF — 8.2% to 5.3% (P = 0.04), 22.8% to 15.2% (P < 0.001) and 32.8% to 22.4% (P = 0.005), respectively.

Length of stay and readmissions

The median length of hospital stay for patients with ACS reduced by one day (from 7.0 to 6.0 days; P = 0.01), while that for patients with CHF remained unchanged (7.0 days). Same-cause readmission rates were not statistically different: 7.4% v 4.9% in ACS; 5.0% v 5.8% in CHF.

Qualitative analyses
Clinician feedback:

Focus-group discussions and surveys suggested that clinicians welcomed regular performance feedback and felt it had influenced their practice. Clinical guidelines and patient management plans were also regarded positively.

Patient feedback:

Program attributes which interviewed patients (n = 36) regarded very positively included provision of self-management materials and pharmacist counselling, support from relatives and GPs, empowerment in making care choices, and therapeutic relationship with clinicians based on good communication. Suggested further improvements included deferring education to a later time after hospital discharge, greater customisation of health information according to individual circumstances, and more psychosocial support. The BCC-CSSP heart failure patient information booklet was rated as one of the best four resources of its kind in Australia in a review recently conducted by the National Institute for Clinical Studies.11

Study limitations

Our results may be subject to bias, as the study was unblinded, used historical controls, and achieved follow-up of after-hospital care in only 50% of registered patients. However, the mortality end-points at 6 and 12 months were ascertained for all patients and were risk-adjusted; in-hospital data were obtained for the entire cohort; and the total intervention period of less than 18 months minimised the effects of secular trends.

Key lessons

Better care of patients with ACS or CHF can be achieved by implementing systems of decision support, targeted provider education and performance feedback, patient self-management, and hospital–community integration. The BCC-CSSP was unique in seeking to optimise patient care across two sectors of healthcare.

Which QIIs account for most of the improvements?

It is impossible to attribute specific process-of-care changes to specific QIIs within a multifaceted program. Systematic reviews reveal no singularly successful QII, and instead recommend deployment of multiple interventions.8

Why did some processes of care improve while others did not?

Processes of care directly controlled by individual clinicians (eg, drug prescribing) are readily amenable to behavioural change. Those more dependent on multilevel interactions within care systems (eg, emergency department triage or treatment responses, and timely access to complex modalities such as stress testing and cardiac rehabilitation) require interdisciplinary collaboration and/or work practice redesign, and hence change more slowly.

How can the uptake of similar programs be facilitated in other institutions?

Our experience suggests the following pre-requisites: identification of problematic areas of care from practice audits; multidisciplinary quality improvement teams; multifaceted change strategies; ongoing performance feedback; and support of both managerial and clinical leaders. Various inhibiting factors will need to be overcome using locally designed solutions (Box 5).

Are the methods used here transferable to other sites and conditions?

The methods of BCC-CSSP have been extended into multihospital collaborations throughout Queensland targeting cardiac care, renal medicine and acute stroke care under the auspices of the Collaboratives for Healthcare Improvement instigated by Queensland Health (details available at <http://www.health.qld.gov.au/chi/default.asp>).

Are programs such as this cost-effective and sustainable?

We are currently collecting data on long-term outcomes, readmissions and hospital costs in an effort to assess return on investment of our program,12 and such analyses will be the topics of future reports.

1: Case definitions

Acute coronary syndromes (ACS): Patients with a recorded acute clinical diagnosis of ACS at 48 hours after presentation and elevated serum cardiac markers: creatine kinase level more than twice, or troponin level more than 1.5 times, upper normal reference range.

Congestive heart failure (CHF): Patients with a recorded acute clinical diagnosis of CHF at 48 hours after presentation and two or more of the following diagnostic features: raised jugular venous pressure; third or fourth heart sounds; bilateral chest crackles to mid-zones; dependent oedema; cardiomegaly or pulmonary oedema on chest x-ray.

2: Patient eligibility criteria for interventions after discharge

Patients meeting the case definition whose usual GP practised in Greater Brisbane, and who had none of the following exclusion criteria: inability to participate in self-care due to mental or physical incapacity; resident of nursing home; terminal non-cardiac illness with life expectancy < 6 months.

3: Changes in process-of-care indicators for patients with acute coronary syndromes after quality improvement interventions (QIIs)

Baseline

After QII

(1/10/00 – 17/4/01)

(15/2/02 – 31/8/02)

Indicator

n = 428

n = 435

P


ECG within 10 minutes of hospital arrival

No. (%) of patients presenting directly to ED

145/238 (61%)

170/243 (70%)

0.05

Thrombolysis

No. (%) of highly eligible patients presenting directly to ED

49/49 (100%)

39/39 (100%)

na

Thrombolysis within 30 minutes of hospital arrival

No. (%) of patients receiving thrombolysis

17/49 (35%)

16/39 (41%)

0.59

Lipid status documented

No. (%) of all patients

311/428 (73%)

335/435 (77%)

0.12

β-Blockers prescribed at discharge*

No. (%) of highly eligible patients

212/251 (84%)

202/239 (85%)

0.97

No. (%) of all patients

284/351 (81%)

298/371 (81%)

0.90

Antiplatelet agents prescribed at discharge*

No. (%) of highly eligible patients

301/318 (95%)

321/334 (96%)

0.39

No. (%) of all patients

319/351 (91%)

344/371 (93%)

0.30

ACE inhibitors prescribed at discharge*

No. (%) of highly eligible patients

105/143 (73%)

113/139 (81%)

0.12

No. (%) of all patients

222/351 (63%)

261/371 (71%)

0.04

Lipid-lowering agents prescribed at discharge*

No. (%) of highly eligible patients

165/202 (82%)

197/223 (89%)

0.04

No. (%) of all patients

254/351 (72%)

293/371 (79%)

0.03

Early coronary angiography (during admission or within 30 days of discharge)

No. (%) of highly eligible patients

41/45 (91%)

43/46 (93%)

0.82

No. (%) of all patients

223/428 (52%)

236/435 (54%)

0.53

Non-invasive stress testing (during admission or within 30 days of discharge)

No. (%) of highly eligible patients

17/57 (30%)

17/55 (31%)

0.89

No. (%) of all patients

72/428 (17%)

86/435 (20%)

0.27

Cardiac counselling before discharge*

No. (%) of all patients

168/351 (48%)

212/371 (57%)

0.009

Referral to outpatient cardiac rehabilitation*

No. (%) of all patients

28/351 (8%)

64/371 (17%)

<0.001


*Denominator is number of patients discharged alive and not transferred. ECG = electrocardiogram. ED = emergency department. na = not applicable. ACE = angiotensin-converting enzyme. Highly eligible patients are those with definite indication and no contraindication to the stated intervention.

4: Changes in process-of-care indicators for patients with congestive heart failure after quality improvement interventions (QIIs)

Baseline

After QII

(1/10/00 – 17/4/01)

(15/2/02 – 31/8/02)

Indicator

n = 220

n = 235

P


Assessment of acute reversible triggers

No. (%) of all patients

166/220 (75%)

211/235 (90%)

<0.001

Prescribing of explicit fluid orders

No. (%) of all patients

89/220 (40%)

128/235 (54%)

0.002

Weighed daily for first 3 days of hospitalisation

No. (%) of all patients

121/220 (55%)

148/235 (63%)

0.59

DVT prophylaxis

No. (%) of highly eligible patients

31/104 (30%)

94/128 (73%)

<0.001

No. (%) of all patients

57/220 (26%)

148/235 (63%)

<0.001

Request for thyroid function tests

No. (%) of highly eligible patients

16/31 (52%)

41/52 (79%)

0.01

No. (%) of all patients

124/220 (56%)

165/235 (70%)

0.002

Imaging of left ventricular function

No. (%) of all patients

135/220 (61%)

164/235 (70%)

0.06

Scheduled clinic follow-up within 30 days of discharge*

No. (%) of all patients

87/191 (46%)

130/219 (59%)

0.005

Clinical pharmacist review before discharge*

No. (%) of all patients

105/191 (55%)

142/219 (65%)

0.04

ACE inhibitors prescribed at discharge*

No. (%) of highly eligible patients

58/71 (82%)

61/71 (86%)

0.68

No. (%) of all patients

136/191 (71%)

163/219 (74%)

0.46

β-Blocker prescribed at discharge*

No. (%) of highly eligible patients

47/135 (35%)

88/152 (58%)

<0.001

No. (%) of all patients

61/191 (32%)

113/219 (52%)

<0.001

Warfarin prescribed at discharge*

No. (%) of highly eligible patients

22/50 (44%)

27/63 (41%)

0.68

No. (%) of all patients

46/191 (24%)

41/219 (19%)

0.19

Avoidance of deleterious agents at discharge*

No. (%) of all patients

180/191 (94%)

214/219 (98%)

0.79


*Denominator is number of patients discharged alive and not transferred to other institutions. † Agents that are negatively inotropic, proarrhythmic or engender fluid retention (non-dihydropyridine calcium antagonists, non-steroidal anti-inflammatory agents, tricyclic antidepressants, class 1 antiarrhythmic agents). DVT = deep venous thrombosis. ACE = angiotensin-converting enzyme. Highly eligible patients are those with a definite indication and no contraindication to the stated intervention.

5: Program inhibitors and solutions

Potential inhibitor

Solution adopted


Failure to engage all key participants on equal terms

  • Multidisciplinary governance structure with central executive

  • Working groups with clearly defined functions and reporting lines

  • Representation of all key stakeholders including management and consumers

  • Formalised partnerships (memoranda of understanding) between hospitals and Divisions of General Practice

Failure to articulate a clear and agreed vision and operational plan

  • Concerted efforts at consensus building

  • Clearly defined program objectives and action plan

  • Choice of target conditions for which strong evidence base existed

Failure to formulate quality targets and measures for evaluating program effects over time

  • Development of quality standards and measures that were evidence based, expert endorsed, and agreed by all participants

  • Serial measurements at pre-specified intervals

Lack of timely, credible and interpretable performance data

  • Establishment of systems for reliably collecting standardised patient data across care continuum

  • Automated databases for generating pre-defined quality reports

  • Data quality verification procedures, including reabstraction audits

  • Minimisation of number of collected data items

  • Simple, graphical feedback formats disseminated via multiple media

Lack of program effects on clinical practice

  • Deployment of multifaceted quality improvement strategies

  • Local adaptation of nationally released guidelines

  • Recruitment of influential lead clinicians

  • Sustained focus on key indicators

Failure to forge collaboration between hospital units and between hospitals and general practice

  • Formation and nurturing of healthcare teams and cross-departmental linkages

  • Focus groups for promoting mutual appreciation of differing perspectives

Failure to gain support from senior office-holders in challenging current culture

  • Explicit, upfront organisational commitment at both senior managerial and health professional levels to respond to identified problems in care and accept need for change

Failure to gain and maintain interest and involvement of practising clinicians

  • Feedback of credible performance data, which raised awareness of opportunities for improvement

  • Illustration of quality improvement strategies that had proved effective in similar settings elsewhere

  • Ongoing iteration of the evidence base behind specific processes of care

Acknowledgements

We gratefully acknowledge the efforts of Melodie Downey, Lisa Mitchell, Alison Ruiz, Katie Foxcroft, Karen Watson, Gabby Jones and the late Jane Davey for performing data abstraction and tracking patients; Nguyen Thinh for constructing the database; Neil Stewart for providing data management; and Professor Jeanette Ward for advice on guideline implementation. We also wish to acknowledge the quality improvement efforts made by the many doctors, nurses, allied health professionals and patient representatives involved in program activities.

The Brisbane Cardiac Consortium Clinical Support Systems Program was an initiative of the Royal Australasian College of Physicians. It was funded by the Australian Government Department of Health and Aged Care, and supported by Queensland Health.

The members of the Brisbane Cardiac Consortium Leadership Group are listed on the inside front cover.

References
  1. Australian Institute of Health and Welfare. Heart, stroke and vascular diseases – Australian facts 2001. AIHW Catalogue No. CVD 13. Canberra: AIHW, National Heart Foundation of Australia, National Stroke Foundation of Australia, 2001. (Cardiovascular Disease Series No. 14.)
  2. Scott IA, Denaro CP, Flores JL, et al. Quality of care of patients hospitalised with acute coronary syndromes. Intern Med J 2002; 32: 502-511. <PubMed>
  3. Scott IA, Denaro CP, Flores JL, et al. Quality of care of patients hospitalised with congestive heart failure. Intern Med J 2003; 33: 140-151. <PubMed>
  4. Mudge AM, Brockett R, Foxcroft KF, Denaro CP. Lipid-lowering therapy following major cardiac events: progress and deficits. Med J Aust 2001; 175: 138-140. <PubMed>
  5. Scott IA, Lindsay K, Harden H. Utilisation of outpatient cardiac rehabilitation in Queensland. Med J Aust 2003; 179: 341-345. <eMJA full text> <PubMed>
  6. Vale MJ, Jelinek MV, Best JD, for the COACH study group. How many patients with coronary heart disease are not achieving their risk-factor targets? Experience in Victoria 1996–1998 versus 1999–2000. Med J Aust 2002; 176: 211-215. <eMJA full text> <PubMed>
  7. Krum H, Tonkin AM, Currie R, et al. Chronic heart failure in Australian general practice. The Cardiac Awareness Survey and Evaluation (CASE) study. Med J Aust 2001; 174: 439-444. <eMJA full text> <PubMed>
  8. Grimshaw JM, Eccles MP, Walker AE. Changing physicians’ behaviour: what works and thoughts on getting more things to work. J Contin Educ Health Prof 2002; 22: 237-243. <PubMed>
  9. Hickey A, Scott I, Denaro C, et al. Using clinical indicators in a quality improvement programme targeting cardiac care. Int J Qual Health Care 2004; 16 Suppl 1: i11-i25.
  10. Centor RM, Schwartz JS. An evaluation of methods for estimating the area under the receiver operator characteristic (ROC) curve. Med Decis Making 1985; 5: 149-156. <PubMed>
  11. National Institute of Clinical Studies. Heart failure resources directory, 2004. Available at: www.nicsl.com.au/projects_projects_detail.aspx?view=15&subpage=19 (accessed Mar 2004).
  12. Severens JL. Value for money of changing healthcare services? Economic evaluation of quality improvement. Qual Saf Health Care 2003; 12: 366-371. <PubMed>

(Received 14 Jan 2004, accepted 18 Mar 2004)

Clinical Services Evaluation Unit, Princess Alexandra Hospital, Brisbane, QLD.

Ian A Scott, MHA, FRACP, Director of Internal Medicine; and Associate Professor, University of Queensland; Annabel C Hickey, MMSc(Clin Epi), BAppSc(OT), Manager, Clinical Services Evaluation Unit; Daniela C J Sanders, BPharm(Hons), Research Clinical Pharmacist; Mark A Jones, BSc(Hons), Biostatistician.

Internal Medicine Research Unit, Royal Brisbane Hospital, Brisbane, QLD.

Charles P Denaro, MD, FRACP, Director of Internal Medicine; and Associate Professor, University of Queensland; Cameron J Bennett, MB BS, MBiomedE, FRACP, Medical Director, Division of Medicine; Alison M Mudge, MB BS, FRACP, Research Physician, Internal Medicine Research Unit; Justine M Thiele, BPharm, Research Clinical Pharmacist.

Department of Medicine, Queen Elizabeth II Hospital, Brisbane, QLD.

Judy L Flores, BA, MD, FRACP, Director of Medicine.

Brisbane North Division of General Practice, Brisbane, QLD.

Beres Wenck, FRACGP, General Practitioner; John W Bennett, BMedSc, MB BS, BA(Hons), FRACGP, General Practitioner.

Correspondence: Dr Ian A Scott, Clinical Services Evaluation Unit, Princess Alexandra Hospital, Ipswich Road, Woolloongabba, Brisbane, QLD 4012. Ian_scottAThealth.qld.gov.au

AntiSpam note: To avoid spam, authors' email addresses are written with AT in place of the usual symbol, and we have removed "mail to" links. Replace AT with the correct symbol to get a valid address.

©The Medical Journal of Australia 2004 www.mja.com.au ISSN: 0025-729X

Home | Issues | eMJA shop | My account | Classifieds | More... | Contact | Topics | Search

The Medical Journal of Australia    eMJA