Design: Selection and assessment of national clinical guidelines for chronic conditions listed as National Health Priority Areas: cardiovascular health, diabetes mellitus, mental health, asthma, arthritis and musculoskeletal conditions, and cancer.
Main outcome measures: Standardised mean scores obtained with the Appraisal of Guidelines Research and Evaluation (AGREE) instrument (criteria grouped into six domains: scope and purpose; stakeholder involvement; rigour of development; clarity and presentation; applicability; and editorial independence). Relevance of guidelines for older people with multiple comorbid conditions.
Results: 17 guidelines were included in the study. Guidelines approved by the National Health and Medical Research Council (NHMRC) scored significantly better than those not approved by the NHMRC in all domains except for editorial independence and clarity and presentation. The mean quality of guidelines not approved by the NHMRC was below 50% in all domains except clarity and presentation. Half of the guidelines addressed treatment for older patients or for patients with one comorbid condition, but only one addressed treatment for older patients with multiple comorbid conditions.
Conclusions: Professional societies and charities should be encouraged and supported to develop clinical guidelines in compliance with NHMRC requirements. Future guidelines should place more emphasis on the management of older people with multiple comorbid conditions.
Clinical guidelines have been developed to improve health care by increasing the uptake of evidence-based treatments and reducing the use of unnecessary, ineffective or harmful interventions.1 However, several recent studies have found that the methodological quality of clinical guidelines was highly variable.2-5 Guidelines developed by professional societies were of poorer quality than guidelines produced by governmental agencies.6-8 Researchers and commentators have noted that most guidelines for chronic diseases do not modify or discuss the applicability of their recommendations to older patients with multiple comorbid conditions, and that they provide limited guidance on combined use of treatments for different diseases.9,10
In Australia, the National Health and Medical Research Council (NHMRC) has adopted standards with which guideline developers need to comply if they want NHMRC approval. NHMRC standards require strong methods, the involvement of a multidisciplinary working panel and a public consultation process.11,12 In practice, many Australian clinical guidelines are produced by professional societies and charities without seeking NHMRC approval.
We aimed to assess the quality of selected Australian national clinical guidelines and their relevance to older people with multiple illnesses. We also compared the quality of NHMRC-approved guidelines with those that were not NHMRC-approved.
National guidelines for the chronic conditions listed as National Health Priority Areas (cardiovascular health, diabetes mellitus, mental health, asthma, arthritis and musculoskeletal conditions, and cancer) were selected, including guidelines for the most prevalent cancers in Australia, colorectal, breast and lung cancers. When several guidelines had been published by the same organisation on the same topic, only the most comprehensive and/or the latest version was included. Position or consensus statements that had not been developed by a systematic approach to the retrieval and the analysis of the literature were excluded.
Guidelines were identified through searches in the MEDLINE database for 2000–2006 (“guideline” [publication type or text word] and “Australia” as keywords) and on the Internet by means of the Google search engine. Additionally, the following Australian websites were searched: Australian Government Department of Heath and Ageing; NHMRC; National Institute of Clinical Studies; Royal Australian College of General Practitioners; National Heart Foundation of Australia; Cardiac Society of Australia and New Zealand; Australian Resuscitation Council; Cancer Council of Australia; Arthritis Foundation; Diabetes Australia; Osteoporosis Australia; Medical Journal of Australia; and the Internal Medicine Journal. International guideline websites (New Zealand Guidelines Group and the National Guideline Clearinghouse) were also searched for Australian guidelines.
All guidelines were assessed independently by two reviewers. The quality of guidelines was assessed by using the Appraisal of Guidelines Research and Evaluation (AGREE) instrument (Box 1).13 The AGREE instrument has been validated and tested in several countries,3,5 and is considered the best current tool for assessing the quality of a guideline.14-16 It includes 23 items within the six theoretical domains shown in Box 1. A four-point Likert scale was used to score each item. Agreement between the two reviewers for quality scores was measured by means of linear weighted κ statistics. Scores of the two reviewers were then summed and standardised domain scores were calculated as the percentage of the maximum possible score. The mean domain scores for guidelines that were and were not NHMRC-approved were compared by means of the Mann–Whitney U test.
The relevance of the guidelines to the care of older people with multiple illnesses was assessed by means of a specific instrument developed in a previous study (Box 2).9 This instrument includes 14 items assessing whether guidelines address treatment for older people and for people with several co-morbid conditions, as well as patient-centred aspects such as patients’ preferences and quality of life. Agreement between the two reviewers for quality scores was measured with κ statistics. Any disagreement was then resolved by discussion.
Seventeen national guidelines were selected, including five for cardiovascular health, four for cancer, four for mental health, two for respiratory health, one for musculoskeletal conditions and one for diabetes mellitus (Box 3).17-33 No national Australian guideline for the management of arthritis was identified. All guidelines were developed by professional or disease-oriented charity organisations. Only two received governmental funding. Six had been approved by the NHMRC, including four for cancer, one for cardiovascular health and one for diabetes.
Box 1 shows that the agreement between the two reviewers was excellent for 10 items (weighted κ > 0.8), good for 10 items (weighted κ > 0.6–0.8) and moderate for three items (weighted κ = 0.4 to < 0.6). The mean and individual standardised AGREE domain scores are presented in Box 4 and Box 3, respectively. Clarity and presentation provided the highest mean score of 71.3%. The six NHMRC-approved guidelines had significantly higher domain scores than the guidelines not approved by the NHMRC for all domains except editorial independence and clarity and presentation, where no difference was observed (Box 4).
The agreement between the two reviewers was excellent for 10 items and good for 4 items (data not shown). Box 2 shows that eight guidelines (47%) addressed treatment for older patients, nine (53%) addressed treatment for patients with multiple comorbid conditions and one (6%) addressed treatment for older patients with multiple comorbid conditions
This study showed that the quality of Australian guidelines for chronic conditions listed as National Health Priority Areas was average (around 50%) in three domains, low in two domains and good in one domain. The mean quality of guidelines not approved by the NHMRC was below 50% in all domains except clarity and presentation. Only a few guidelines not approved by the NHMRC described the methods used for selecting the evidence or formulating the recommendations. Our findings are consistent with the results of other studies which showed that quality of guidelines produced by specialist societies was low,6 and that high-quality guidelines were more likely to be produced by government-funded agencies or developed within a structured and coordinated program to produce clinical practice guidelines.5,7,34
Compared with an international study that assessed the quality of 86 guidelines in 10 European countries and Canada,5 mean scores achieved by Australian guidelines in our study were higher for stakeholder involvement (45.8% v 33.6%), rigour of development (40.9% v 36.9%), clarity and presentation (71.3% v 57.2%), and lower for scope and purpose (54.3% v 66.1%), applicability (21.6% v 31.3%) and editorial independence (11.8% v 47.8%). This was mainly owing to the high mean scores for NHMRC-approved guidelines, which were higher than mean scores observed in all other international studies that have used the AGREE instrument, in all domains except scope and purpose, and editorial independence.2-5,34-39
All the Australian guidelines we studied, regardless of whether they were approved by the NHMRC, performed poorly in the domain of editorial independence. Disclosure of conflicts of interest of guideline development members was not a requirement of the NHMRC until a recent update of its standards in October 2007.11
The applicability domain evaluates issues that are pertinent to guideline implementation, such as organisational barriers, cost implications and monitoring criteria. The mean score for the Australian guidelines was low, suggesting implementation barriers were not sufficiently overcome in Australian guidelines. However, it should be noted that the AGREE instrument does not assess the actual implementation of guidelines.
Low-quality scores in the rigour of development domain raise concern about guideline validity. This needs to be addressed as the promotion of invalid guidelines may result in ineffective or unsafe treatments, or inefficient use of resources. The costs involved with the development of rigorous guidelines based on systematic reviews of the evidence may be a barrier for a single Australian professional society or charity. There is a limited amount of public funding for guideline development. Only two guidelines in our sample were funded by state or federal government departments. Increased resources for developing of guidelines may be required to overcome this barrier.
Another reason for not following the NHMRC process is that several professional organisations have chosen to adapt existing international guidelines for use in Australia rather than develop their own. This option avoids duplicating efforts when high-quality international guidelines are already available on similar topics. However, there is no validated process for adapting guidelines produced in one cultural and organisational setting for use in another.40 NHMRC guidance in this area would be welcome to support the efforts of professional societies.
Only half of the guidelines studied covered treatment for older patients and for patients with a single comorbid condition. Only one discussed issues of older people with multiple conditions. These results are consistent with a previous American study.9 The 2004–05 Australian National Health Survey reported that 80% of Australians aged 65 years or older had three or more chronic conditions,41,42 and the absence of guidance on how to manage older patients with multiple chronic conditions means that treatment decisions are left with individual practitioners. It has been proposed that meta-guidelines should be developed for the most common patterns of chronic conditions, which would define how to prioritise the many interventions that could be undertaken in people with multiple conditions.43
The AGREE instrument and the instrument used to assess relevance examined different aspects of patient-centred care. A minority of guidelines reported having involved at least one consumer in part of the guideline development (35%), discussed patients’ preferences (47%) and considered the burden of comprehensive treatment on patients or caregivers (29%). The revised NHMRC standards for externally developed guidelines require that consumers should be involved early in the development process.11 However, “effective” patient involvement in the development process is still being debated.44 Research on how patient preferences can be integrated into an evidence-based model of care is also still in its infancy.44
Our study has several limitations. The selection criteria allowed the inclusion of a wide range of important health conditions and of guideline producers. However, our results may not be generalisable to all Australian guidelines on chronic diseases or to guideline producers. We used the generally agreed definition of a guideline as “a set of systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances”.45 Because of the definition adopted and the format of the AGREE instrument, we did not include documents produced by the Royal Australian College of General Practitioners (RACGP) or Therapeutic Guidelines Limited in our analysis.
Poor quality of guidelines could also be the result of poor reporting of the methods used during guideline development rather than actual poor process. However, guideline readers should be able to assess the reliability of guidelines for themselves. Clear documentation of how guidelines were developed may also enhance their acceptance by clinicians.
The AGREE criteria provide a well validated assessment instrument, and the inter-rater agreement was excellent or good for most items in our study. However, ratings can be influenced by the reviewer’s background knowledge of the guideline topic or the method of guideline development. In our study, one reviewer had significant experience in guideline development, while the other had not been involved in guideline development. The high κ scores suggest the background knowledge of the reviewers did not significantly influence the results.
Our results show that the quality of Australian clinical guidelines is low when they have not been approved by the NHMRC. Professional societies and charities should be encouraged and supported to adopt NHMRC standards and procedures for developing guidelines. Future Australian guidelines should also provide more guidance on the management of older people with multiple comorbid conditions.
1 Overview of the AGREE criteria, and agreement between the two reviewers
2 Relevance of Australian clinical guidelines for the treatment of older patients with comorbid conditions9
4 Overall mean domain scores for the AGREE instrument, and a comparison of mean scores for guidelines that were and were not approved by the National Health and Medical Resarch Council (NHMRC)
Received 18 December 2007, accepted 11 March 2008
- 1. Woolf SH, Grol R, Hutchinson A, et al. Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines. BMJ 1999; 318: 527-530.
- 2. Arnau J, Vallano A, Lopez A, et al. A critical review of guidelines for low back pain treatment. Eur Spine J 2006; 15: 543-553.
- 3. Stiegler M, Rummel C, Wahlbeck K, et al. European psychiatric treatment guidelines: is the glass half full or half empty? Eur Psychiatry 2005; 20: 554-558.
- 4. Harpole LH, Kelley MJ, Schreiber G, et al. Assessment of the scope and quality of clinical practice guidelines in lung cancer. Chest 2003; 123: 7S-20S.
- 5. Burgers JS, Cluzeau FA, Hanna SE, et al. Characteristics of high-quality guidelines: evaluation of 86 clinical guidelines developed in ten European countries and Canada. Int J Technol Assess Health Care 2003; 19: 148-157.
- 6. Grilli R, Magrini N, Penna A, et al. Practice guidelines developed by specialty societies: the need for a critical appraisal. Lancet 2000; 355: 103-106.
- 7. Hasenfeld R, Shekelle PG. Is the methodological quality of guidelines declining in the US? Comparison of the quality of US Agency for Health Care Policy and Research (AHCPR) guidelines with those published subsequently. Qual Saf Health Care 2003; 12: 428-434.
- 8. Minhas R. Eminence-based guidelines: a quality assessment of the second Joint British Societies’ guidelines on the prevention of cardiovascular disease. Int J Clin Pract 2007; 61: 1137-1144.
- 9. Boyd CM, Darer J, Boult C, et al. Clinical practice guidelines and quality of care for older patients with multiple comorbid diseases: implications for pay for performance. JAMA 2005; 294: 716-724.
- 10. Tinetti ME, Bogardus ST Jr, Agostini JV. Potential pitfalls of disease-specific guidelines for patients with multiple conditions. N Engl J Med 2004; 351: 2870-2874.
- 11. National Health and Medical Research Council. NHMRC standards and procedures for externally developed guidelines. Canberra: NHMRC, 2007. http://www.nhmrc.gov.au/publications/synopses/nh56syn.htm (accessed Jul 2008).
- 12. National Health and Medical Research Council. A guide to the development, implementation and evaluation of clinical practice guidelines. Canberra: NHMRC, 1999. http://www.nhmrc.gov.au/ publications/synopses/cp30syn.htm (accessed Jul 2008).
- 13. AGREE (appraisal of guidelines research and evaluation) Collaboration. AGREE instrument. http://www.agreecollaboration.org/instrument/ (accessed Jul 2008).
- 14. Vlayen J, Aertgeerts B, Hannes K, et al. A systematic review of appraisal tools for clinical practice guidelines: multiple similarities and one common deficit. Int J Qual Health Care 2005; 17: 235-242.
- 15. National Institute of Clinical Studies. Summary report — assessing the implementability of guidelines. Melbourne: NICS, 2006. http://www.nhmrc.gov.au/nics/asp/index.asp?page=materials/materials_subject_article&cid=5212&id=544 (accessed Jul 2008).
- 16. MacDermid J, Brooks D, Solway S, et al. Reliability and validity of the AGREE instrument used by physical therapists in assessment of clinical practice guidelines. BMC Health Serv Res 2005; 5: 18.
- 17. National Asthma Council. Asthma management handbook. Melbourne: National Asthma Council Australia, 2006. http://www.nationalasthma.org.au/cms/index.php?option=com_frontpage&Itemid=1 (accessed Jul 2008).
- 18. iSource National Breast Cancer Centre. Clinical practice guidelines for the management of early breast cancer: second edition. Canberra: NHMRC, 2001. http://www.nhmrc.gov.au/publications/synopses/_files/cp74.pdf (accessed Jul 2008).
- 19. Australian Cancer Network Colorectal Cancer Guidelines Revision Committee. Clinical practice guidelines for the prevention, early detection and management of colorectal cancer. Sydney: The Cancer Council Australia and Australian Cancer Network, 2005. http://www.nhmrc. gov.au/publications/synopses/cp106/_files/cp106.pdf (accessed Jul 2008).
- 20. National Health and Medical Research Council. Clinical practice guidelines: evidence-based information and recommendations for the management of localised prostate cancer. A report of the Australian Cancer Network Working Party on Management of Localised Prostate Cancer. Canberra: NHMRC, 2003. http://www.nhmrc. gov.au/publications/synopses/_files/cp88.pdf (accessed Jul 2008).
- 21. Acute Coronary Syndrome Guidelines Working Group. Guidelines for the management of acute coronary syndromes. Med J Aust 2006; 184 (8 Suppl): S1-S31. <MJA full text>
- 22. McKenzie DK, Frith PA, Burdon JGW, et al. The COPD-X plan: Australian and New Zealand guidelines for the management of chronic obstructive pulmonary disease, 2007. http://www.copdx.org.au/guidelines/index.asp (accessed Jul 2008).
- 23. National Heart Foundation of Australia and Cardiac Society of Australia and New Zealand. Reducing risk in heart disease 2007. Guidelines for preventing cardiovascular events in people with coronary heart disease. http://www.heartfoundation.org.au/document/NHF/Reducing Risk_HeartDisease_FullGuide_2007.pdf (accessed Jul 2008).
- 24. National Heart Foundation of Australia. Hypertension management guide for doctors. Canberra: NHF, 2004.
- 25. National Health and Medical Research Council. National evidence-based guidelines for the management of type 2 diabetes mellitus. Part 4. Blood pressure control in type 2 diabetes. Prepared by the Australian Centre for Diabetes Strategies, Prince of Wales Hospital, Sydney, for the Diabetes Australia Guideline Development Consortium. Canberra: NHMRC, 2005. http://www.nhmrc.gov.au/publications/synopses/di7todi13syn.htm (accessed Jul 2008).
- 26. O’Neill S, MacLennan A, Bass S, et al. Guidelines for the management of postmenopausal osteoporosis for GPs. Aust Fam Physician 2004; 33: 910-917.
- 27. National Breast Cancer Centre and National Cancer Control Initiative. Clinical practice guidelines for the psychosocial care of adults with cancer. Prepared by the National Breast Cancer Centre and the National Cancer Control Initiative. Funded by the Department of Health and Ageing. A National Health Priority Area Initiative Canberra: NHMRC, 2003. http://www.nhmrc.gov.au/publications/synopses/_files/cp90.pdf (accessed Jul 2008).
- 28. NSW Health. Care of patients with dementia in general practice. Sydney: NSW Health, 2003. http://www.health.nsw.gov.au/pubs/2003/care_dementia_guide.html (accessed Jul 2008).
- 29. Ellis P; Royal Australian and New Zealand College of Psychiatrists Clinical Practice Guidelines Team for Depression. Australian and New Zealand clinical practice guidelines for the treatment of depression. Aust N Z J Psychiatry 2004; 38: 389-407.
- 30. Royal Australian and New Zealand College of Psychiatrists Clinical Practice Guidelines Team for the Treatment of Schizophrenia and Related Disorders. Royal Australian and New Zealand College of Psychiatrists clinical practice guidelines for the treatment of schizophrenia and related disorders. Aust N Z J Psychiatry 2005; 32: 1-30.
- 31. National Health and Medical Research Council. Clinical guidelines for stroke rehabilitation and recovery. National Stroke Foundation 2005. Canberra: NHMRC, 2005. http://www.nhmrc.gov.au/publications/synopses/_files/cp105.pdf (accessed Jul 2008).
- 32. National Heart Foundation of Australia and the Cardiac Society of Australia and New Zealand (Chronic Heart Failure Guidelines Expert Writing Panel). Guidelines for the prevention, detection and management of chronic heart failure in Australia, 2006. http://www.heartfoundation.org.au/document/NHF/CHF_2006_Guidelines_NHFA-CSANZ_WEB_PDF-1.2MB.pdf (accessed Jul 2008).
- 33. Ellis PM, Smith DAR. Treating depression: the beyondblue guidelines for treating depression in primary care. Med J Aust 2002; 176 (10 Suppl): S77-S83. <MJA full text>
- 34. Fervers B, Burgers JS, Haugh MC, et al. Predictors of high quality clinical practice guidelines: examples in oncology. Int J Qual Health Care 2005; 17: 123-132.
- 35. Hurdowar A, Graham ID, Bayley M, et al. Quality of stroke rehabilitation clinical practice guidelines. J Eval Clin Pract 2007; 13: 657-664.
- 36. Brosseau L, Graham I, Casimiro L, et al. What is the quality of clinical practice guidelines accessible on the world wide web for the treatment of musculoskeletal conditions in physiotherapy? Physiother Theory Pract 2004; 20: 91-105.
- 37. de Haas ERM, de Vijlder HC, van Reesema WS, et al. Quality of clinical practice guidelines in dermatological oncology. J Eur Acad Dermatol Venereol 2007; 21: 1193-1198.
- 38. Appleyard TL, Mann CH, Khan KS. Guidelines for the management of pelvic pain associated with endometriosis: a systematic appraisal of their quality. BJOG 2006; 113: 749-757.
- 39. Boluyt N, Lincke CR, Offringa M. Quality of evidence-based pediatric guidelines. Pediatrics 2005; 115: 1378-1391.
- 40. Fervers B, Burgers JS, Haugh MC, et al. Adaptation of clinical guidelines: literature review and proposition for a framework and procedure. Int J Qual Health Care 2006; 18: 167-176.
- 41. Australian Bureau of Statistics. 2004–05 National health survey summary of results. Canberra: ABS, 2006. (ABS Cat. No. 4364.0.) http://www.ausstats.abs.gov.au/Ausstats/subscriber.nsf/0/3B1917236618A042CA25711F00185526/$File/43640_2004-05.pdf (accessed Jul 2008).
- 42. Caughey GE, Vitry AI, Gilbert AL, Roughead EE. Prevalence of comorbidity of chronic diseases in Australia. BMC Public Health 2008, 8: 22. http://www.biomedcentral.com/1471-2458/8/221 (accessed Sep 2008).
- 43. Wallace P. Fitting guidelines into the real world: addressing complexity in guidelines. Fourth Annual G-I-N Conference; 2007 Aug 22-25; Toronto. http://www.g-i-n.net/download/files/plenary5_wallace.pdf (accessed Jul 2008).
- 44. Boivin A, Légaré F, van der Weijden T, et al. Are clinical practice guidelines compatible with respecting patient preferences? Conceptual framework and research agenda. Fourth Annual G-I-N Conference; 2007 Aug 22-25; Toronto. http://www.g-i-n.net/download/files/b59_boivin.pdf (accessed Jul 2008).
- 45. Field MJ, Lohr KN. Clinical practice guidelines: Directions for a new program. Washington, DC: Institute of Medicine, 1990.
Publication of your online response is subject to the Medical Journal of Australia's editorial discretion. You will be notified by email within five working days should your response be accepted.