Design, setting and participants: Clinical practice guidelines that were produced or reviewed between 2003 and 2007 for use in Australia at a national or state level were identified by approaching health-related organisations and searching websites. Their characteristics were abstracted from the published guidelines and publicly accessible accompanying material.
Main outcome measures: Number of clinical practice guidelines, key health areas, documentation of evidence search and appraisal processes, numbers and types of guideline producers and funders, presence of competing interest statements.
Results: 313 clinical practice guidelines were identified, of which 91 (29%) were evidence-documented, either in the guideline itself or in an accessible accompanying document. Over 80 guideline producers were identified. Federal or state government agencies produced or contributed funding to 53% of the guidelines (167/313); 28% of the guidelines supported by government agencies (46/167) were categorised as evidence-documented. A review date was specified in 52% of evidence-documented guidelines (47/91), but a third of these had passed the review date at the time of our study and no updated guidelines were found. Areas with a large burden of disease did not necessarily receive government support for guideline development. Most guidelines (246/313; 79%) made no mention of possible competing interests of members of the guideline development group.
Conclusions: A more coordinated approach to identifying national priorities for developing and updating clinical practice guidelines may produce better returns on investment in Australian guidelines. In addition, more transparency in documenting the guideline development process, including details on competing interests, is needed.
Evidence-based clinical practice guidelines translate findings from health research into recommendations for clinical practice1 and, when implemented, can improve health outcomes.2-4 Currently, Australian clinical practice guidelines are produced by disparate groups, including government agencies and professional societies. Without information on the number and topics of current Australian guidelines, and documentation of evidence search and appraisal processes on which they are based, it is not possible to assess the degree to which clinical practice in Australia matches best practice recommendations in specific areas, or to identify priorities for guideline implementation.
A 1993 study of national and state clinical practice guidelines identified 34 clinical practice guidelines produced by 32 organisations.5 None of these guidelines fully described their methods for finding and synthesising evidence, and 29 of the 34 guidelines gave no description of the methods used to identify evidence.
In 2006, the National Institute of Clinical Studies (NICS) initiated a study of national and state Australian clinical practice guidelines to identify priorities for future implementation programs. The study continued after NICS became an institute of the National Health and Medical Research Council (NHMRC) in 2007. Here, we report the study findings, including the extent to which clinical practice guidelines produced between 2003 and 2007 documented their evidence search and appraisal processes.
In 2006, 179 Australian national and state health-related organisations were approached to obtain clinical practice guidelines that they had produced or endorsed for use within the previous 5 years. Non-responding organisations received a follow-up letter and telephone call. Organisations that were identified as guideline producers were contacted again in 2007 and 2008 to ensure that new and updated guidelines had been identified. This approach was supplemented by searching websites of the following Australian sources: the Department of Health and Ageing, the Department of Veterans’ Affairs, the NHMRC, the Australian Commission on Safety and Quality in Health Care, all state and territory health departments, and The Medical Journal of Australia. The United States National Guideline Clearinghouse and the New Zealand Guidelines Group websites were also searched for Australian guidelines.
Clinical practice guidelines were defined as documents containing recommendations to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances. A modified form of the US National Guideline Clearinghouse inclusion criteria was used.6 Included guidelines did not need to show evidence of systematic development, but did need to contain recommendations concerning individual health care and be: produced or reviewed between 2003 and 2007 (the most recent version was assessed); produced by a national medical specialty association, professional society, government agency or other health organisation; and intended for use at national or state level. The following were excluded: policy, procedural and protocol statements; ethics guidelines; formularies; documents produced by a health service or hospital (unless endorsed for wider use by a national or state organisation); and guidelines produced before 2003 and reissued without review.
All identified documents were screened against the inclusion criteria by two people (K C C, plus one other person). Those that met the inclusion criteria were assessed independently by two people (K C C or H A B, plus one other person); this involved abstracting information about the key health area (based on the primary topic of the guideline), sources of funding, producers, documentation of the evidence search and appraisal process, citation of NHMRC guidance for guideline development, competing interest declarations, and plans for review. Documentation of the evidence appraisal process was determined by reviewing the information provided in the guideline or in a publicly accessible accompanying document. A guideline was classified as “evidence-documented” if, as a minimum, a brief description of a literature review that contained some specific details of the search and appraisal process was given. Guidelines without mention of a literature review, and those that stated that a literature review had occurred but gave no specific details, were classified as “not evidence-documented”, even if references were listed in the guideline. A guideline producer was defined as the group responsible for development of the guideline. An organisation was classified as a funder if the guideline explicitly stated that the organisation had provided funding. Disagreements between assessors were resolved by discussion and recourse to a third assessor if necessary (K C C or H A B).
Of the 1014 documents received from national and state health-related organisations or identified through website searches, 313 met the inclusion criteria for clinical practice guidelines produced or reviewed between 2003 and 2007. The reduction from 1014 to 313 documents resulted from document duplications and documents that were not related to clinical practice or were public health guidelines. Ninety-one of the 313 clinical practice guidelines (29%) were classified as evidence-documented, and 60 of these described their evidence search and appraisal process in sufficient detail for it to be replicated. Box 1 shows the numbers of guidelines, with and without evidence documentation, by key health area.
Over 80% of the evidence-documented guidelines (76/91) cited one or more of the NHMRC “guidelines for guidelines” series of publications.1,7-12 Twenty-three percent of the guidelines without evidence documentation (50/222) also cited one or more of the publications from this series.
Box 2 shows numbers of guidelines, with and without evidence documentation, by category of guideline producer. More producers than guidelines were identified, because some guidelines had multiple producers. Eighty-two separate organisations or one-off collaborations were involved in producing guidelines.
More than half of the guidelines (173/313) did not include a statement about funding sources. Box 3 shows the numbers of guidelines, with and without evidence documentation, by key health areas and federal or state government agency support. Federal or state government agencies produced or funded 53% of the guidelines (167/313). Twenty-eight percent of government-supported guidelines (46/167) were categorised as evidence-documented. Almost all of the guidelines on drugs and alcohol received some government support. In contrast, most of the guidelines on cardiovascular and renal disease were funded by non-government sources. Pharmaceutical companies were identified as funders of 27 guidelines, 20 of which were evidence-documented.
Box 4 shows the numbers of evidence-documented guidelines produced or reviewed between 2003 and 2007 that would expire or require review if expiry dates of 5 years and 3 years were applied. Guideline producers specified a review date for 52% (47/91) of the evidence-documented guidelines in our study. For 14 of these guidelines, the guideline producer specified that a review should happen in 2007 or earlier but the guideline had not been reviewed by the specified date.
Most of the guidelines made no mention of possible competing interests (Box 5), and specific information on how any competing interests were managed during guideline development (eg, whether individuals with financial ties to specific companies participated in formulating recommendations for use of these companies’ products) was not published in any of the guidelines. Of 22 organisations that produced or co-produced five or more guidelines during the study period, competing interest policies specific to guideline development were available on the websites of four, while eight had general competing interest policies applying to committees, authorship or relationships with pharmaceutical companies. No information about competing interest policies was published on the websites of the remaining 10 organisations.
We identified 313 Australian clinical practice guidelines, including 91 evidence-documented guidelines. About half of the guidelines were funded or produced by government agencies, of which 28% were evidence-documented. Most of the guidelines made no mention of possible competing interests.
There has been a substantial growth in the number of Australian clinical practice guidelines in the past 15 years. Our study identified nine times more guidelines than identified in 1993,5 although our search was more extensive and not confined to guidelines for medical practitioners. In 1993, 15% of guidelines described the methods used to identify evidence and none gave full descriptions of these methods.5 In our study, nearly a third of guidelines provided specific details of the search and appraisal process, and almost 20% fully described the process in the guideline or an accompanying document. Given the extent to which NHMRC guidelines for guidelines1,7-12 (which have a strong focus on review and use of evidence) were cited by the guidelines identified in our study, it is reasonable to conclude that the NHMRC has been influential in improving this aspect of the quality of Australian clinical practice guidelines.
A 1997 study of 286 Australian general practitioners showed that 88% of GPs thought it was extremely or very important that a guideline be evidence-based.13 Documentation of the evidence search and appraisal process provides prospective users with information about the evidence basis for guideline recommendations. In our study, if there was a guideline in a clinical area where there was no evidence to guide practice, the guideline would still have been classified as evidence-documented if some details had been given about a literature search, even if the search proved fruitless. A majority of guidelines produced in several key health areas — such as asthma, cardiovascular disease, drugs and alcohol, and infectious diseases — did not document the evidence review process. This may be because the guidelines are based on consensus rather than evidence, because a non-systematic approach to evidence review was used, or because the guideline producers feel it is unnecessary to provide basic details about the process to prospective guideline users. Although there is little point in producing voluminous information about guideline development methods, if the information needed to guide clinical practice is not easily identifiable or accessible, it is important that guideline users are able to reach informed judgements about the basis for recommendations. Transparency and conciseness are not incompatible — many of the guidelines classified as evidence-documented in our study described their evidence-appraisal process in less than a page or provided information about the review in a separate, publicly available document.
Our study had some limitations. Assessors did not check whether the evidence search and appraisal process described for guidelines categorised as evidence-documented was the most appropriate for the topic under review, so some guidelines included in this category may be based on inappropriate or poor-quality reviews. On the other hand, some guidelines that were based on structured literature reviews may have been categorised as not evidence-documented because no details of the process were provided in a publicly available document. Similarly, data on funding and competing interests were derived from information provided in the guidelines and accompanying material. Also, despite extensive efforts to identify all national and state guidelines produced during the study period, some documents may have been missed.
The need for transparency and appropriate management of competing interests has been highlighted recently in Australia and overseas.14-18 It is important that guidelines are not biased, or perceived as biased, because of financial or other competing interests of individuals or groups involved in the guideline development process. Competing interests of guideline developers were poorly recorded in the guidelines that we assessed — they were infrequently reported within the guidelines, and policies on management of competing interests were not clearly stated by many organisations that produced multiple guidelines. This highlights the importance of clear local guidance and the need for revised NHMRC guidelines in this area.
The effort and resources needed to produce clinical practice guidelines are largely wasted if guidelines are not easily available to their target audience beyond the initial publication date. We found that there was no easy way to find locally produced guidelines on specific topics. For example, clinical guidelines on websites are hard to locate, particularly if the documents do not have the term “clinical guidelines” in their title or content. To help improve access to current guidelines, the NHMRC launched a Clinical Practice Guidelines Portal in February 2010 (http://www.clinicalguidelines.gov.au/) which provides information on current Australian guidelines, with links to the websites of agencies that publish guidelines. This should also help identify gaps and duplications in the pool of currently available guidelines.
Government agencies produced or funded just over half the clinical practice guidelines identified in our study. Despite this, most health departments had no central point of reference for guidelines and were unable to provide comprehensive information about the guidelines they produced, commissioned or helped fund. This probably reflects the “bottom-up” way in which guideline activity has grown in Australia, with government support for guideline development occurring as one of a range of activities within particular departmental health program areas rather than as a specifically identified function of government. The benefits of a bottom-up approach are that government health departments can be locally responsive to needs expressed by clinicians, and that there may be less clinical concern about bureaucratic involvement in clinical practice issues. However, the lack of coordination associated with this approach leads to gaps and duplications, and an inconsistent approach to ensuring that guidelines are of high methodological quality.
One key rationale for development of clinical practice guidelines is the rapid growth in research knowledge in some clinical areas, which means that recommendations for best practice may change as the results of important studies or new treatments become available. The value of evidence-based guidelines therefore depends on the extent to which guideline recommendations reflect current knowledge. However, a sporadic one-off approach to guideline funding means that there is no ongoing commitment to ensuring that guidelines in clinically important areas are kept up to date. Also, little research on when and how guidelines should be updated has been carried out. The only empirical study on how quickly good-quality guidelines become outdated that has been published to date showed that half of such guidelines were outdated within 5.8 years of being produced.19 The authors recommended that, as a rule, guidelines should be reassessed for validity every 3 years. If a scheduled review date of 3 years was applied to the group of evidence-documented guidelines in our study, 60 of the 91 guidelines would have required reassessment by the end of 2008; if a scheduled review date of 5 years was applied, 32 would have required review by the end of 2009. Within the study period, 15% (14/91) of the evidence-documented guidelines had already passed the review dates recommended by their authors.
High-quality clinical practice guidelines provide an opportunity to close the gaps between current clinical practice and best available evidence. Given the large number of clinical practice guidelines being produced in Australia, a substantial amount of money and effort is being spent on guideline production, with federal and state governments supporting much of this activity. However, there are deficiencies in the documentation of basic information that could help users assess guideline quality, such as information about literature searches and review processes and management of competing interests. A requirement that government-funded guidelines undertake systematic literature reviews and document these processes may result in the production of fewer high-quality guidelines, but would help assure potential users that guideline recommendations reflect the best available research knowledge. There would be similar value in a requirement for a transparent approach to identifying and managing competing interests. In addition, although a large number of the guidelines that we identified were government-funded, some health areas with heavy disease burden do not appear to have received government support for guideline production.
A coordinated national approach to funding, producing and updating high-quality guidelines in priority health areas may provide opportunities to gain greater returns on the current investment in clinical practice guidelines.
1 Clinical practice guidelines by key health area
2 Clinical practice guidelines by category of guideline producer*
3 Clinical practice guidelines by key health area and federal and state government agency support
4 Year of expiry or review for evidence-documented clinical practice guidelines (n = 91)
5 Clinical practice guidelines by inclusion and type of competing interest statement
Received 14 September 2009, accepted 11 February 2010
- 1. National Health and Medical Research Council. A guide to the development, implementation and evaluation of clinical practice guidelines. Canberra: NHMRC, 1999.
- 2. Menendez R, Torres A, Zalacain R, et al. Guidelines for the treatment of community-acquired pneumonia: predictors of adherence and outcome. Am J Respir Crit Care Med 2005; 172: 757-762.
- 3. Fonarow GC, Abraham WT, Albert NM, et al. Association between performance measures and clinical outcomes for patients hospitalized with heart failure. JAMA 2007; 297: 61-70.
- 4. Du Pen SL, Du Pen AR, Polissar N, et al. Implementing guidelines for cancer pain management: results of a randomized controlled clinical trial. J Clin Oncol 1999; 17: 361-370.
- 5. Ward JE, Grieco V. Why we need guidelines for guidelines: a study of the quality of clinical practice guidelines in Australia. Med J Aust 1996; 165: 574-576.
- 6. National Guideline Clearinghouse. Inclusion Criteria. http://www.guideline.gov/submit/inclusion.aspx (accessed Dec 2008).
- 7. National Health and Medical Research Council. How to review the evidence: systematic identification and review of scientific literature. Canberra: NHMRC, 2000.
- 8. National Health and Medical Research Council. How to present the evidence for consumers: preparation of consumer publications. Canberra: NHMRC, 2000.
- 9. National Health and Medical Research Council. How to put evidence into practice: implementation and dissemination strategies. Canberra: NHMRC, 2000.
- 10. National Health and Medical Research Council. How to use the evidence: assessment and application of scientific evidence. Canberra: NHMRC, 2000.
- 11. National Health and Medical Research Council. How to compare the costs and benefits: evaluation of the economic evidence. Canberra: NHMRC, 2001.
- 12. National Health and Medical Research Council. Using socioeconomic evidence in clinical practice guidelines. Canberra: NHMRC, 2003.
- 13. Gupta L, Ward JE, Hayward RS. Clinical practice guidelines in general practice: a national survey of recall, attitudes and impact. Med J Aust 1997; 166: 69-72.
- 14. Boyd EA, Bero LA. Improving the use of research evidence in guideline development: 4. Managing conflicts of interests. Health Res Policy Syst 2006; 4: 16.
- 15. Van Der Weyden MB. Doctors and the pharmaceutical industry: time for a national policy [editorial]? Med J Aust 2009; 190: 407-408. <MJA full text>
- 16. Choudhry NK, Stelfox HT, Detsky AS. Relationships between authors of clinical practice guidelines and the pharmaceutical industry. JAMA 2002; 287: 612-617.
- 17. Van Der Weyden MB. Clinical practice guidelines: time to move the debate from the how to the who [editorial]. Med J Aust 2002; 176: 304-305.
- 18. Lo B, Field MJ, editors. Conflict of interest in medical research, education, and practice. Washington: National Academies Press, 2009. http://www.nap.edu/openbook.php?record_id=12598&page=R1 (accessed Mar 2010).
- 19. Shekelle PG, Ortiz E, Rhodes S, et al. Validity of the Agency for Healthcare Research and Quality clinical practice guidelines: how quickly do guidelines become outdated? JAMA 2001; 286: 1461-1467.