Accreditation as a quality improvement tool: is it still relevant?

Despite lack of quantitative evidence, accreditation still works, although there is room for improvement

Accreditation is, as a method of reviewing the quality of medical education, more than 100 years old1 and is traditionally based on the episodic “biopsy” model, which involves periodic assessment, usually against defined standards.2 Despite acknowledgement of the importance of accreditation,3,4 there is limited hard evidence to support its effectiveness or impact, reflecting in particular the social constructs of the accreditation model and of the education setting in which it is applied. Accreditation in this context assesses to a significant degree the quality of human interactions, obligating a less measurable research approach, more qualitative than quantitative.5 Applicable social constructivist research methodologies are common in the education setting, as detailed in the recent MJA article by Durning and Schuwirth.6 Although there is some quantitative evidence for the effectiveness of accreditation, such as better clinical learning climate survey scores in accredited programs,7 critical outcomes of impacts on health care have not been quantitated, and a causative link between accreditation and educational quality has not yet been clearly established.4 Further, ongoing changes and innovations in health care and medical education call for a flexible approach to accreditation design,4,8 so even if hard evidence was available, its applicability may be limited to specific and perhaps outdated settings.

While quantitative proof may be elusive, accreditation remains a widely applied quality improvement approach. Together with a general acceptance of its role,3,9 this continued usage has driven a research-based approach to accreditation, focusing on the aspects of accreditation that are useful,10 how to use accreditation optimally, and analysis of different designs and methods.

In this article, using our experience with international accreditation processes and within the Australian system, we apply the four themes of an accreditation strategy framework — objectives, quality domains, quality management and stakeholders’ responsibilities8 — to explore the design of the accreditation system for Australian medical specialist training. For each theme, we consider opportunities for improvement from an international perspective drawn from the literature and practice (Box 1). The system and mandate of the Australian Medical Council (AMC) in undertaking the accreditation process for Australian medical specialist training are explained in Box 2.

Objectives of accreditation: why is medical specialist training accredited?

Accreditation has primary and secondary objectives. The primary objectives relate to the quality of training and health care, summarised in the primary aim of the AMC as being “to ensure that standards of education, training and assessment of the medical profession promote and protect the health of the Australian community”.11 Both training and health care are embedded in the standards developed for the specialist training programs. In the accreditation strategy framework, the primary objectives may be supported by four secondary objectives: standardisation, coaching, self-evaluation and accountability. The current AMC accreditation processes mainly focus on compliance with standards. The AMC does not specifically prescribe how the standards are achieved, but it does have a coaching role in which it supports the medical specialist colleges and facilities; for example, with illustrations of best practices or with accreditors’ own experiences. Self-evaluation is encouraged in the accreditation processes as well as in the standards encouraging the training provider to have flexibility in its own quality management. Accountability by being answerable to different stakeholders is well represented by the AMC, particularly accountability to the community. For example, lay committees are part of the process and all accreditation reports and outcomes are publicly accessible on the AMC website (http://www.amc.org.au).

Opportunities for improvement from an international perspective

The AMC’s primary objectives are in line with those of other accreditation authorities around the world, although practices differ. Internationally, there is an ongoing focus on the benefits of self-evaluation and continuous quality improvement cycles.12,13 The AMC could reorient its accreditation of specialist programs from a focus on compliance with standards, to emphasise the colleges’ and facilities’ self-assessment and quality management processes, which also could promote each college’s ownership of quality.

The reward for colleges, and for hospitals and other facilities which are in turn accredited by the colleges, could be a more nimble and focused accreditation process when they conduct thorough self-evaluation and demonstrate reflective practice and continuous quality improvement. Perception by colleges and facilities of the standards and processes as supportive is likely to raise intrinsic motivation;14 conversely, experience from other regulatory fields shows that a rigid application of standards leads to an unresponsive process which can undermine the intrinsic motivation of the accredited party.5,14 It should be noted that self-evaluation still includes a role for the accreditation authority as an external body to validate the training provider’s own processes.15

Quality domains: what is accredited in medical specialist training?

The quality domains, in essence the educational content of the standards, consist of quality of education (eg, curriculum), trainee performance (eg, assessment) and quality of physician practice (eg, graduate outcomes). There are ten AMC standards (https://www.amc.org.au/files/059c6d1dee49caa6c434c18138f6373fb4be8b36_original.pdf) that cover all three domains, even though the quality of physician practice is limited to the requirement of defining graduate outcomes. The AMC does not specify the domains in detail, as definition of the specialty-specific requirements and collection of the relevant quantitative and qualitative information are college responsibilities.

Opportunities for improvement from an international perspective

The AMC standards require the colleges to define outcomes in concordance with the current international emphasis on outcomes in medical education.4 However, there is also a discourse to bring balance between this outcome focus and to embrace markers of a continuous quality improvement culture.13 Blouin and Tekian advocate an emphasis on a program’s process rather than only its product when determining the impact of accreditation.13

While superficially attractive, overspecification can be counterproductive. For example, the development of new standards to cover topical issues such as health professional wellbeing is often not an effective solution.16,17 Making rules instead of solving the problems, described as rule ritualism,5 can limit accreditation effectiveness.5,18 Other examples are legal ritualism, in which the letter of the law is followed instead of the spirit, and protocol ritualism, in which the protocol is followed even if this makes outcomes worse. Thematic reviews, in which topical issues in training or health care are evaluated without necessarily making new standards, could resist ritualism. Thematic reviews do not replace regular visits, but give the accreditation authority new tools to focus on topical issues. The General Medical Council (GMC) of the United Kingdom, for example, conducts risk-based spot checks to explore issues of which they are aware. While these are performed outside regular visit cycles, they take into account the GMC standards and outcomes. Such an approach has to date had limited international uptake.

Quality management: how is medical specialist training accredited?

Quality management aims to achieve the objectives of accreditation as effectively as possible through a variable mix of quality assurance, quality improvement and quality control. The choice of approach influences the nature and number of standards. In the quality assurance approach, minimum standards are set to assure the quality of education and graduates produced.8 The legislation under which the AMC accredits programs requires this approach and focuses on compliance. The quality improvement approach is based on striving for excellence. In its accreditation reports, the AMC creates opportunities for quality improvement by including recommendations for improvement for each of the ten standards. These recommendations consist of non-binding advice as opposed to conditions that must be met, although colleges and facilities must respond to these recommendations in their progress reports. It is assumed that accredited parties are striving to demonstrate optimal performance when the accreditation visit is near, although there is a danger that after the visit impetus is lost until the next visit.12 The AMC attempts to minimise this imbalance by requiring regular progress reports from the colleges. Outcome-based standards are the focus of quality control. Defining the specific outcomes is mandated to the colleges by the AMC.

Opportunities for improvement from an international perspective

The annual progress reports are in concordance with the international discourse to move away from traditional accreditation cycles and use more continuous monitoring processes. Annual data collection systems as used by the Accreditation Council for Graduate Medical Education in the United States and the national training surveys conducted by the GMC are examples of continuous monitoring. Whichever approaches are chosen by the accreditation authority, it is fundamental that the process is fair and transparent for the education provider.

Stakeholders’ responsibilities: who is responsible for the quality of medical specialist training?

The major players in the Australian accreditation process are the AMC, the providers of specialist training and the training sites. The colleges set the curriculum and assessment, and the standards for training at training sites, and so accredit these sites themselves. The college is thereby entrusted with significant power and responsibility for training by the AMC. The AMC visits a sample of the training sites (about ten sites) to gather information about trainee satisfaction with the training program and to validate the college’s information.

Opportunities for improvement from an international perspective

Through its recognition of college responsibilities for ensuring the standards for training in the clinical workplace, the AMC is remote from the workplace in comparison to other jurisdictions. Canada, the US, the UK and the Netherlands focus not at a national (college) level but at a regional or training site level.19-22 The literature has yet to establish the relative advantages and disadvantages of these approaches.

Balancing accreditation systems towards continuous quality improvement

We have explored Australian and international perspectives on accreditation system design using the accreditation strategy framework model.8 Based on our appraisal of the AMC accreditation system against the four themes, the most important opportunity for the AMC, and in turn the colleges, would appear to be to stimulate a more systematic continuous quality improvement cycle at college and training site level. This finding concurs with a current review of accreditation systems for health professions by the Council of Australian Governments Health Council, which recommends a more consistent and common approach to training site accreditation conducted by the colleges.23

Although the importance of accreditation is increasingly acknowledged in the literature and the accreditation strategy framework provides insight in accreditation design, deeper understanding about the effectiveness of different designs in relation to the quality of medical specialist training is needed. If we consider accreditation as a diagnostic tool to determine the quality of training and to reveal shortcomings in this quality, we can assess an accreditation system by asking whether or not it complies with standards for reliable and valid tools. Put simply: are the results of accreditation consistent and does accreditation measure what it is supposed to measure? Systematic appraisal of accreditation as well as international exchange of best practices may lead to a number of globally accepted theories about accreditation. Both qualitative and quantitative research should challenge these theories and determine the effectiveness of different accreditation designs.