Volume 199 - Issue 9
Assessment of medical students’ learning outcomes in Australia: current practice, future possibilities
Authors: David Wilkinson, Benedict J Canny, Jacob V Pearce, Hamish B Coates and Daniel J Edwards
Med J Aust 2013; 199 (9): 578-580. || doi: 10.5694/mja13.10061
Published online: 4 November 2013
Published online: 4 November 2013
Several projects already underway may lead to national standards in competencies for medical students, but more work remains to be done before any model of nationwide, collaborative assessment is acceptable to most medical schools.
Programs such as the Australian Medical Assessment Collaboration will allow nationwide comparisons between medical schools
The outcomes of medical education in Australia are generally regarded as being very good, and our accreditation standards are also generally recognised as being of world standard. Although individual cases of medical malpractice typically make headline news,1 there is no evidence of widespread or systemic failure of medical training. Indeed, the limited reports of intern performance that do exist suggest that most interns perform at or above a standard that is expected by their supervisors.2,3 However, there is other evidence that some graduates perceive themselves to be underprepared.4
All 19 medical schools in Australia examine and assess the performance of their students, but do so largely in isolation from each other. That is, most schools design, develop and deliver their own exams, against their own curriculum and standards, and students pass, fail and are graded with little external moderation or comparison. Accreditation of schools by the Australian Medical Council (AMC) provides some reassurance that assessment practices are appropriate in medical schools.5 However, very limited data are available for benchmarking performance against any national standard, or between medical schools in Australia.
Several projects already underway may lead to national standards in competencies for medical students, among other things. For example, the Learning and Teaching Academic Standards Project, funded by the former Australian Learning and Teaching Council, developed threshold learning outcomes that effectively define the scope of competency expected of health science graduates, including medical graduates.6
Further, Medical Deans Australia and New Zealand, the peak body representing professional-entry-level medical education, training and research in Australia and New Zealand, has developed a framework of competencies for medical graduate outcomes. One part of this framework is a list of common procedural and diagnostic skills that are expected of all graduates.7
These developments complement the current review by the AMC of its accreditation standards for primary medical education and attributes of medical graduates,8 and have occurred in the context of international developments in assessing educational outcomes in various educational settings, including medical settings.
Globally, launched under the auspices of the Organisation for Economic Co-operation and Development, the worldwide program Assessment of Higher Education Learning Outcomes tests “what students in higher education know and can do upon graduation”.9 Across 248 higher education institutions in 17 countries, the program has directly examined and compared generic and discipline-specific skills in engineering and economics. Its rationale is to reliably measure the learning outcomes that result from the massive investments made by society in higher education.
In the United States, all doctors must take the US Medical Licensing Examination (USMLE) to be able to practise medicine. It is a three-step exam (with two steps usually taken before graduation), developed and delivered by the National Board of Medical Examiners (NBME). The USMLE is independent of individual medical schools’ assessments, but most US schools require students to pass it as well as their own internal exams in order to graduate.
The NBME has developed a global version of the USMLE, called the International Foundations of Medicine (IFOM) examinations, that is being used by medical schools in Europe, Asia and Latin America.10 Some Australian medical schools are working with the NBME on the integration of the IFOM into their assessment programs, and the medical schools of the Group of Eight universities are scoping a benchmarking process using the IFOM.11
In the United Kingdom over the past decade, a collaborative approach to assessment among a small number of medical schools has moved to sector-wide collaboration.12 And, lastly, in Europe, a project is underway that seeks to harmonise assessment of medical student competency across member states.13
Returning to the local picture, in Australia, the relatively rapid expansion in the number of medical schools and the doubling of medical student numbers has put considerable pressure on educational resources. The value of collaboration across the sector has been recognised, with the recent development of several collaborative assessment networks (Box 1).
In addition, the AMC encourages schools to make use of assessment items used in its international medical graduate exams, in order to benchmark performance of these items against Australian students.
Although these efforts reflect positively on the collaborative nature of medical education in Australia, they are limited in scope, scale and impact. Benchmarking data has only been produced by the Australian Medical Schools Assessment Collaboration,14 and relates to biomedical sciences. Stakeholders — including the AMC, Medical Board of Australia, Colleges, health services and the broader community — may increasingly expect more quantitative confirmation of the robustness of quality assurance of medical education.
The concept of a national exit examination for medical students has been explored previously in Australia,15 and the pros and cons of this approach have been considered globally.16 As a group, Australia’s medical deans appear to be opposed to a national licensing exam, as it risks removing responsibility for quality assurance from medical schools, and fracturing the important nexus between curriculum, teaching and assessment.
We regard nationwide, collaborative assessment as a preferred model to measure student outcomes. The wording is important. “Nationwide” implies “across Australia”, without mandating that every medical school should take part. “Collaborative” seeks to make it clear that participating medical schools own and are responsible for any assessment process.
One example of a model that could possibly fit the bill is the Australian Medical Assessment Collaboration (AMAC). The AMAC was initially conceived as a joint project between the University of Queensland, Monash University and the Australian Council of Educational Research, and was funded as a strategic project by the Australian Learning and Teaching Council over an 18-month period to mid 2012. The project included scoping, engagement, development of an assessment framework, compilation and validation of sample test items, and pilot exams (Box 2).17
The project progressed with sector-wide engagement, including national symposia held in May 2011 and June 2012. As the AMAC assessment framework was developed, additional consultation occurred directly with the medical schools involved. Ten schools provided a total of 420 assessment items; most took part in review, revision and redrafting of the items during two workshops. A final collection of 120 multiple choice questions was selected for the AMAC item bank. The bank is mapped to a comprehensive assessment framework, and each question has psychometric analysis of its performance to ensure the construction of a valid, reliable and unbiased assessment instrument.17 In November 2011, a small pilot involving 49 medical students was conducted in two universities (Monash University and the University of Queensland); and, in the first half of 2012, 512 students and 25 staff across seven medical schools took part in a large-scale online pilot. The early indications were that assessment through the AMAC framework could constitute a substantial fraction (perhaps 25%) of the final exam in each of the collaborating medical schools. This would ensure that the shared assessment sufficiently samples the curriculum, and is valid and reliable, while individual schools retain responsibility for assessment. Accordingly, AMAC has been expanded to currently involve 16 medical schools in Australia and New Zealand (and is open to all schools in both countries) and has been awarded a grant for further development of the collaboration.
Although more work remains to be done before any model of nationwide, collaborative assessment is acceptable to most medical schools, key issues have already emerged. These include: that an externally designed and delivered national licensing exam is not supported by schools in Australia; that a model that is collaborative and owned by schools with appropriate external verification has support; and that a collaborative approach to assessment across schools can foster development of assessment expertise and capacity within the sector. In future, in addition to assessment that tests knowledge and knowledge applications, clinical skills and professionalism must be assessed.18
Overall, it seems that regulatory developments in higher education and the medical profession will support, or even require, some form of formal process for assuring and comparing standards of medical education across medical schools. The medical education sector is embracing this evolution, and by taking a proactive, responsible leadership position, it can ensure that the learning outcomes from Australian medical schools are world-class — and can be shown to be so.
1 Australian collaborations involved in assessing medical students’ learning outcomes
ACCLAIM (The Australian Collaboration for Clinical Assessment in Medicine): Six medical schools (James Cook University, Deakin University, Griffith University, University of Newcastle, University of Tasmania, University of Wollongong) collaborate on common clinical assessment instruments.
AMAC (Australian Medical Assessment Collaboration): Created by the University of Queensland and Monash University with the Australian Council of Educational Research, there are now 16 medical schools involved. The focus of AMAC is on common assessment of clinical sciences and clinical medicine student outcomes towards the end of a medical degree.
AMSAC (Australian Medical Schools Assessment Collaboration): Led by the University of Sydney; partners include Monash University, University of New South Wales, University of Western Sydney, Australian National University, Flinders University and the University of Queensland. The intent of this collaboration is to benchmark performance of schools by making use of a common set of assessment items in examinations that assess biomedical sciences.
IDEAL (International Database for Enhanced Assessments and Learning) Consortium: Based at Bond University, 27 partner schools around the world generate databanks of summative and formative assessment items (http://temporary.idealmed.org/wp/about-us).
2 Main objectives of the project funded by the Australian Learning and Teaching Council*
1. The establishment of the Australian Medical Assessment Collaboration as a body to facilitate the ongoing implementation of a national assessment of learning outcomes in the medical field.
2. Sector-wide involvement in an aspect of educational practice (ie, outcome assessment) that is gaining serious momentum in the global landscape.
3. A model for the implementation of discipline-based objective testing of learning outcomes.
4. Framework and item development for the medical field to facilitate national benchmarking and evaluation of outcomes.
5. Initial psychometric evaluation of the results of the pilot testing of draft items. Psychometric data are provided for each question included in the assessment; these data indicate how the question performed in the exam, and hence, how good a question it is. This is regarded as best practice in evaluating assessment.
6. Development of an item bank, formed to provide the basis for the future development of a robust assessment instrument in the field.
* Collaboration between the University of Queensland, Monash University and the Australian Council of Educational Research, funded over an 18-month period to mid 2012.17
Competing interests
No relevant disclosures.
Acknowledgements
Several projects already underway may lead to national standards in competencies for medical students, but more work remains to be done before any model of nationwide, collaborative assessment is acceptable to most medical schools.
References
- Australian Associated Press. Jayant Patel to face jury on manslaughter charge. The Australian 2012; 27 Nov. http://www.theaustralian.com.au/news/nation/jayant-patel-to-face-jury-on-manslaughter-charge/story-e6frg6nf-1226524785539 (accessed Dec 2012).
- Zhang JJ, Wilkinson D, Parker MH, et al. Evaluating workplace-based assessment of interns in a Queensland hospital: does the current instrument fit the purpose? Med J Aust 2012; 196: 243. 0_i1115669
- Bingham CM, Crampton R. A review of prevocational medical trainee assessment in New South Wales. Med J Aust 2011; 195: 410-412. 0_i1115671
- Australian Government Department of Education, Science and Training. Australian Medical Education Study. A review of clinical education in undergraduate medicine: a literature review. Canberra: Commonwealth of Australia, 2007. 0_i1115673
- Australian Medical Council. Assessing basic medical education. http://www. amc.org.au/index.php/ar/bme (accessed Oct 2013).
- O’Keefe M, Henderson A, Pitt R. Learning and Teaching Academic Standards Project. Health, medicine and veterinary science Learning and Teaching Academic Standards Statement, June 2011. Sydney: Australian Learning and Teaching Council Ltd (an initiative of the Australian Government Department of Education, Employment and Workplace Relations), 2011. http://www. olt.gov.au/resources/3899,4060?text=accreditation&solrsort=score %20desc (accessed Oct 2013).
- Medical Deans Australia and New Zealand. Competencies. http://www. medicaldeans.org.au/projects-activities/competencies (accessed Oct 2013).
- Australian Medical Council. Accreditation standards for primary medical education providers and their program of study and graduate outcome statements. http://www.amc.org.au/index.php/about (accessed Oct 2013).
- Organisation for Economic Co-operation and Development. Skills beyond schools. Testing student and university performance globally: OECD’s AHELO. http://www.oecd.org/education/skills-beyond-school/testingstudentanduniversityperformancegloballyoecdsahelo.htm (accessed Oct 2013).
- National Board of Medical Examiners. IFOM: International Foundations of Medicine. http://www.nbme.org/Schools/iFoM/index.html (accessed Dec 2012).
- National Board of Medical Examiners. 2012 annual report. NBME, 2013. http://www.nbme.org/PDF/Publications/2012Annual-Report.pdf (accessed Oct 2013).
- Medical Schools Council. Medical Schools Assessment Alliance. http://www. medschools.ac.uk/MSCAA (accessed Dec 2012).
- European Board of Medical Assessors. [Website.] www.ebma.eu.com (accessed May 2013).
- Wilson I, O’Mara D. The Australian Medical Schools Assessment Collaboration: what do the differences mean? In: Proceedings of the Australian and New Zealand Association for Health Professional Educators Conference, 2011: Local? Global? Health professional education on social accountability; 2011 Jun 27-30; Alice Springs, Northern Territory. 0_i1115693
- Koczwara B, Tattersall MHN, Barton MB, et al. Achieving equal standards in medical student education: is a national exit examination the answer? Med J Aust 2005; 182: 228-230. 0_i1115695
- Harden RM. Five myths and the case against a European or national licensing examination. Med Teach 2009; 31: 217-220. 0_i1115698
- Australian Council for Educational Research. Australian Medical Assessment Collaboration (AMAC). Framework. http://www.acer.edu.au/amac (accessed Dec 2012).
- van der Vleuten C. Validity of final examinations in undergraduate medical training. BMJ 2000; 321: 1217-1219. 0_i1115702
Provenance: Not commissioned; externally peer reviewed.